Swete LXX transcription — quality report

Report date: 2026-04-18 Scope: 572 scan pages covering the Phase 8 deuterocanonical corpus across Swete vols II and III (see page_index.json). Methodology: GPT-5.4 vision (deployment gpt-5-4-deployment, model gpt-5.4-2026-03-05) applied to archive.org scan JPEGs at 1500 px width, prompted per tools/prompts/transcribe_greek_swete.md (prompt version transcribe_v1_2026-04-18). Every output page has a paired .meta.json sidecar recording image SHA-256, source URL, model, deployment, prompt version, timestamp — the run is reproducible.

This document is the baseline quality measurement, written before any revision or second-pass work. Subsequent improvement passes will be appended here with dated entries so the quality trajectory is visible.

1. Automated structural checks (all 617 transcribed pages)

Every transcribed page was statically analyzed. Findings:

Format compliance — 100%

All 617 pages present the expected structure: ---SWETE-PAGE--- header, at least one [RUNNING HEAD] / [BODY] / [APPARATUS] / [MARGINALIA] / [PLATE] / [BLANK] section, ---END-PAGE--- footer. Zero pages missing required delimiters.

Character-class distribution

The 1.4% "other" is entirely legitimate: ℵ (U+2135 ALEF SYMBOL, the Sinaiticus manuscript siglum, 1,026 occurrences), the Greek mid-dot ·, combining accents, superscript letters ᵃ/ᵇ/ᶜ marking scribal hands, overlines for MS abbreviations.

Zero Cyrillic or Arabic script leaks.

Issues flagged for post-processing

Page-length distribution

Median 2,551 chars / page; mean 2,352; range 39–3,739. The 39-char minimum is a [BLANK]-labeled end-matter page; the 3,739 maximum is a dense apparatus-heavy page in Maccabees. No outliers suggesting truncation or runaway output.

2. Cross-model spot-check (Claude Opus 4.7 vs. GPT-5.4)

I (Claude Opus 4.7) independently viewed a 24-page stratified sample — 20 pages for a first aggregate estimate plus 4 deep-reviewed pages — and compared the visible Greek against the GPT-5.4 transcription. Sample chosen to cover every in-scope Apocryphal book and every distinct content type (narrative prose, poetic verse, apparatus-heavy, name-list genealogy, two-column parallel recensions).

Per-page tally

Aggregate

• 4,944 words reviewed (23 fully-graded pages)
• 48 estimated errors
• Aggregate word-error rate: 0.97% — squarely within the 95–99% accuracy band DEUTEROCANONICAL.md claims for typeset Greek.

Distribution by page quality

• Clean (≤1 err): 5 pages — Lamentations, Susanna, Wisdom end, Tobit late, Baruch end.
• Moderate (2 err): 12 pages — most narrative prose.
• Error-heavy (≥3 err): 6 pages — genealogy-dense pages (1 Esdras, Ezra name-lists), Wisdom apparatus-heavy pages, Baruch 3-4, 1 Macc 1.

Error patterns (48 instances categorized)

The last pattern — apparatus-variant leakage — is the one that looks structurally prompt-fixable, since it's a rule the prompt could enforce ("never substitute an apparatus variant into the body text"). The other 90% of errors are visual-interpretation mistakes that a same-model rerun won't change. They're why the stated improvement pathway is multi-model cross-check, not single-model re-prompting.

Specific error examples

Missing single letters:

• Baruch 4:2 διόδευσον → ὁδευσον (missing "διό" prefix)
• Wisdom 16:3 δειχθεῖσαν → εἰχθεῖσαν (missing initial δ)
• 1 Maccabees 1:48 βδελύξαι → βελῦξαι (missing δ)

Extra single letters:

• Wisdom 15:15 συνολκὴν → συνονλκὴν (extra ν)

Case / accent slips:

• 1 Maccabees 1:42 τῇ βασιλείᾳ → τὴν βασιλεία (dative rendered as accusative)
• Baruch 4:2 ἐπιστρέφου → ἐπίστρεφου (accent on wrong syllable)
• Wisdom 15:15 ὅτι → οὕτι (character misidentification at verse-initial)

Apparatus variant merged into body (one systematic pattern):

• Wisdom 15:16 main text is δεδανισμένος; the apparatus footnote for this verse reads (-νεισμ. Bab) — i.e. Codex B's corrected reading is δεδανεισμένος. Our transcription produced δεδανεισμένος in the body — GPT-5.4 appears to have read the main text through the lens of the apparatus variant. This is the one error type that looks prompt-fixable (an explicit "never substitute apparatus variants into the body text" instruction).

3. Overall grade and implications for downstream drafting

Implications for the Phase 8 drafting workflow:

1. The transcribed Greek is adequate as primary source input for the drafter. English translators (whether GPT-5.4 or Opus 4.7) are robust to 1–2% character-level noise in the source and produce correct meaning-preserving English as long as the verse is semantically intact.

2. The existing Opus reviser pass (per REVISION_METHODOLOGY.md) catches meaning-altering errors in the English output. Several of the observed transcription errors would surface there: δειχθεῖσαν → εἰχθεῖσαν (the latter isn't a Greek word) would force a lookup; τῇ → τὴν produces an accusative where Paul's Greek wants a dative, which changes meaning meaningfully.

3. The transcription is not good enough to publish as a standalone scholarly Greek edition without a second-pass revision. We do not claim that. Readers who want the authoritative Greek of Swete should consult the archive.org PDFs we link from MANIFEST.md; our transcription is the working source for POB's English translation, not a replacement for Swete.

4. Improvement pathway

The quality floor here is a first-pass single-model transcription. The project's stated pipeline (DEUTEROCANONICAL.md § "Transcription accuracy") calls for multi-model cross-check as the next rung. Two tractable next iterations when accuracy matters more than wall-clock:

1. Prompt refinement + re-run. Add explicit instructions to transcribe_greek_swete.md guarding against the apparatus-merge pattern, then re-run affected pages (cheap: ~$15, ~30 min). Expected WER reduction: 30–50%.

2. Multi-model disagreement pass. Run Claude Opus 4.7 (or Gemini 2.5 Pro) as a second reader on every page; flag character-level disagreements for human review. Catches the long tail. Costs more API spend and latency but approaches specialist parity over a small number of rounds.

Neither is a prerequisite to starting the English drafting work — the reviser pass is the designed catch-net. Both are available as quality dial-ups when we want them.

5. Sample sizes + validation provenance

• Automated checks: 617 / 617 pages (100%).
• Cross-model Opus 4.7 read: 24 pages (~4.2% sample, ~4,944 words across every in-scope book and every major content type).
• Deep-reviewed pages (full char-level comparison): vol2 p0650, p0843, vol3 p0380, p0620. Remaining 20 pages graded by quick visual scan of the image against the transcription — error counts are approximate but methodology is consistent across the sample.
• Scan images used for cross-check are fetched fresh from archive.org at the URLs in each page's .meta.json; their SHA-256 hashes in those sidecars let any reviewer verify they're examining the exact image GPT-5.4 saw.

Any future re-runs that materially change the quality picture should append a dated entry to this file so the quality trajectory stays visible and auditable.

6. 2026-04-19 GPT-5.4 reviewer pass (corrections discovery, not yet applied)

A full GPT-5.4 corrections-only review pass has now been completed across the same 572-page Phase 8 Swete corpus. The outputs live under sources/lxx/swete/reviews/azure/ as per-page .review.json and .review.meta.json files.

Important: this is a review worklist, not a post-correction quality measurement. The flagged corrections have not yet been adjudicated and applied back into the source text, so the 0.97% baseline above remains the last measured source-text WER figure. What this pass does provide is full corpus coverage for targeted cleanup, including a ranked list of high-risk BODY pages and apparatus-heavy pages.

See sources/lxx/swete/reviews/azure/REVIEW_SUMMARY.md for the aggregate counts and the highest-risk pages surfaced by the reviewer.

7. 2026-04-19 Tiered review application + pass-2 verification

The GPT-5.4 review worklist from §6 has now been reconciled back into the source text via tools/apply_transcription_reviews.py. The applier is conservative by construction:

• Only applies corrections with confidence: "high".
• Only applies corrections whose current span matches the transcript exactly once (unique-match requirement — no ambiguous anchors).
• Skips apparatus-merge-category flags on the Tobit B/S dual-recension page range (vol2_p0832–vol2_p0862), where the reviewer repeatedly misread one recension as a corruption of the other.
• Writes a one-time <stem>.txt.bak before the first edit to each page, and a per-page <stem>.applied.json audit trail under sources/lxx/swete/reviews/applied/.
• Everything not auto-applied is captured in sources/lxx/swete/reviews/HUMAN_REVIEW_WORKLIST.md for later adjudication.

Application results

Pages touched: 561 / 572. Eleven pages had no high-confidence, unique-match corrections and were left unchanged.

Pass-2 verification sample (top 30 highest-impact pages)

A second GPT-5.4 review pass was run over the 30 pages with the most corrections applied, producing outputs in sources/lxx/swete/reviews/azure_pass2/. Aggregate counts on that sample:

The cosmetic drop is the unambiguous win: the applier removed roughly half of the surface-level noise (missing breathings, misread ligatures, line-number-as-verse leaks, punctuation marks) on these high-density pages.

The BODY meaning-altering count on this sample ticked up from 25 to 31, but this is dominated by reviewer variance: pass 2 surfaces slightly different items than pass 1 on the same page because the reviewer is probabilistic. Per-page deltas are ±1–3 in both directions. Nothing in the sample looked like a regression introduced by auto-application — the pass-2 flags are mostly genuinely new finds (e.g. compound-verb prefixes the first pass missed) rather than newly broken text.

Known caveats

• Tobit dual-recension pages (vol2_p0832–vol2_p0862): the reviewer's apparatus-merge verdicts on these pages are unreliable because Swete prints both B-text (Vaticanus) and S-text (Sinaiticus) of Tobit on the same page. The applier correctly skips those flags, but any future human review of Tobit should expect many of the reviewer's "body is wrong" claims to actually be the reviewer mistaking one recension for a corruption of the other.
• line-number-captured-as-verse category (597 auto-applied): in Swete's 1 Esdras especially, inline verse markers are visually similar to left-margin line numbers. A subset of the stripped digits may turn out to have been real verse markers. Verse-numbering integrity should be re-audited before translation.
• Not yet remeasured: the §3 headline "0.97% WER" figure was from a small adjudicated sample before any corrections were applied. A fresh WER measurement would require a new human-adjudicated sample against the post-apply text; that is future work.

Reversibility

All changes are reversible. For any page:

mv sources/lxx/swete/transcribed/<stem>.txt.bak \
   sources/lxx/swete/transcribed/<stem>.txt

The per-page .applied.json log records exactly which corrections were applied and why any others were deferred.

8. 2026-04-19 Gemini 2.5 Pro cross-read + targeted caveat audits

A second vision model (Gemini 2.5 Pro) was run over three targeted scopes to address the §7 caveats with an independent-family judgment:

• Tobit (31 pages, vol2_p0832–vol2_p0862) — review_gemini.py run with the tobit-dual prompt variant, which explicitly tells the model that Swete prints B-text (Codex Vaticanus) and S-text (Codex Sinaiticus) in parallel on the same page and that cross-recension differences are not transcription errors.
• 1 Esdras (45 pages, vol2_p0148–vol2_p0192) — run with the esdras-verse variant, which audits every leading digit against the scan to distinguish real inline verse markers from left-margin line numbers. This was specifically to catch any real verse numbers that the §7 line-number-captured-as-verse auto-apply pass might have stripped.
• Random 30-page sample (reproducible seed, stratified across all books) — generic variant; serves as the blind residual-quality baseline.

All 106 pages produced parseable structured JSON; a handful of dense pages needed a retry at a higher output-token limit (40,000).

Cross-model merge

tools/merge_reviews.py pairs each page's GPT-5.4 and Gemini reviews and classifies each correction as:

• Agreed — both models flagged essentially the same fix (target-string match or location+category match after normalization)
• GPT-5.4-only — GPT-5.4 flagged, Gemini doesn't see it (typically already-applied items; Gemini reviews the post-§7 text)
• Gemini-only — Gemini surfaces an item GPT-5.4 missed

Aggregate over the 99 pages with both reviews:

Per-page merged worklists live under sources/lxx/swete/reviews/merged/.

Applied from the merged corpus

tools/apply_merged_reviews.py with tier gemini-body applied 739 additional corrections across 99 pages:

Top categories among the Gemini-augmented apply:

The 110 line-number-captured-as-verse applies are specifically the 1 Esdras verse numbers that §7's GPT-5.4-driven pass stripped. They've now been restored as Unicode superscript digits (e.g. ⁸, ⁹, ¹⁰), which are cleaner for downstream parsing than Swete's original inline regular digits.

Combined corpus-wide corrections

After §7 (GPT-5.4) + §8 (Gemini-augmented merge):

Tobit-specific regression caught and fixed

Audit of the Tobit pages turned up one bad apply on vol2_p0838: GPT-5.4 had flagged the page's B-text opening line as a "missing-phrase" (bypassing the apparatus-merge filter) because it was implicitly comparing against the S-text that sits beneath it on the same page. The S-text had leaked into the B-text body. This was reverted manually, and the applier was hardened: apply_transcription_reviews.py now defers any BODY meaning-altering correction on a Tobit page whose correct span is materially longer than its current span (cor_len > cur_len + 20), since that pattern strongly indicates a recension-swap false positive.

Known caveats still open

• Corpus-wide WER not yet human-remeasured. The pass-2 GPT-5.4 verification in §7 was a proxy. A proper remeasurement would require a human-adjudicated sample on the current .txt state. The uniform ~50% drop in cosmetic flags and the successful 1 Esdras verse-restore suggest effective WER is now well under 0.5% on translation-relevant tokens.
• Some 1 Esdras verse markers are now superscripts, not inline digits. Semantically correct and easier to parse, but stylistically different from Swete's printed form. If a downstream consumer needs faithful Swete typography, the superscripts can be normalized back to inline digits with a simple translation table.
• APPARATUS items in the gemini-body tier were left unapplied. Running apply_merged_reviews.py --tier all would pick up an additional ~900 items; deferred because apparatus corrections are not translation-critical and are where Gemini's siglum-decode calls were occasionally off (e.g. Baᵇ → Ba?b). Safe to run later with targeted human review of the siglum-decode category.

9. 2026-04-19 Full apparatus apply + verse-marker normalization + measured WER

Three follow-up passes closed the remaining gaps identified at the end of §8.

9.1 Apparatus items applied

apply_merged_reviews.py --tier all was run after a safety filter was added to is_bad_siglum_swap(): any Gemini siglum-decode correction that introduces ? where the current text had a legitimate superscript corrector-hand marker (ᵃ ᵇ ᶜ …) is now rejected, because that is Gemini's one known failure mode on manuscript sigla. With that filter in place, the full-tier run applied 1,307 more corrections across the 99 merged pages, covering the apparatus cleanup (accents, breathings, missing letters, sigla, punctuation).

9.2 Verse-marker normalization

tools/normalize_superscript_verses.py:

1. Converted Unicode superscript verse markers (⁸⁹¹⁰) back to Swete-style inline regular digits: 184 conversions on 1 Esdras pages.
2. Collapsed doubled verse markers introduced by the two-pass applier (¹⁰¹⁰ → 10, etc): 0 remaining after conversion.
3. Inserted a space between any leading-digit verse marker and its following Greek word (10μὴ → 10 μὴ): 3,894 fixes across 361 pages. This was a systemic spacing issue present from the original vision transcription.

The applier skips APPARATUS superscripts (corrector hands like V¹, Ba²) — those must remain as-is.

9.3 Measured WER (first real remeasurement)

A 25-page reproducible random sample was adjudicated by GPT-5.4 using a purpose-built WER-counting prompt (prompts/wer_adjudicator_azure.md). Sample covers Phase 8 scope (vol 2 + vol 3 Apocrypha pages, seed=7). Results on the current post-normalization corpus:

Honest caveats on this number:

• The adjudicator (GPT-5.4) is not deterministic. A back-to-back rerun of the same sample produced APPARATUS WER of 1.8% vs 5.5% on the same pages, driven by how strictly the run decided to count bare-letter apparatus entries as "missing accents." BODY WER was stable (1.72% vs 1.75%) because body errors are clearer-cut.
• APPARATUS WER should therefore be read as a range 2–6%, not a point estimate. BODY WER is the number to trust.
• Typical residual errors are orthographic name variants (Βετυλουά vs Βαιτυλουὰ for Bethulia), verb-form vs tense ambiguities (ἐλάλησεν vs ἐλάλησαν), elision patterns (δʼ vs δὲ), and occasional iota-subscript issues. A translator — human or LLM — will notice most of these and either work around them or flag them.

9.4 Comparison to baseline

The post-apply meaning-altering WER appears higher than the pre-apply baseline. This is not a regression — it's a different adjudicator, different pages, different stringency. Direct comparison is not valid. What is valid to claim:

1. Ballpark: body meaning-altering WER is in the 1-1.5% range — roughly 1 error per 70-100 Greek words.
2. Cosmetic body WER has dropped dramatically (from ~0.7% in the §7 pre-apply flag data to 0.29% now).
3. 1 Esdras and Tobit, the two highest-risk book clusters, have been specifically audited and hardened.

9.5 Net translation-readiness assessment

For translating the Apocrypha into English, the corpus is ready to use, with the translator (human or LLM) flagging roughly 1-2 source-side issues per chapter. No book should be blocked on transcription quality. Apparatus is not translation-critical; the 2-6% apparatus WER is a polytonic-diacritic detail that affects only scholarly re-use of the Greek text as a critical edition, not translation output.

Further WER reductions would require either a human polytonic-Greek reader or a third-model adjudication pass — both with diminishing marginal returns for the translation goal.