How to Verify Transcription Accuracy: A Working Method for Historical Documents

This is a practical guide to verifying transcription accuracy in automated output — what the error figures actually mean, why fluent text is more dangerous than garbled text, and how to check a page so that the highest-consequence errors surface first. A transcription you can trust is one you have checked against the image, whatever tool produced it.

To verify automated transcription accuracy, work from the manuscript or print itself — not from the model's fluent output — and concentrate your effort where errors do the most damage: proper names, numbers, dates, negations, and any passage your argument will rest on. A headline character error rate tells you how a model performed on someone else's corpus, not how it performed on your page. Verification closes that gap. It is the step that converts a plausible draft into a defensible source, and it is not optional.

This guide sets out a working method: what the accuracy numbers mean, why fluent output is more dangerous than garbled output, and a practical sequence for checking a page. It applies whether you are transcribing an Elizabethan will, a French notarial register, a Dutch parish book, or an early modern printed pamphlet.

First, understand what the accuracy figure is telling you

Before you can verify a transcription, you need to know what claim you are verifying against. Most automated tools report a character error rate (CER): the minimum number of character-level edits — substitutions, insertions, deletions — needed to turn the machine's output into a correct reference transcription, divided by the total number of characters in that reference. The calculation runs on Levenshtein edit distance, and it is worth being precise about what it does and does not say; our guide to AI transcription accuracy and the CER formula goes further into the metric.

A 5% CER does not mean five words in a hundred are wrong. It means roughly five character edits per hundred characters. In practice a 5% CER corresponds to somewhere around 95–99% word accuracy and only 70–80% line accuracy — the ratios shift with script, language, and model, so treat them as orientation, not law. Word error rate (WER) is the same computation at the word level, and it is brittle in a specific way: one misplaced space can throw off every word match that follows it, so a low CER can sit alongside a handful of genuinely unusable words.

Two consequences follow for verification.

First, a headline figure does not transfer to your document. Reported CER is conditioned on the model's training distribution — script, hand, period, language, image quality, layout. This is precisely why benchmark corpora like IMPACT (over 600,000 historical images) and GT4HistOCR (313,173 line pairs) exist: no single test set represents everything. There is no portable adjustment factor that tells you how much accuracy degrades when you move from one collection to another. The number on the vendor's page is a starting hypothesis about your material, not a measurement of it.

Second, the metric normalizes by reference length, which means CER can exceed 100% when the hypothesis runs longer than the reference — a real signal that a model has invented text rather than merely misread it. That failure mode is where verification earns its keep.

Why fluent output is the more dangerous failure

There is a temptation to assume the errors worth hunting are the obvious ones: garbled characters, split words, dropped lines. Those are real, but they are the recoverable kind. You can see them, and a glance at the image fixes them.

The dangerous errors are the ones that read perfectly. General-purpose AI models — ChatGPT, Claude, Gemini and the like — are increasingly reached for as quick transcription tools, and although they have improved quickly, they are not built for historical documents. They often downsample the high-resolution image they are given and lean on their language model to produce text that fits the layout and the surrounding sense. Their most common slip is quiet normalisation, a more probable modern spelling in place of the one on the page; at worst, a plausible word or line that is not there at all. The Crosilla, Klic and Colavizza benchmark (2025) found specialist HTR models outperforming the general models of that date on historical handwritten text across the conditions tested — a single-corpus study that, like any benchmark, still requires you to verify against your own material rather than take the ranking on faith.

The mechanism matters for how you verify. A specialist recognition model errs locally — a wrong character, a split word — and those errors sit visibly against the image. A general LLM errs coherently: a plausible name where the manuscript has a different one, a smoothed clause where the scribe wrote something awkward, an entire invented line that reads as period-appropriate prose. Coherent errors survive a casual read precisely because nothing looks wrong. This is the reason ChatGPT struggles with historical handwriting in ways that are hard to catch, and why LLM transcription errors demand a different verification posture than OCR noise. The whole argument for keeping a human in the loop rests here: the more fluent the output, the more disciplined the check must be.

The verification method, step by step

Verification is not re-transcription. It is a targeted comparison of the output against the image, ordered so that the highest-consequence errors surface first. Here is a working sequence.

Read the transcription against the image, not on its own

The most common mistake is proofreading the transcription as text — checking whether it reads sensibly. That check rewards exactly the fluent fabrications you most need to catch. Put the image and the transcription side by side and read the page, line by line, with your eye on the source. You are not asking "does this make sense?" You are asking "is this what is written?"

Triage with confidence scores, but do not trust them

Some engines expose per-character or per-word confidence values. Tesseract returns character-level confidences; Transkribus surfaces them in its validation interface. These are useful for triage — they point you at the tokens the model was least sure of — but they are model posteriors, not calibrated probabilities of correctness. No comprehensive study has established that confidence scores from the major engines are calibrated against actual correctness on difficult historical hands, so treat a high-confidence reading as a place you can look faster, never as a reading you can skip. And note that general LLM transcription typically exposes no per-token confidence at all, removing even this rough triage signal.

Verify the high-consequence tokens first

Not all errors cost the same. Concentrate scrutiny on:

• Proper names — people and places, where a single substituted letter changes an identification and where the model has no linguistic prior to lean on
• Numbers and dates — sums in an account, ages in a census, regnal or calendar dates. These carry no redundancy: a wrong digit is undetectable from context. One misread word can do the same: in 1515 Guillaume Budé argued that a single misread inflection in Pliny had inflated the sums of Roman public finance a hundredfold.
• Negations and quantifiers — "not," "never," "none," "all." A dropped negation inverts meaning while reading perfectly.
• Anything your argument will cite. If a clause is going to appear in a footnote, verify it character by character.

Check that historical spelling survived

Ask specifically whether the output preserves what was on the page or has quietly modernized it. Many engines normalize in the background — long s (ſ) mapped to round s, u/v and i/j interchange resolved, "ye" silently turned to "the," abbreviations expanded without notice. This is normalization, not transcription, and it erases evidence that orthographic and dating studies depend on; if you want a modernised text, make it as a separate layer, as historical language normalization explains. A published CER can look excellent while the historical record has been modernized out of existence. If your source is printed, the same discipline applies to the long s, ligatures, and the macron or titulus standing in for an omitted nasal — marks that OCR built for modern type routinely drops or "corrects." Reading these correctly is a matter of paleographic literacy as much as tooling; our guides to manuscript abbreviations and ligatures and reading early modern paleography cover the marks you are checking for, and the Leo Academy Handbook has a page for each common secretary hand abbreviation, from the abbreviation y^t for that onwards.

Verify against ground truth where the stakes justify it

For high-value material — a corpus you will publish, a reference edition — the discipline standard for establishing correctness is double-keying, also called four-eyes verification. Two transcribers independently produce the same text, and an adjudicator reconciles the differences; the adjudicated text becomes ground truth. Note that double-keying means independent transcription, not one person checking another's work. And ground-truth quality dominates quantity: roughly twenty carefully corrected pages outperform a hundred pages carrying errors in the reference. If you are measuring a model's real CER on your material — rather than trusting its headline — this is how you build the reference to measure against, and open evaluation tools such as dinglehopper will compute CER and WER transparently once you have it.

Keep interpretation separate from the base transcription

Whatever you do to a transcription after verifying it — correcting residual errors, modernizing spelling, translating, extracting names — should be layered on top of the faithful base reading, not baked into it. Once you overwrite the verbatim transcription with a corrected or modernized version, you have lost the ability to check either against the image, and you have lost the record of what the source actually says. Keep the base reading intact and reversible; this is the throughline in any serious historical research workflow.

Where the tool you choose changes the verification burden

Verification is unavoidable, but the tool that produced the draft determines how much of it you do and which errors you are hunting. If a model returns fluent fabrications, verification means re-reading the whole page against the image, because you cannot trust the parts that look right. If a model errs locally and preserves what is on the page, verification narrows to a targeted check of the high-consequence tokens.

This is the ground on which Leo is built. Its own model, ATR-1, is tuned for historical documents in Latin script — English, French, German, Dutch, Spanish, Italian, Latin and the other languages written in that alphabet, across manuscript hands and historical print alike. It is trained to transcribe what is on the page rather than normalise it: it does not over-correct or silently modernise, so strikethroughs, marginal additions, editorial expansions and archaic orthography survive, which is exactly what the spelling check above is watching for. It reads the image at high resolution, where general models downsample, needs no training on your hand, and checks its own output: anything that looks like a failure is hidden and retried, and the credit is refunded if it can't succeed. The errors that do get through are the recoverable kind, a wrong letter you check against the image beside the text. None of that describes your document; the verification method above still applies in full.

What a model that errs locally changes is the shape of the work: the base transcription in Leo stays faithful to the page, and every transformation — correction, modernization, translation, entity extraction — writes to a separate tab, leaving the verbatim reading untouched and available to check against the image. That is verification made structurally easier, not verification made unnecessary. The image sits beside the transcription so the side-by-side read is the default view, not something you have to assemble. Leo's model library helps with triage too: ATR-1 is the default, and the major OpenAI, Gemini and Claude models sit in the same menu, so you can run a second reading into its own tab and look first at the places where the two disagree (an agreement is not proof; both can normalise the same word). And because corrections users make feed back into training, the model's accuracy compounds across releases — though nothing about that removes the human check at the point of use.

The habit that outlasts any tool

No model, Leo included, absolves you of reading the page. The accuracy figures are useful for choosing where to start and misleading if you treat them as a warrant. What makes a transcription trustworthy is not the number on the vendor's site but the discipline you bring to the check: reading against the image rather than against the model's fluency, spending your attention where errors are undetectable from context, and keeping the verbatim reading intact so that anyone — including you, later — can go back to the source and see for themselves. That habit is the oldest thing in paleography, and the part no automation reaches. Keep it, and the tools become what they should be: fast first drafts, honestly checked.

Frequently Asked Questions

How do you verify transcription accuracy in automated output?

Verify a transcription by working from the manuscript or print itself — not the model's fluent output — and concentrate effort where errors do the most damage: proper names, numbers, dates, negations, and any passage your argument will rest on. Put the image and transcription side by side and read line by line, asking "is this what is written?" rather than "does this make sense?" Confidence scores can help triage, but treat them as pointers, not guarantees. Check that historical spelling survived rather than being quietly modernized. For published or reference material, establish ground truth through double-keying. A headline accuracy figure describes someone else's corpus, not your page.

Q: Can I validate transcription accuracy automatically? Partly. If you have a reference transcription, tools such as dinglehopper compute CER and WER automatically, which tells you how a model did on that sample. Without a reference, automatic signals only triage: confidence scores point at uncertain tokens, a second model's reading shows where two readings disagree, and some tools detect failed output (Leo's ATR-1 hides, retries and refunds anything that looks like a failure). None of these confirms that a reading matches the page, so the final check is still a person reading the transcription against the image, starting with names, dates, numbers and negations.

What is a character error rate (CER) and what does it actually mean?

A character error rate is the minimum number of character-level edits — substitutions, insertions, deletions — needed to turn a machine's output into a correct reference transcription, divided by the total characters in that reference, calculated using Levenshtein edit distance. A 5% CER does not mean five words in a hundred are wrong; it means roughly five character edits per hundred characters, corresponding to somewhere around 95–99% word accuracy and only 70–80% line accuracy. Those ratios shift with script, language and model. A reported figure is conditioned on the model's training material, so it is a starting hypothesis about your document, not a measurement of it.

Why is fluent transcription output more dangerous than garbled output?

Fluent output is more dangerous because its errors read perfectly and survive a casual proofread. Garbled characters, split words and dropped lines are recoverable — you can see them and a glance at the image fixes them. General-purpose AI models fail differently: they often downsample the image and lean on their language model to produce text that fits the layout, so their errors are quiet modernised spellings, smoothed clauses, and occasionally plausible words or lines that are not on the page at all. Coherent errors survive because nothing looks wrong, so the check must be more disciplined.

Can I trust confidence scores to find transcription errors?

Confidence scores are useful for triage but should never let you skip a reading. Some engines expose per-character or per-word values — Tesseract returns character-level confidences, and Transkribus surfaces them in its validation interface — and they point you at the tokens the model was least sure of. However, they are model posteriors, not calibrated probabilities of correctness. No comprehensive study has established that confidence scores from the major engines are calibrated against actual correctness on difficult historical hands, so treat a high-confidence reading as a place you can look faster, never one you can skip. General LLM transcription typically exposes no per-token confidence at all, removing even this rough signal.

What is double-keying and when should I use it?

Double-keying, also called four-eyes verification, is the discipline standard for establishing ground truth: two transcribers independently produce the same text, and an adjudicator reconciles the differences, with the adjudicated text becoming the reference. It means independent transcription, not one person checking another's work. Use it for high-value material — a corpus you will publish or a reference edition — or when you need to measure a model's real error rate on your own material rather than trusting its headline figure. Ground-truth quality dominates quantity: roughly twenty carefully corrected pages outperform a hundred pages carrying errors in the reference.