Fluent but Wrong: Understanding LLM Transcription Errors on Historical Documents

This article explains why LLM transcription errors on historical manuscripts are harder to catch than ordinary machine mistakes, what those errors look like on the page, and how to structure your work so fluent fabrications never survive into a finished transcription. If you rely on a general model to read a difficult hand, the risk is not garbled text. It is plausible text that reads correctly and is wrong: a modernised spelling, an abbreviation quietly written out, a passage that simply isn't there.

When a large language model transcribes a historical document, its mistakes do not look like mistakes. They read as clean, confident, grammatical prose, which is precisely what makes them dangerous. The most consequential LLM transcription errors are not garbled characters a reviewer would catch at a glance. They are plausible fabrications: a word the model expected to see, a spelling it quietly modernised, an archaic form it invented, a passage it smoothed into sense. A model built to read historical documents, when it stumbles, produces a visibly broken reading you can correct. A general LLM that stumbles produces a fluent error you may never notice.

That difference (detectability, not raw error count) is the whole of the problem. What follows is an account of why it happens, what the failure looks like on the page, and how to structure your work so fluent errors do not survive into your finished transcription.

Why fluent errors are the ones that hurt

Start with a distinction that most accuracy discussions skip. Two transcription systems can post the same character error rate and pose entirely different risks.

Character error rate (CER) measures surface edit distance against a reference: substitutions, deletions and insertions divided by the total characters in the reference. It is the standard metric in the field, and it is genuinely useful. But it tells you how many characters differ from the truth, not how catchable those differences are. A garbled OCR output announces itself: rn where the page reads m, a broken ligature, a dropped diacritic. Any reviewer glancing between image and transcription will stop on it. A fluent fabrication that scores an identical CER slips past the same reviewer, because nothing about it looks wrong.

This is why the danger ordering runs by fluency, not magnitude. The worst error is not the largest one. It is the one that reads correctly, scores competitively, and embeds a false reading into a dataset that later work will cite. This is the throughline of why keeping a human in the loop still matters: the tools that produce the most confident output are the ones that most need checking, and the checking is hardest exactly where the output is smoothest.

What produces the fluent error

To see why general LLMs fail this way, it helps to know what they are actually doing when they "read" a page.

A vision-language model couples a vision encoder to an autoregressive language model. The image is broken into patches. A Vision Transformer typically processes a 224×224 or 336×336 pixel image as a few hundred patches (patchification is the standard image tokenization approach). The language model then generates text one token at a time, optimizing next-token prediction against fluent natural-language targets. Two structural facts follow from this design, and both matter for historical material.

First, the image is downsampled. High-resolution detail is compressed away before the model ever reasons about it: the hairline that distinguishes one minim from two, the faint macron over a vowel, the difference between a long s and an f. General LLMs hallucinate on historical handwriting because they lean heavily on the text side and work from a downsampled image.

Second, the model is trained to produce fluent language. When the visual evidence is thin, its language prior fills the gap. It completes a partially obscured word, normalizes an archaic spelling, or inserts a word it expects from context. This is hallucination: fluent, plausible text not grounded in the input. OpenAI's own account frames it as a consequence of training and evaluation procedures that reward guessing over acknowledging uncertainty. A 2025 benchmark, KIE-HVQA, documented the multimodal models of the time, across the GPT-4o and Qwen2.5-VL classes, frequently misinterpreting or ignoring textual visual content.

There is also a specifically historical variant of the failure. Levchenko and colleagues evaluated twelve multimodal LLMs on eighteenth-century Russian Civil-font texts and named it over-historicization: the model inserts archaic characters and spellings that suit the genre and period but are not present on the page. The transcription looks more authentically old than the source. It is wrong in the most flattering possible direction, which is to say the direction least likely to trigger suspicion.

Quiet normalisation: the fluent error you won't notice

Over-historicization is the rarer direction. The common one runs the other way: a general model tends to give you the page as a modern reader would expect it, not as the scribe or printer set it down, and it does so without marking what it changed. Three forms recur:

• Spelling is modernised. Antient becomes ancient, vnto becomes unto, sonne becomes son. Each change is plausible, and each erases evidence about the writer, the date or the place.
• Abbreviations are expanded or dropped without a mark. A raised letter, a macron or a p with a stroke through its descender is written out in full, or simply left out, with nothing in the transcription to show that the page carried a sign at all.
• Parts of the page go missing without a word. On a dense page a general model can skip a column or stop early and still hand back a clean, complete-looking transcription.

These are not hypothetical. When we tested a small general model on historical printed pages for our own guidance, it silently modernised early-modern spelling, and on a dense, multi-column newspaper from 1854 it stopped after a small part of the page and reported a normal finish. The transcription it returned was tidy, grammatical, and missing most of the text. This was print, which is easier than handwriting, and the model was small. Larger general models do better, but published tests of the strongest of them, Gemini included, still find quiet normalisation among their commonest remaining errors.

Every one of these errors reads well, raises no flag, and destroys evidence. Normalising a text is a legitimate editorial step, but only as a declared layer beside the faithful reading, not in place of it. Our guide to historical language normalization sets out how editors and TEI keep the two apart. A general model doing it inside the transcription step, unasked and unrecorded, is the opposite of that practice.

The failure on the page

Consider what these mechanisms do to real material.

An early-modern English scribe writes yo^r: a superscript r standing for your. It is the same pattern as the raised-letter abbreviation oʳ for our. A faithful transcription preserves the abbreviation or expands it with a visible editorial marker: yo[u]r. A fluent model silently renders your, dropping the evidence that the page carried a contraction at all. The reading is not wrong in meaning. It is wrong as a record.

A page carries the long s (ſ). Conventional OCR built for modern type reads it as f, producing a visible garble. A general LLM often does something subtler and worse: it recognizes the word from context and "corrects" the spelling to its modern form, so the transcription reads sister where the page shows a form your own eye would have flagged. The correction erased the thing you needed to see.

A partially damaged word, whether from ink loss, show-through from the verso or a warped gutter, gives a model built to read the image a garbled or incomplete reading, and you know to go back to the image. A general LLM completes the word. It supplies the letters it expects. You get a whole, grammatical sentence with a hole in the evidence you cannot see, because it has been seamlessly filled.

These are not exotic edge cases. They are the ordinary texture of reading early modern paleography: the superscripts and brevigraphs of scribal abbreviation, the letterforms of secretary hand, the u/v and i/j interchange that early-modern convention treats as normal and a language model treats as error. Every one of them is an invitation for a fluent model to improve the page into something it was not. To check these things yourself, the free secretary hand course at Leo Academy teaches the letterforms and abbreviations from real scribes' hands.

Why "I'll just check it" is not enough

The instinctive reply is that a careful scholar will catch these things on review. The evidence on human oversight is not encouraging.

Automation bias is the tendency of human reviewers to over-trust plausible automated output and under-detect its errors, and it is one of the better-documented effects in the human-factors literature. A systematic review found it to be a fairly robust and generic effect across research fields. Strikingly, it also found that decision-making was in some settings more accurate without a decision aid than with a flawed one. The mechanism is exactly the fluency profile of LLM transcription: when the output is coherent and confident, the reviewer's own judgment gets sidelined.

There is an honest limit to note here. That research comes from decision-support contexts, chiefly clinical systems, and no controlled study has yet isolated the effect of fluency specifically on human proofreading of transcriptions. But the direction of the finding is consistent, and the reason is intuitive. You cannot proofread against a reference you do not have. When you are transcribing a difficult hand, the transcription often is your reference; a fluent error becomes the thing you check the image against, rather than the other way around. And you cannot proofread a line that is not there. The fabrication does not merely survive review. It can quietly steer it.

What LLMs are, and are not, good for

None of this makes general LLMs useless for historical work. The critique is targeted, not blanket.

On clean, legible text the general models perform strongly. In a 2025 study of the models then current, the 2024 GPT-4o-mini reached 1.71% CER on the IAM modern-handwriting database. On clean print you only need the text of, such as a 19th-century book or a typescript, even a small model is a sensible, cheap choice. And on a range of downstream tasks performed over an already-faithful transcription (translation, summarization, named-entity extraction, reconciling reading order) they are genuinely valuable. On difficult historical material the picture is mixed: the Levchenko study found LLMs well ahead of traditional OCR on one eighteenth-century corpus, a single-corpus result that still turns on verification. What no study supports is trusting a general LLM as the primary transcriber of a difficult hand and skipping the check.

The safe division of labour follows from that. Use a model built to read the page for the reading; reserve the general models for a second opinion and for analysis performed on top of a transcription you already trust. The difference between the two is the whole subject of ChatGPT's specific struggles with historical handwriting; Can AI read old handwriting? sets out which model suits which job.

In Leo, that division is one menu in the transcription window: ATR-1, Leo's own model, is the default, with the major OpenAI, Gemini and Claude models alongside. Run a general model as a second reading into its own tab and compare, or use a small model on clean, legible print for a fraction of a credit an image. Handwriting is not one of those jobs.

Building the workflow so fluent errors cannot hide

The practical answer is not to abandon machine transcription. It is to choose a model whose errors are the catchable kind, and to keep interpretation separate from the base record.

This is the design principle behind Leo's own model, ATR-1. It is a Transformer specially tuned to read images of historical documents, handwritten and printed. Where a general model downsamples the page, ATR-1 reads the image at high resolution, weighing the visual evidence against context rather than leaning on a language prior to guess. And it is trained to transcribe what is on the page rather than normalise it. It does not over-correct or silently modernise: the long s stays a long s, the superscript abbreviation is preserved rather than quietly written out, and the archaic spelling survives. It works zero-shot, so there is no training on your hand before you can read a page.

ATR-1 also watches for its own failures. Any text-generating model can break down on material unlike its training data; the classic sign is a line repeated over and over. ATR-1 checks its output for failure patterns like that: suspect output is hidden, the image is retried automatically, and if it still cannot succeed the job stops and the credit is refunded. A general model that stops partway down a page and reports a normal finish raises no flag of its own. ATR-1 still makes mistakes, but they tend to be the recoverable kind: a wrong letter or word you catch against the image displayed beside the text, not a fluent passage that was never on the page.

The second half of the principle matters as much as the first. If you want a language model's help to modernize, translate, or reconcile several readings, layer it on top of the faithful transcription, in its own tab (in Leo, a Transformation), leaving the base text untouched. That way the fluent output never overwrites the record. It sits beside it, clearly marked as interpretation, where you can compare it against the source and discard it if it drifts. Fluent errors are contained because they are kept out of the layer that claims to be the source.

That structure is worth insisting on whatever tools you use: a faithful base and a separate interpretive layer make fluent errors visible instead of letting them pass for the record. Our guide to verifying transcription accuracy covers how to check that base efficiently.

The habit worth keeping

The appeal of fluent output is that it seems to save you the hardest part of the work, and it does not. It relocates it. A transcription that reads well is not a transcription you have verified. It is a transcription that has made verification feel unnecessary, which is a different and more dangerous thing.

The discipline that protects your scholarship is old and unglamorous: read the transcription against the image, not the image against the transcription. Preserve what the page actually shows before you decide what it means. Treat the machine's confidence as a hypothesis, not a finding. The tools have improved, and they will keep improving. In Leo's case that happens as users' corrections feed back into ATR-1's training. But the judgment about what is really on the page stays with the reader who knows the hand. That is not a limitation to engineer away. It is the work.

Frequently Asked Questions

What are LLM transcription errors on historical documents?

LLM transcription errors on historical documents are plausible fabrications rather than garbled text. Examples include a word the model expected to see, a spelling quietly modernised, an abbreviation silently expanded, or a passage it smoothed into sense. Because a general large language model is trained to produce fluent language, it fills thin visual evidence with its language prior instead of reading what is on the page. The result reads as clean, confident, grammatical prose, which is exactly what makes it dangerous. Unlike a broken OCR reading that announces itself, a fluent error scores competitively and slips past review, embedding a false reading into a dataset that later work will cite.

Why are fluent LLM errors harder to catch than OCR mistakes?

Because they read correctly, nothing about them signals a problem. Two systems can post the same character error rate and pose entirely different risks. A garbled OCR output announces itself (rn for m, a broken ligature, a dropped diacritic), and any reviewer glancing between image and transcription will stop on it. A fluent fabrication with an identical score does not. The danger runs by fluency, not magnitude. The worst error is not the largest one. It is the one that looks right, which is why detectability, not raw error count, is the real problem.

What is over-historicization in LLM transcription?

Over-historicization is a failure where a language model inserts archaic characters and spellings that suit the genre and period but are not present on the page. Levchenko and colleagues named it while evaluating twelve multimodal LLMs on eighteenth-century Russian Civil-font texts. The transcription ends up looking more authentically old than the source itself. This is wrong in the most flattering direction, the one least likely to trigger suspicion, because it matches what a reader expects an old document to look like. It is a specifically historical variant of hallucination, and it makes faithful proofreading against the image essential rather than optional.

Can I just proofread an LLM transcription to catch the errors?

Proofreading alone is not a reliable safeguard. Automation bias is the tendency to over-trust plausible automated output and under-detect its errors. It is one of the better-documented effects in the human-factors literature, found to be robust and generic across research fields; in some settings decision-making was more accurate without a flawed aid than with it. The mechanism matches the fluency profile of LLM transcription. You cannot proofread against a reference you do not have. When you are transcribing a difficult hand, the transcription often becomes your reference, so a fluent error becomes the thing you check the image against rather than the other way around.

Are LLMs ever useful for historical documents?

Yes. The critique is targeted, not blanket. On clean, legible text general models perform strongly: in a 2025 study, the 2024 GPT-4o-mini reached 1.71% CER on the IAM modern-handwriting database, and on clean print such as a 19th-century book even a small model is a cheap, sensible choice. They are also valuable for tasks performed on an already-faithful transcription: translation, summarization, named-entity extraction. The problem is trusting a general LLM as the primary transcriber of a difficult hand and skipping the check. Use a model built for historical documents, such as Leo's ATR-1, for the reading, and keep general models for a second opinion and for analysis.

Do AI models modernise spelling when they transcribe old documents?

General AI models often do, and quietly. They tend to return the most probable modern form, so vnto becomes unto and antient becomes ancient, and raised-letter abbreviations are written out with no mark. The newest models, Gemini included, still tend to do this, and on dense pages they can skip text without saying so. Leo's ATR-1 is built the other way: trained to transcribe what is on the page rather than normalise it, so archaic spelling, the long s and the scribe's abbreviations survive. A modernised version, if you want one, is a separate Transformation in its own tab.