Historical Text Translation AI: What It Does Well, Where It Fails, and How to Use It Defensibly

This is a methods guide to historical text translation AI for researchers who work from manuscript sources and publish from them. It sets out why transcription, normalisation, and translation must stay separate operations, what the published evidence actually supports, and how to keep a translated passage answerable to the page it came from.

AI-assisted translation of transcribed historical texts is most defensible when it is treated as three separate operations rather than one: transcription of the image into source-language text, optional and reversible normalisation of spelling, and translation into a modern language. The source-language transcription is the citable base. The translation is a derived layer that must be verified against it, not substituted for it. Current models — general large language models in particular — produce genuinely useful draft translations of early modern vernacular material, but the published evidence is small, corpus-specific, and consistently identifies the same weak points: specialist terminology, proper names, archaic morphology, and code-switching. Those are exactly the places a historian's argument tends to rest.

This is written for researchers who already have transcriptions, or are about to, and want to move into a language they read less comfortably than the one they publish in. It sits within a wider set of questions about what AI can and cannot responsibly do in humanities scholarship.

Three operations, one page

A single seventeenth-century letter can invite four distinct interventions, and conflating them is the most common source of unusable output.

Transcription

Transcription renders the marks on the image into text in the language of the source. It is a reading of the page, and it is where errors enter the pipeline that nothing downstream can detect.

Normalisation

Normalisation regularises spelling and forms — ſ to s, u/v interchange, variant orthography to a modern standard. It is neither transcription nor translation. It changes the text without changing the language, and it should be reversible and labelled as its own layer.

Translation

Translation renders the source-language text into a modern language. It presupposes that the previous two steps are correct, and it has no mechanism for noticing when they are not.

Verification

Verification is the historian's step, and it is not optional. Post-editing literature treats human revision as a defined stage after machine translation, not as a quality nicety.

The reason to keep these apart is practical, not pedantic. If translation and transcription happen in one undifferentiated pass — the way they do when you paste a manuscript image into a chatbot and ask for the English — you lose the ability to say which layer introduced an error. You also lose the thing you actually cite. This is the substance behind why transcription and translation are two separate jobs, and why the search for a single tool that does both at once usually ends in a text no one can defend.

What the evidence actually supports

Modern machine translation sets a demanding contemporary baseline. The NLLB project reported a 44% relative BLEU improvement over the previous state of the art across more than 40,000 translation directions on the human-translated FLORES-200 benchmark. But that baseline is built on contemporary text, and the WMT shared tasks — which combine automatic metrics with human evaluation across news, speech, literary and educational domains — conclude plainly that human evaluation remains the final judge, noting sentence-level metric limitations, noisy annotations, and metric bias along the way. If that is the caution attached to modern languages with abundant training data, historical material warrants more.

The historical evidence is promising and thin. Two studies are worth knowing in detail.

Early New High German

Volk et al. (2024) tested GPT-4 and Gemini on 16th-century letters translated into modern German, using a test set of ten letters and 201 sentences. GPT-4 reached a BLEU score of 28.2, with Gemini averaging 26.8 and a two-step DeepL pipeline averaging 16.7; on passages code-switching between Latin and ENH-German rendered into English, GPT-4 scored BLEU 23.2 / chrF 46.3 and Gemini BLEU 25.4 / chrF 48.2. The authors also report finding no other literature on machine translation from ENH-German into modern languages, and believe no dedicated system for it exists. That is a real finding on a genuinely difficult source — and it is ten letters. It tells you general models can do useful work on the text of a 500-year-old vernacular hand. It does not tell you they will do so on your notarial series.

Early Modern Dutch

Work on 1,281 aligned Early Modern Dutch court-testimony pairs found that Unsloth fine-tuning improved a Mistral model's average automatic scores by 8–10 points, while ORPO tuning reduced mean scores by 0.20–0.35 and nearly tripled variance. The authors' stated limitations are the ones that matter to a legal historian: incomplete rendering of specialised terminology, and inconsistency.

Normalisation, separately

Work on 16th-century Middle French spelling normalisation measured a productivity increase of more than 20 tokens per minute when experts post-edited automatic normalisation instead of normalising from scratch. Note what that is and is not: a measured gain on normalisation, not on translation into a modern language.

Studies on Old English and on Latin show the same pattern — capability mixed with inconsistency. We are not aware of a comparable, independently human-verified benchmark set for historical Spanish or Italian. The honest summary: for the major Latin-script European vernaculars, historical machine translation is an active research area with encouraging small-scale results and no agreed diachronic test sets, no standard treatment of orthographic variants, and no consensus on how to score preservation of historical sense, legal force, or uncertainty.

Where historical translation breaks

The failure modes are specific enough to check for deliberately.

Obsolete senses rendered in modern meaning

A word that survives into the modern language with a shifted sense is the hardest case, because the output reads correctly. Conversation, industry, prevent, husbandry — a model trained overwhelmingly on modern text has every incentive to give you the modern reading.

Legal and technical terminology

Domain-specific vocabulary is where neural systems have long underperformed, and terminology error studies in the legal domain confirm the pattern in domain-specific and under-resourced settings. Feoffment, jointure, escheat, demesne — a translation that smooths these into approximate modern equivalents has destroyed the thing you were reading the document for.

Proper names, dates, measures, currency

Names are not translatable and are frequently mangled anyway. Old Style / New Style dating, regnal years, local measures of grain or cloth, £ s. d. and its equivalents — these are silent-error territory.

Archaic syntax and long periodic sentences

Where the source is a 200-word sentence with three subordinate clauses, models will occasionally shorten it. Omission is the least visible error class in machine translation and the least reliably detected.

Code-switching

Early modern documents move between vernacular and Latin, sometimes mid-clause. This is where the ENH-German study's scores dropped, and it is common in wills, accounts, and court records.

Inherited transcription error

Downstream NLP research establishes consistent harm from OCR noise to tasks like named-entity recognition. What it does not provide is a universal multiplier telling you how much a character error costs at the translation stage. Treat the direction as established and the magnitude as unmeasured. Practically: a misread word does not announce itself in translation. It becomes a fluent, plausible, wrong sentence — which is why the fluency-versus-fidelity problem belongs at the front of your attention, not the back.

Scoring output you cannot check line by line

If you are comparing systems or reporting a method, you will meet the standard metrics. Know what each one measures.

BLEU

Papineni et al., 2002 defines modified word n-gram precision with a brevity penalty. In the original paper it correlated 0.99 with monolingual and 0.96 with bilingual human judgments on a 500-sentence Chinese–English corpus — but the absolute scores in that study were low by modern standards, and BLEU punishes legitimate variation in phrasing.

chrF

Popović, 2015 uses character n-gram precision and recall, which makes it more tolerant of morphological variation — useful for historical material. On WMT14 system-level evaluation, Pearson correlations were chrF 0.805, chrF3 0.857, BLEU 0.845 and TER 0.814; segment-level chrF correlations were considerably weaker (fr-en 0.402, de-en 0.318).

TER

Snover et al., 2006 counts the minimum normalised edits, including shifts, needed to match a reference — the closest proxy for post-editing effort.

COMET

Rei et al., 2020 is a learned evaluator that uses source, hypothesis and reference together.

All four require a reference translation. For most archival material you do not have one, which means these metrics are of limited use for your actual corpus and of real use only if you build a small gold set yourself.

MQM

The Multidimensional Quality Metrics framework is the alternative worth adopting: a human error taxonomy with categories such as accuracy, fluency and terminology, scored by severity. For a historian, an MQM-style tally — twenty sampled passages, errors classified and weighted — is more informative than any single number, and it is reportable in a methods note. It is also unsettled whether learned evaluators or LLM-as-judge approaches detect historically plausible but wrong meanings at all; that is precisely the error type they were not designed for.

A working pipeline

1. Retain the image. Everything else is a claim about it.
2. Produce a source-language transcription and verify it against the image, prioritising names, numbers, and other high-stakes tokens. This is the citable base.
3. If you normalise, do it as a separate, labelled, reversible layer — never in place. Standard transcription conventions for expansions and uncertainty apply here.
4. Translate from the verified transcription, not from the image.
5. Verify the translation against the source text, sampling systematically and reading every consequential passage in full.
6. Record what you did: model, date, prompt or setting, verification method, error tally.

Steps 2 and 5 are the ones grant timelines tend to compress. They are the ones that make the rest citable.

Keeping the layers apart in practice

The pipeline above is easy to describe and awkward to run across three tools and a folder of loose files. This is the part of the problem Leo is built for.

Leo transcribes what is on the page. That is the base layer, and translation is deliberately not part of it. Its transcription model, ATR-1, reads Latin-script material, handwritten and printed, from roughly the past 500 years: any language written in the Latin alphabet, with performance strongest in English and strong across French, German, Spanish, Italian, Dutch and Latin among others. Non-Latin scripts — Greek, Cyrillic, Hebrew, Arabic, Indic and East Asian writing systems — are outside its scope, which is worth stating plainly, since script, not language, is what actually determines coverage. The transcription preserves archaic orthography, abbreviation marks, strikethroughs and marginalia rather than smoothing them into modern prose, because a normalised base makes the normalisation layer unrecoverable.

Translation is a separate operation: the Translate Transformation, run on a finished transcription, writing its output to a new tab. The base transcription is untouched. So is the image, which sits beside the text while you check it. Correct, Modernize and Interpolate work the same way — each result is its own layer, so a modernised reading never quietly replaces the diplomatic one, and you can always see which text your argument actually rests on. Export to TEI XML, Word, PDF or HTML when you need the material outside the workspace. ExLatinis, a free public corpus of Latin works printed in Europe between 1450 and 1750, is built on exactly this pairing of faithful transcription plus a translation layer — one corpus among the many languages the model reads, not the boundary of them.

What Leo does not do is verify the translation for you. Nothing does.

Encoding, citing, disclosing

If you are publishing an edition, the TEI Guidelines on linking, segmentation and alignment already provide the machinery for representing a source text and its translation as aligned, corresponding structures — `corresp`, `link`, `linkGrp`. Use it. It encodes the relationship between layers rather than collapsing them, which is the same discipline the workflow enforces, expressed in markup.

For citation, the manuscript takes the reference; the machine-assisted stages belong in a method note covering platform, model, verification procedure and editorial decisions. The conventions for citing AI-assisted transcription in scholarly work extend naturally to translation. Publisher expectations are hardening: Wiley's author guidance on AI use is one example of a disclosure standard you should assume applies unless told otherwise. Standards are not yet uniform across publishers and editions. Disclose anyway.

The habit that matters

Machine translation of historical text is best understood the way an experienced researcher understands a nineteenth-century printed calendar of state papers: an aid to finding and gisting, produced by someone with less context than you, useful for deciding where to look and never adequate as the reading of record. The gains are real — a working draft in an hour instead of a week — and they are gains in access, which is the stage before scholarship, not a substitute for it.

What survives the technology is the habit of reading back. Take the translated passage that carries the weight of your argument, put it beside the source-language transcription, put both beside the image, and read all three. If the three agree, you have a source. If they do not, you have found something more interesting than a clean translation: a place where the document resists, which is usually where the history is.

Frequently Asked Questions

What is historical text translation AI and how accurate is it?

Historical text translation AI means using machine translation — usually general large language models — to render transcribed historical source text into a modern language. Accuracy is corpus-specific and the published evidence is thin. In a 2024 study of 16th-century letters into modern German, GPT-4 reached a BLEU score of 28.2, with Gemini at 26.8 and a two-step DeepL pipeline at 16.7, on a test set of ten letters and 201 sentences. Fine-tuning work on Early Modern Dutch court testimony improved automatic scores by 8–10 points but still reported incomplete rendering of specialised terminology. Useful drafts, verified readings — not the same thing.

Can AI translate old handwritten documents directly from an image?

It can produce something, but the output is hard to defend. When transcription and translation happen in one undifferentiated pass — pasting a manuscript image into a chatbot and asking for the English — you lose the ability to say which layer introduced an error, and you lose the source-language text you would actually cite. The defensible sequence is: transcribe the image into source-language text, verify that transcription against the image, then translate from the verified transcription. Leo works this way: ATR-1 transcribes the page, and the Translate Transformation runs on a finished transcription, writing to a new tab while the base text stays untouched.

Why should transcription, normalisation, and translation be kept as separate layers?

Because each one can fail differently, and only separation lets you locate the failure. Transcription reads the marks on the page; normalisation regularises spelling without changing the language; translation renders verified source text into a modern language and has no mechanism for noticing when the earlier steps were wrong. A misread word does not announce itself downstream — it becomes a fluent, plausible, wrong sentence. Keeping the layers apart, labelled and reversible, means a modernised reading never quietly replaces the diplomatic one, and you can always see which text your argument rests on.

What are the most common failure modes when AI translates early modern documents?

Six recur often enough to check for deliberately. Obsolete senses rendered in modern meaning — conversation, industry, prevent, husbandry — are hardest, because the output reads correctly. Legal and technical terminology such as feoffment, jointure, escheat gets smoothed into approximate modern equivalents. Proper names, Old Style / New Style dates, local measures and currency are silent-error territory. Long periodic sentences get shortened, and omission is the least visible error class. Code-switching between vernacular and Latin depresses scores and is common in wills, accounts and court records. And any transcription error is inherited invisibly.

Which translation quality metrics work for archival material without a reference translation?

None of the standard automatic metrics work well, because BLEU, chrF, TER and COMET all require a reference translation you usually do not have. They are useful only if you build a small gold set yourself. The practical alternative is an MQM-style human error taxonomy: sample twenty passages, classify errors by category — accuracy, fluency, terminology — and weight them by severity. That tally tells a historian more than any single number and is reportable in a methods note. It is also unsettled whether learned evaluators detect historically plausible but wrong meanings at all.