Historical Language Normalization: Keep the Original Reading Intact

Historical language normalization — replacing archaic spellings, letterforms, and abbreviations with modern equivalents — makes manuscript text searchable and analyzable. It also erases evidence when it overwrites the source instead of sitting beside it. This article sets out the layered practice that scholarly editing and TEI already specify, and explains what changes now that normalization increasingly happens inside a machine transcription pipeline, before any historian reads the page.

Historical language normalization is the deliberate replacement of historical forms — spellings, letterforms, punctuation, word division, abbreviations — with regularized or modern equivalents. It is useful and often necessary: normalized text searches, tokenizes, lemmatizes, and tags far better than diplomatic text does. The risk is not the normalization itself but the loss of what it replaced. The defensible practice, encoded in the TEI Guidelines and required by documentary-editing standards, is layering rather than substitution: keep a faithful transcription of what is on the page, add the normalized reading beside it as a separate documented layer, and never let an untraceable modernization become the only surviving witness to the document.

That principle sounds obvious when stated. In practice it is violated constantly, and increasingly by machines rather than editors — which is what makes it worth working through carefully.

Two disciplines, one word

The term arrives from two directions, and the collision causes real confusion.

In scholarly editing, normalization describes a transcription's position on a fidelity spectrum. M. J. Driscoll's account of levels of transcription sets the poles: at one end, strictly diplomatic transcription, "in which every feature which may reasonably be reproduced in print is retained" — often so close to the original as to be "all but unreadable for those unfamiliar with early palaeographical or typographical conventions." At the other, fully modernized transcription, where the substantives survive but everything else is brought up to date, "in some cases to such an extent as to make it questionable whether they are to be regarded as transcriptions at all." Between them sits the large middle ground where most working historians actually operate, and where, as Driscoll notes, practice varies greatly: it is common to normalize some features while retaining others.

In natural language processing, normalization means mapping a historical token to a modern equivalent so that variants can be searched, compared, tagged, or lemmatized. Tools like VARD and MorphAdorner exist precisely for this, and the work on historical spelling normalization in computational linguistics treats it as a well-defined engineering problem: reduce surface variance so downstream models perform.

The two usages overlap in the act — changing surface form — but diverge entirely in accountability. The editor is answerable for representing a document and declaring every intervention. The NLP pipeline is answerable for a metric, and may treat the normalized form as the target output, with the original discarded once it has served its purpose as input. When a research workflow imports NLP-style normalization without editorial-style accountability, that is where source integrity quietly disappears.

A vocabulary note

The distinctions matter when you write a method note. Regularization is usually narrower and more systematic. Modernization is broader. Silent emendation is an unmarked correction. Abbreviation expansion turns a sign or shortened form into supplied words and therefore adds editorial information that was not on the page. These are not synonyms, and a transcription policy that treats them as one thing is a policy you cannot defend under review.

One more clarification, since the material at issue is often described loosely: this is about vernacular documents written in the Latin alphabet — English wills, French notarial acts, German parish books, Dutch registers, Spanish and Italian correspondence — not about the Latin language alone. The marks and conventions travel across the languages that share the alphabet.

What normalization actually deletes

The standard defence of modernizing is that historical spelling is noise. It is not. Consider what disappears with each supposedly cosmetic change.

Long s (ſ). Its presence, distribution, and interaction with ligatures is evidence about compositorial practice and fount. It is also the single most consequential character in machine transcription of early-modern print. In Hill and Hengchen's measured comparison of ECCO's OCR corpus against ECCO-TCP — over 180,000 titles, some 200,000 volumes, more than 32 million pages — the only letter with a statistically measurable negative relationship to OCR correctness was s (p<0.001), and ligatures containing long s, along with ct, ff, and ffl, were significantly more likely to occur in incorrectly recognized words. That is an eighteenth-century English finding for a particular corpus, not a universal error rate, but the mechanism generalizes: the letterform scholars are most tempted to normalize away is also the one machines most often get wrong. Once it has been silently rendered as f, you cannot tell an OCR failure from a genuine reading without going back to the image.

u/v and i/j. Interchange follows positional convention, not modern phonemic logic. Normalize vnderstoode to understood and you have made a claim about orthographic practice that the page itself does not make — one of the very pairs the TEI Guidelines use to illustrate the original/regularized distinction.

Thorn and "ye" forms. Regularizing ye to the erases a paleographic fact about letterform survival and, incidentally, makes any later argument about scribal habit unverifiable.

Ligatures (æ, œ, ct, ff, ffl). Bear on printing technology, dating, and — in manuscript — on the hand.

Macron, titulus, and scribal abbreviation. These are the sharpest case. A mark standing for an omitted nasal is not a spelling variant; it is a sign whose expansion is an inference. Resolve it silently and you have converted a piece of evidence into an editorial conclusion with nothing to mark the seam.

Variant spelling generally. An odd form may be genuine orthography, dialect, a writer's self-correction, a clerk's habit, or evidence about the artifact. It may also be a mistake. Distinguishing those cases is the historian's work; a normalizer cannot do it, and a normalizer that has overwritten the original has taken the evidence away from whoever tries.

None of this argues against normalization as a layer. Spelling-aware access to corpora like EEBO-TCP and EarlyPrint is demonstrably valuable; a search for favour that misses favor is a worse research instrument than one that catches both. The argument is against normalization as a replacement.

The layered model, concretely

The mechanism already exists and is well specified. TEI's `<choice>` groups alternative encodings for the same point in a text. Inside it, `<orig>` holds "a reading which is marked as following the original, rather than being normalized or corrected," and `<reg>` holds "a reading which has been regularized or normalized in some sense." For abbreviations, `<abbr>` and `<expan>` do the same job: the sign as written, and its expansion, both present, both distinguishable.

The consequence is reversibility. A reader can render the diplomatic text or the regularized text from the same file; a search index can be built on the normalized layer while citations resolve to the original; and an argument that depends on a specific reading can be checked without returning to the archive. If you expand abbreviations silently instead, the TEI Guidelines are explicit that the practice must be documented in the `editorialDecl`, either via a `<normalization>` element or in prose.

Documentary-editing standards run parallel. The Association for Documentary Editing's Minimum Standards for Electronic Editions states that the editor's primary function is "to provide accurate documents to the reader," and that "clear distinctions must be made between editorial insertions and the original text." The General Descriptive Guide makes the related point that printed transcription silently incorporates editorial judgment, and that arbitrary modern punctuation or spelling normalization loses information — adding, usefully, that eighteenth-century Spanish, French, and Dutch differ from their modern forms exactly as English does, so a transcriber should know the era's usage, not just the language.

What the evidence does not support is a claim of settled field consensus on where the lines fall. Editors still differ on acceptable silent regularization, on punctuation, on how much abbreviation to expand, and on how much visual detail to encode. Comparative measurement of information loss across languages, genres, periods, and hands is uneven — the strongest quantitative evidence available concerns English eighteenth-century print. So the honest position is narrower than "everyone agrees": the architecture of source-plus-layer is well founded, while the specific policy remains yours to set and to declare.

That is a manageable obligation. Three lines in a method note — what you preserved, what you regularized, and where the untouched transcription lives — discharge most of it. If you are working out those conventions for the first time, the practical decisions are the same ones covered in a guide to transcription conventions and marking uncertainty, and the boundary between expanding an abbreviation and interpreting one is worth thinking through against the repertoire of scribal shorthand itself.

Where machine transcription enters the problem

Historically, normalization was an editorial act by a named person who could be asked why. Now it frequently happens upstream, inside a recognition pipeline, before any human has read the page. This is the shift that makes an old methodological argument newly pressing, and it belongs to the broader question of how computation can support historical scholarship without quietly rewriting its sources.

Three distinct things happen inside machine pipelines, and they need separating.

Recognition error

Historical type and hands break systems trained on modern letterforms: long s read as f, ligatures split or dropped, blackletter and Fraktur mishandled, uneven inking and show-through read as character evidence. These are honest failures. They are also visible — a garbled string announces itself.

Post-correction normalization

A downstream language model turns an uncertain character sequence into a fluent, modern-looking word. This is the failure that matters most here, because it destroys the evidence of its own occurrence. Work on OCR error post-correction with LLMs in historical documents is aptly subtitled "No Free Lunches": correction stages trade one error class for another. The framing deserves care — the research does not establish that every engine normalizes by default. Behaviour is configuration-, language-, and model-dependent. What it establishes is that post-correction is a risk you must inspect and document rather than assume away.

Convention learning

Where a project trains its own model, the conventions become the model's. Transkribus's data-preparation guidance is direct about this: whether the recognition model produces a diplomatic transcription or normalizes words to modern orthography "depends on you," and with consistent ground truth at sufficient scale a network can learn to separate combined words, normalize historical spelling, and resolve abbreviations. That is genuine capability. It is also a decision baked into weights, which makes it invisible in the output and difficult to reverse later. Whichever route you take, the choice about long s, i/j, ß, and abbreviation expansion must be made once, consistently, and written down.

The generative case

A 2025 evaluation of 12 multimodal LLMs on historical document OCR found that some models outperformed conventional OCR on the material tested while exhibiting "over-historicization" — inserting archaic forms the page does not contain. Read that carefully rather than as a headline: it is a single scoped evaluation, prompt- and dataset-sensitive, on a fast-moving class of systems, and it documents a fresh failure mode rather than a solved problem. A model that invents a plausible archaism is doing the same epistemic damage as one that invents a plausible modernism. In both directions the output is fluent, and fluency is not image-grounded evidence.

The practical asymmetry is what should govern your workflow. A wrong character is recoverable: you see it, you check the image, you fix it. A plausible fabricated word is not, because nothing in the text signals that it needs checking. This is the substance of why fluent LLM transcription errors are harder to catch than garbled OCR, and it is the reason a base transcription's job is to be faithful, not readable.

Keeping the layers separate in a working transcription

This becomes a tooling question rather than a philosophical one at the point where you need to read several hundred pages and still be able to defend a reading in a footnote three years later.

Leo's model, ATR-1, is built for that requirement. It transcribes what is on the page — preserving archaic orthography, the long s, strikethroughs, additions, margin notes, and editorial expansions marked as such — rather than smoothing a difficult page into modern, plausible prose. It reads Latin-script material, handwritten and printed, whatever the language written in that alphabet: English, French, German, Dutch, Spanish, Italian, Latin among them. It works out of the box, with no per-corpus model training, which also means no convention decision is silently compiled into a model you inherit.

The architectural point matters more than the model, though. Normalization in Leo is a separate operation, not a setting on the transcription. The base transcription stays as recognized; Transformations — Modernize, Correct, Translate — write their output to a new tab, leaving the original untouched and side by side with the page image. That is TEI's `<orig>`/`<reg>` logic expressed as workflow: the source reading and the regularized reading coexist, and either can be exported, including to TEI XML, where the distinction survives into your edition. If you want a normalized layer for searching and a diplomatic layer for citation, you do not have to choose.

On accuracy, one figure is worth having rather than several. On a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library at ATR-1's release, Leo scored roughly 5% character error rate against about 13% for Transkribus's Text Titan I, 23.3% for Claude Opus, 24.8% for Gemini 2.5 Pro, and 56.7% for GPT-4.1 — 61% fewer errors than the next-best model, with the full comparison data published here. That is one corpus, one language, one point in time. It says nothing about your notarial hands, and it does not remove the verification step. Errors do get through; the design intent is that they be the recoverable kind, checked against the image displayed beside the text.

The habit that outlasts the tools

Every generation of transcription technology has offered the same bargain: a more readable text in exchange for a less recoverable one. Printed editions took it, silently regularizing punctuation and spelling in ways that have kept editors busy re-collating manuscripts ever since. Mass digitization took it again, and a great deal of eighteenth-century scholarship now rests on OCR whose long-s errors are indistinguishable from historical orthography. The current generation of language models offers the same bargain in more fluent packaging.

The discipline that survives all of it is small and unglamorous. Keep the image. Keep one transcription that answers only to the page. Put every intervention — expansion, regularization, correction, modernization — in a layer you can name, date, and remove. Write down the policy, even briefly, even for a project only you will use. Then normalize as aggressively as your analysis requires, because a normalized layer you can peel back costs you nothing.

What you are protecting is not the transcription. It is the possibility that someone — a reviewer, a successor, yourself in five years with a different question — can go back and see what the document actually said.

Frequently Asked Questions

What is historical language normalization in manuscript transcription?

Historical language normalization is the deliberate replacement of historical forms — spellings, letterforms, punctuation, word division, abbreviations — with regularized or modern equivalents. It makes text easier to search, tokenize, lemmatize and tag, which is why it is often necessary. The defensible practice is layering rather than substitution: keep a faithful transcription of what is on the page, add the normalized reading beside it as a separate documented layer, and never let an untraceable modernization become the only surviving witness to the document. Normalize as aggressively as your analysis requires, provided the layer can be peeled back.

What is the difference between normalization in scholarly editing and in NLP?

In scholarly editing, normalization describes where a transcription sits on a fidelity spectrum, from strictly diplomatic transcription — retaining every reproducible feature — to fully modernized text. In natural language processing, it means mapping a historical token to a modern equivalent so variants can be searched, compared, tagged or lemmatized. The act overlaps; the accountability does not. An editor is answerable for representing a document and declaring every intervention. An NLP pipeline is answerable for a metric, and may treat the normalized form as the output with the original discarded. Importing the second without the first is where source integrity disappears.

How does TEI encode original and normalized readings together?

TEI's `<choice>` element groups alternative encodings for the same point in a text. Inside it, `<orig>` holds a reading marked as following the original rather than being normalized or corrected, and `<reg>` holds a reading that has been regularized or normalized. For abbreviations, `<abbr>` and `<expan>` do the same job: the sign as written and its expansion, both present and distinguishable. The result is reversibility — diplomatic or regularized text can be rendered from the same file, a search index can be built on the normalized layer while citations resolve to the original. Silent expansion must instead be documented in the `editorialDecl`.

Why is long s (ſ) a problem for OCR of early modern print?

Long s is both the letterform scholars are most tempted to normalize away and the one machines most often misread, usually as f. In a measured comparison of ECCO's OCR corpus against ECCO-TCP — over 180,000 titles and more than 32 million pages — s was the only letter with a statistically measurable negative relationship to OCR correctness, and ligatures containing long s, along with ct, ff and ffl, were significantly more likely to occur in incorrectly recognized words. That is one eighteenth-century English corpus, not a universal error rate. Once long s has been silently rendered as f, an OCR failure is indistinguishable from a genuine reading without the image.

Do AI transcription models normalize historical spelling automatically?

It depends on the system and its configuration. Three separate things happen in machine pipelines: recognition error, which is visible because garbled strings announce themselves; post-correction normalization, where a language model turns an uncertain character sequence into a fluent modern word and destroys the evidence of its own occurrence; and convention learning, where a project's ground truth bakes normalization decisions into model weights. A 2025 evaluation of twelve multimodal LLMs also documented "over-historicization" — models inserting archaic forms the page does not contain. Post-correction is a risk to inspect and document rather than assume away.