How Handwritten Text Recognition Works

This article explains how HTR works, stage by stage — from the image on the scanner to the text you can search. It names the architectures that do the reading, marks the places where the pipeline still breaks, and shows why the accuracy you get depends far more on how closely your material matches a model's training than on any headline benchmark. If you work with manuscripts, knowing where a pipeline fails is what separates a usable transcription from a plausible-looking one.

Handwritten text recognition (HTR) is a computer-vision task that turns an image of handwriting into machine-readable text. It works as a pipeline: acquire the image, prepare it, find the text lines on the page, run a neural network that reads each line as a continuous sequence rather than letter by letter, decode that sequence into characters, and export the result. The reading step is the heart of it, but every stage carries its own error budget, and the accuracy you get in practice depends far more on how closely your material resembles what the model was trained on than on any headline benchmark number.

The rest of this article walks through each stage, names the architectures that do the work, and marks the places where the process still breaks.

HTR Is Not OCR — and the Difference Sets Up Everything Else

Optical character recognition (OCR) and HTR are treated as separate subfields for a reason. OCR exploits the regularity of printed type: fixed letterforms, uniform spacing, one reproducible shape per glyph. That regularity lets an OCR engine segment a line into characters and classify each one against a known set of shapes. Handwriting offers no such regularity. Strokes connect, letterforms vary from writer to writer and even within a single hand, and there is often no clean boundary between one character and the next. A system built to slice a line into discrete glyphs has nothing stable to slice.

The boundary blurs at one point worth naming, because it matters for historical work: early modern print. Fraktur and blackletter type, the long s (ſ) that generic OCR reads as f, ligatures, and typographic abbreviation push printed material toward the same shape-variation problems that make handwriting hard. That is why dedicated historical tooling gets deployed on old books as well as manuscripts. If you want the full treatment of where the line falls, we cover it separately in HTR vs OCR; here it is enough to say that HTR was built to cope with variation OCR assumes away, and the two now overlap on difficult historical print. This is one strand of the broader story we tell across our handwritten text recognition hub.

The Pipeline, Stage by Stage

There is broad consensus on the ordering of an HTR pipeline. The stages below are the ones you will find in nearly every serious system, open-source or commercial.

Image Acquisition

Everything downstream is bounded by what the camera or scanner captured. Resolution and colour depth set a ceiling on accuracy that no later stage can lift: a model cannot recover a stroke the sensor never recorded. High-resolution capture matters more than it looks, because the fine distinctions between similar letters — a worn e and c, a minim count in m versus ni — live in exactly the detail that low resolution throws away. This is also why downsampling quietly costs you accuracy, a point that returns when we reach general-purpose models.

Preprocessing

Historically this meant deskewing, denoising, and binarizing — reducing the image to black and white before recognition. The intuition was that a cleaner image is an easier image. Modern neural models have largely overturned it. Coquenet and colleagues, in their 2024 study of best practices for HTR systems, show that aggressive binarization and aspect-ratio-breaking resizing actively hurt deep models; their recommendations include preserving aspect ratio when resizing and skipping binarization altogether. Preprocessing is no longer the dominant source of error, and more of it is not better. A deep model reads grayscale evidence — the pressure of a stroke, the fade of iron-gall ink — that binarization would have discarded.

Layout Analysis and Segmentation

Before a model can read a line, the system has to find the lines: detect text regions, establish reading order, and locate baselines. On a clean single-column page this is nearly solved. On historical material it is not. Marginalia, multi-column layouts, tables, drop caps, and interlinear annotations remain genuinely hard, and the 2025 survey of text-line segmentation and baseline detection by Rabaev and colleagues identifies this as one of the main unresolved areas in the field. Layout analysis carries its own error budget, separate from recognition. A perfect reading model still produces garbled output if the reading order was wrong.

Text Recognition

This is the stage most people mean when they say "HTR." A feature extractor turns the line image into a sequence of visual features; a sequence model turns those features into a sequence of characters. The architecture that does this has changed several times, and the generations are worth knowing because they are still all in production somewhere.

The crucial thing to understand is what this stage does not do: it does not read letter by letter.

Decoding and Language Modeling

The recognition model emits, in effect, a probability distribution over characters. Decoding turns that into final text — greedy selection of the most likely character at each step, or beam search, sometimes with a lexicon or a language model layered on to bias the output toward plausible words.

Here lies a trade-off that matters enormously for historical work. A language model raises surface accuracy on common words, but it does so by pulling the output toward what is common — which is precisely wrong when the source contains rare spellings, dialect forms, abbreviations, and unusual names. Coquenet and colleagues document this trade-off directly. It is the same mechanism, at a smaller scale, as the one that makes general LLMs risky for transcription: correction and normalization improve the average case and quietly corrupt the exceptional one. For a family historian, the "exceptional one" is often the surname you were searching for.

Post-processing and Export

Finally the text is serialized into a format that preserves layout and provenance. The common standards are PAGE XML and ALTO XML for layout-plus-text, and TEI XML for scholarly encoding downstream. This is not a trivial stage: a transcription that cannot be exported in a form your edition, database, or catalogue can consume is a transcription you will end up re-keying.

Why the Model Does Not Read Letter by Letter

This is the most common misconception about HTR, and correcting it explains why modern systems work at all.

Older systems did try to segment characters before classifying them, and pre-segmentation failure dominated their error budgets — if you cut the line in the wrong place, every downstream classification inherited the mistake. The advance was learning to skip segmentation entirely. Alex Graves and colleagues introduced Connectionist Temporal Classification (CTC) in 2006 precisely to label unsegmented sequence data: the model consumes a whole line and emits a character sequence directly, using a blank token and dynamic programming to align input to output without ever being told where one character ends and the next begins.

Every dominant architecture since works this way. It reads the line as a whole, using context to resolve locally ambiguous shapes — much as a human paleographer infers an unclear letter from the word around it rather than staring at the glyph in isolation.

The Four Generations of Recognition Models

The recognition stage has passed through several architectural generations, and understanding them clarifies the choices behind any tool you might use.

Generation 1 — Hidden Markov Models

Handcrafted features (stroke slopes, loops, projection profiles) fed into HMMs. The pre-deep-learning baseline, now historical.

Generation 2 — CNN + BLSTM + CTC

Deep convolutional features feed a bidirectional LSTM trained with the CTC loss. This has been the workhorse since roughly 2013 and remains production-grade. It is the architecture behind Transkribus and PyLaia. Notably, Coquenet and colleagues show that with three careful fixes — aspect-ratio preservation, column-wise max-pooling, and an auxiliary CTC loss — this older design stays competitive with much larger Transformers on standard benchmarks. Newer is not automatically better.

Generation 3 — Attention Encoder–Decoder

A CNN or RNN encoder paired with an attention-based decoder. Flexible, but data-hungry.

Generation 4 — Transformer / TrOCR

Pure attention: an image-Transformer encoder and a text-Transformer decoder, pretrained on large synthetic and unlabeled corpora. Microsoft's TrOCR is the reference implementation, and this generation is the research state of the art on benchmarks like IAM.

Generation 5 (Emerging) — Multimodal Foundation Models

GPT-4o, Gemini, and similar systems applied zero-shot to handwriting. Competitive on clean English material — and unreliable on historical scripts, for reasons the next section makes concrete.

What the Benchmark Numbers Actually Tell You

You will see impressive figures quoted for HTR. TrOCR reports 3.42% CER on the IAM test set. The best system in the ICDAR 2017 competition on the Bentham dataset hit 1.91% CER. A 2025 benchmark of large language models for HTR by Crosilla and colleagues reports an average character error rate of around 2.95% across English, German, and French handwriting.

Character error rate (CER) is the standard measure: the edit distance between the model's output and a correct reference, divided by the reference length. Lower is better. It is a good metric. The problem is not the metric but the transfer.

Benchmark CER does not survive contact with a real archive. A model scoring single-digit CER on IAM can score 30% or more on out-of-distribution material — a different hand, a different period, worse image quality, another language. The generalization study frames this gap as the central unsolved deployment problem in HTR. Script, period, image quality, language, and individual writer all move the number, and almost no vendor publishes performance on representative samples of actual archive scans. When you read a single accuracy figure, the right question is never "how low is it?" but "on what material, and how close is that to mine?"

This is also why the claim that HTR is "solved" is misleading. Headlines about a general model conquering handwriting rest on clean English material; the same model class can fail on historical manuscripts. A 2024 study found Gemini 1.5 Pro underperforming a fine-tuned TrOCR on real historical documents — one corpus, and still requiring verification, but a useful corrective to the headline. HTR for difficult material is not solved.

The Two Operating Models: Train-Your-Own Versus Zero-Shot

Faced with the recognition stage, tools split into two camps.

User-trained engines — Transkribus, Kraken, PyLaia — ask you to supply ground-truth transcriptions for a representative sample of pages, then return a model tuned to your collection. On a genuinely idiosyncratic hand this remains the most reliable path, and it is the honest recommendation when your material is far from anything a general model has seen. The cost is real: transcribing ground truth is the very bottleneck you were trying to relieve, and the effort recurs with each new collection.

Zero-shot models — TrOCR out of the box, and the general multimodal systems — require no training data. Their accuracy depends entirely on how close your material sits to their training distribution. For clean modern handwriting that can be excellent. For historical hands it is a gamble, and it is where the failure mode turns dangerous rather than merely inconvenient.

That failure mode is worth stating plainly, because it is where the choice of tool stops being academic. On a safety spectrum, image-first recognition engines tend to err on individual characters — a misread letter, a dropped mark — errors that are visible and recoverable against the page. General LLMs sit at the other end: because they lean heavily on textual plausibility and downsample the high-resolution image that carries the real evidence, they err by producing fluent, confident, well-formed text that is simply wrong. GPT-4 Vision has been observed generating plausible but incorrect transcriptions of historical handwriting. A character error announces itself. A fluent fabrication does not — and in a scholarly edition or a family tree, the fabrication you cannot see is the one that does the damage.

Where a Purpose-Built Model Fits

This is the stage where the choice of tool matters most, and it is worth being direct about it. A model trained specifically on images of historical documents occupies useful ground between the two camps: zero-shot, so there is no per-collection training step, but built for manuscript hands rather than for clean modern text. This is where Leo's ATR-1 sits. It is a Transformer, like TrOCR and the general models, but fine-tuned on historical material and trained to transcribe what is on the page rather than to normalize it — so the long s stays a long s, an archaic spelling survives, a macron standing for an omitted nasal is not silently resolved into modern letters.

On a randomized 97-image sample of early modern English manuscripts from the Folger Shakespeare Library, ATR-1 recorded roughly 5% character error rate at release — about 61% fewer errors than the next-best system tested (Transkribus/Text Titan I at ~13%, Claude Opus ~23.3%, Gemini 2.5 Pro ~24.8%, GPT-4.1 ~56.7%), with the full comparison published here. The usual caveat applies with full force: that is one corpus, one script family, one period, and your material will move the number. It reads Latin-script material in any language written in that alphabet — English wills, French notarial records, Dutch registers, German parish books, and Latin among them — not any single language. Corrections users make feed back into training, so accuracy compounds across releases. It is not perfect, and no honest HTR is; the point is that its errors tend to be the recoverable kind, and it does not smooth the page into modern prose.

Reading the Pipeline Critically

The value in understanding the HTR pipeline is not academic. It tells you where to look when a transcription disappoints. A garbled paragraph in the middle of a clean page usually points to layout analysis — the reading order was wrong, not the reading. Consistently mangled rare words point to an over-eager language model or a post-correction step normalizing what it should have preserved. A whole document that reads fluently but diverges from the source on names and dates points to a model erring on plausibility rather than evidence, and that is the pattern to fear most, because it hides.

Match the model to the material. Evaluate on held-out pages from your own collection, never on a public benchmark. Treat any language-model or post-correction layer as something to inspect rather than trust. And read the output against the page — because the transcription that looks most confident is not always the one that tells you the truth about what the writer actually wrote.

Frequently Asked Questions

How does HTR work?

HTR works as a pipeline that turns an image of handwriting into machine-readable text. It acquires the image, prepares it, finds the text lines on the page, runs a neural network that reads each line as a continuous sequence rather than letter by letter, decodes that sequence into characters, and exports the result. The reading step is the heart of it, but every stage carries its own error budget. In practice, the accuracy you get depends far more on how closely your material resembles what the model was trained on than on any headline benchmark number.

What is the difference between HTR and OCR?

OCR reads printed type by exploiting its regularity: fixed letterforms, uniform spacing, and one reproducible shape per glyph, which lets the engine slice a line into characters and classify each against a known set of shapes. HTR was built to cope with the variation OCR assumes away — connected strokes, letterforms that vary between and within hands, and no clean boundary between characters. The two now overlap on difficult historical print such as Fraktur, blackletter, the long s, and typographic abbreviation, which push printed material toward the same shape-variation problems that make handwriting hard.

Why doesn't an HTR model read letter by letter?

Modern HTR models read a whole line at once and use context to resolve locally ambiguous shapes, rather than segmenting characters first. Older systems did try to cut lines into characters before classifying them, and that segmentation step dominated their errors — cut in the wrong place and every downstream classification inherited the mistake. Connectionist Temporal Classification, introduced by Graves and colleagues in 2006, let a model label unsegmented sequences directly, using a blank token and dynamic programming to align input to output. Every dominant architecture since works this way, much as a paleographer infers an unclear letter from the surrounding word.

Why do benchmark accuracy numbers not match what I get on my own documents?

Benchmark accuracy does not survive contact with a real archive. A model scoring single-digit character error rate on a clean dataset like IAM can score 30% or more on out-of-distribution material — a different hand, a different period, worse image quality, another language. Script, period, image quality, language, and individual writer all move the number, and almost no vendor publishes performance on representative samples of actual archive scans. When you read a single accuracy figure, the right question is never "how low is it?" but "on what material, and how close is that to mine?"

Why are general AI models like GPT-4o risky for transcribing historical handwriting?

General multimodal models lean heavily on textual plausibility and downsample the high-resolution image that carries the real evidence, so they tend to produce fluent, confident, well-formed text that is simply wrong. A character error from an image-first engine announces itself and is recoverable against the page; a fluent fabrication does not, and in a scholarly edition or family tree the error you cannot see does the damage. These models can be excellent on clean modern handwriting, but on historical hands the failure mode turns dangerous rather than merely inconvenient. GPT-4 Vision has been observed generating plausible but incorrect transcriptions of historical handwriting.