How Does HTR Work? A Look Inside the HTR Pipeline

This article explains how HTR works, stage by stage — from the image on the scanner to the text you can search. It names the architectures that do the reading, marks the places where the pipeline still breaks, and shows why the accuracy you get depends far more on how closely your material matches a model's training than on any headline benchmark. If you work with manuscripts, knowing where a pipeline fails is what separates a usable transcription from a plausible-looking one.

Handwritten text recognition (HTR) is a computer-vision task that turns an image of handwriting into machine-readable text. It works as a pipeline: acquire the image, prepare it, find the text lines on the page, run a neural network that reads each line as a continuous sequence rather than letter by letter, decode that sequence into characters, and export the result. The reading step is the heart of it, but every stage carries its own error budget, and the accuracy you get in practice depends far more on how closely your material resembles what the model was trained on than on any headline benchmark number.

The rest of this article walks through each stage, names the architectures that do the work, and marks the places where the process still breaks.

Why Handwriting Needs Its Own Pipeline

Printed type is regular enough that a system can slice a line into characters and classify each one; handwriting is not, because strokes connect, letterforms vary even within one hand, and there is often no clean boundary between one character and the next. Early-modern print (Fraktur, the long s, ligatures) sits between the two, which is why a model built for historical documents, such as Leo's ATR-1, reads early-modern print as well as handwriting. We set out the full comparison in HTR vs OCR; this article is one strand of the broader story we tell across our handwritten text recognition hub.

The Pipeline, Stage by Stage

There is broad consensus on the ordering of an HTR pipeline. The stages below are the ones you will find in nearly every serious system, open-source or commercial.

Image Acquisition

Everything downstream is bounded by what the camera or scanner captured. Resolution and colour depth set a ceiling on accuracy that no later stage can lift: a model cannot recover a stroke the sensor never recorded. High-resolution capture matters more than it looks, because the fine distinctions between similar letters — a worn e and c, a minim count in m versus ni — live in exactly the detail that low resolution throws away. This is also why downsampling quietly costs you accuracy, a point that returns when we reach general-purpose models. If you are planning capture, our guide to what resolution to scan old documents at sets out the trade-offs.

Preprocessing

Historically this meant deskewing, denoising, and binarizing — reducing the image to black and white before recognition. The intuition was that a cleaner image is an easier image. Modern neural models have largely overturned it. Coquenet and colleagues, in their 2024 study of best practices for HTR systems, show that aggressive binarization and aspect-ratio-breaking resizing actively hurt deep models; their recommendations include preserving aspect ratio when resizing and skipping binarization altogether. Preprocessing is no longer the dominant source of error, and more of it is not better. A deep model reads grayscale evidence — the pressure of a stroke, the fade of iron-gall ink — that binarization would have discarded.

Layout Analysis and Segmentation

Before a model can read a line, the system has to find the lines: detect text regions, establish reading order, and locate baselines. On a clean single-column page this is nearly solved. On historical material it is not. Marginalia, multi-column layouts, tables, drop caps, and interlinear annotations remain genuinely hard, and the 2025 survey of text-line segmentation and baseline detection by Rabaev and colleagues identifies this as one of the main unresolved areas in the field. Layout analysis carries its own error budget, separate from recognition. A perfect reading model still produces garbled output if the reading order was wrong. In some modern tools this stage is invisible to the user: in Leo, for instance, you upload the page image and there is no separate preprocessing or line-segmentation step to run.

Text Recognition

This is the stage most people mean when they say "HTR." A feature extractor turns the line image into a sequence of visual features; a sequence model turns those features into a sequence of characters. The architecture that does this has changed several times, and the generations are worth knowing because they are still all in production somewhere.

The trained network at this stage is what people mean by an HTR model: a set of learned weights that maps images of writing to text. What it was trained on decides what it reads well. A model trained on modern English handwriting and one trained on early-modern court hands can share an architecture and still behave like different tools, which is why "which model?" matters more than "which software?".

The crucial thing to understand is what this stage does not do: it does not read letter by letter — a point developed in the section below, along with the architectural generations that have taken turns doing the reading.

Decoding and Language Modeling

The recognition model emits, in effect, a probability distribution over characters. Decoding turns that into final text — greedy selection of the most likely character at each step, or beam search, sometimes with a lexicon or a language model layered on to bias the output toward plausible words.

Here lies a trade-off that matters enormously for historical work. A language model raises surface accuracy on common words, but it does so by pulling the output toward what is common — which is precisely wrong when the source contains rare spellings, dialect forms, abbreviations, and unusual names. Coquenet and colleagues document this trade-off directly. It is the same mechanism, at a smaller scale, as the one that makes general LLMs risky for transcription: correction and normalization improve the average case and quietly corrupt the exceptional one. For a family historian, the "exceptional one" is often the surname you were searching for. If you do want regularized spelling, it is safer as a separate, visible step after transcription than as a hidden one inside it — the distinction our guide to historical language normalization works through.

Post-processing and Export

Finally the text is serialized into a format that preserves layout and provenance. The common standards are PAGE XML and ALTO XML for layout-plus-text, and TEI XML for scholarly encoding downstream. This is not a trivial stage: a transcription that cannot be exported in a form your edition, database, or catalogue can consume is a transcription you will end up re-keying.

Why the Model Does Not Read Letter by Letter

This is the most common misconception about HTR, and correcting it explains why modern systems work at all.

Older systems did try to segment characters before classifying them, and pre-segmentation failure dominated their error budgets — if you cut the line in the wrong place, every downstream classification inherited the mistake. The advance was learning to skip segmentation entirely. Alex Graves and colleagues introduced Connectionist Temporal Classification (CTC) in 2006 precisely to label unsegmented sequence data: the model consumes a whole line and emits a character sequence directly, using a blank token and dynamic programming to align input to output without ever being told where one character ends and the next begins.

Every dominant architecture since works this way. It reads the line as a whole, using context to resolve locally ambiguous shapes — much as a human paleographer infers an unclear letter from the word around it rather than staring at the glyph in isolation.

The Four Generations of Recognition Models

The recognition stage has passed through four established architectural generations, with a fifth now emerging, and understanding them clarifies the choices behind any tool you might use.

Generation 1 — Hidden Markov Models

Handcrafted features (stroke slopes, loops, projection profiles) fed into HMMs. The pre-deep-learning baseline, now historical.

Generation 2 — CNN + BLSTM + CTC

Deep convolutional features feed a bidirectional LSTM trained with the CTC loss. This has been the workhorse since roughly 2013 and remains production-grade. It is the architecture behind Transkribus and PyLaia. Notably, Coquenet and colleagues show that with three careful fixes — aspect-ratio preservation, column-wise max-pooling, and an auxiliary CTC loss — this older design stays competitive with much larger Transformers on standard benchmarks. Newer is not automatically better.

Generation 3 — Attention Encoder–Decoder

A CNN or RNN encoder paired with an attention-based decoder. Flexible, but data-hungry.

Generation 4 — Transformer / TrOCR

Pure attention: an image-Transformer encoder and a text-Transformer decoder, pretrained on large synthetic and unlabeled corpora. Microsoft's TrOCR is the reference implementation, and this generation is the research state of the art on benchmarks like IAM.

Generation 5 (Emerging) — Multimodal Foundation Models

The large general multimodal models from OpenAI, Google and Anthropic, applied zero-shot to handwriting. They have improved quickly since 2024, and as of October 2026 the best of them read many later, more regular hands well. They remain generalists: they downsample the image and lean on what is linguistically probable, so on historical hands they tend to quietly normalize — for reasons the next sections make concrete.

What the Benchmark Numbers Actually Tell You

You will see impressive figures quoted for HTR. TrOCR reports 3.42% CER on the IAM test set. The best system in the ICDAR 2017 competition on the Bentham dataset hit 1.91% CER. A 2025 benchmark of large language models for HTR by Crosilla and colleagues reports an average character error rate of around 2.95% across English, German, and French handwriting.

Character error rate (CER) is the standard measure: the edit distance between the model's output and a correct reference, divided by the reference length. Lower is better. It is a good metric. The problem is not the metric but the transfer.

Benchmark CER does not survive contact with a real archive. A model scoring single-digit CER on IAM can score 30% or more on out-of-distribution material — a different hand, a different period, worse image quality, another language. The generalization study frames this gap as the central unsolved deployment problem in HTR. Script, period, image quality, language, and individual writer all move the number (and which languages and scripts a model can actually read depends on the same interaction of script, language, and hand), and almost no vendor publishes performance on representative samples of actual archive scans. When you read a single accuracy figure, the right question is never "how low is it?" but "on what material, and how close is that to mine?"

This is also why the claim that HTR is "solved" is misleading. Headlines about a general model conquering handwriting rest on particular material, mostly English hands of the eighteenth and nineteenth centuries; the same model class can still stumble on older and less regular hands. On the models then current, a 2024 study found Gemini 1.5 Pro underperforming a fine-tuned TrOCR on real historical documents — one corpus, and the general models have moved on since, but a useful corrective to the headline. HTR for difficult material is not solved.

The Two Operating Models: Train-Your-Own Versus Zero-Shot

Faced with the recognition stage, tools split into two camps. (For a survey of the HTR software on each side, see our comparison.)

User-trained engines — Transkribus, Kraken, PyLaia — ask you to supply ground-truth transcriptions for a representative sample of pages, then return a model tuned to your collection. On a genuinely idiosyncratic hand this remains the most reliable path, and it is the honest recommendation when your material is far from anything a general model has seen. The cost is real: transcribing ground truth is the very bottleneck you were trying to relieve, and the effort recurs with each new collection.

Zero-shot models — TrOCR out of the box, and the general multimodal systems — require no training data. Their accuracy depends entirely on how close your material sits to their training distribution. For clean modern handwriting that can be excellent. For historical hands it is a gamble, and it is where the failure mode turns dangerous rather than merely inconvenient.

That failure mode is worth stating plainly, because it is where the choice of tool stops being academic. On a safety spectrum, image-first recognition engines tend to err on individual characters — a misread letter, a dropped mark — errors that are visible and recoverable against the page. General multimodal models sit at the other end: because they lean heavily on textual plausibility and downsample the high-resolution image that carries the real evidence, they err fluently, putting a more probable spelling or word in place of the one on the page. A character error announces itself. A fluent substitution does not — and in a scholarly edition or a family tree, the error you cannot see is the one that does the damage.

Where a Purpose-Built Model Fits

This is the stage where the choice of tool matters most, and it is worth being direct about it. A model trained specifically on images of historical documents occupies useful ground between the two camps: zero-shot, so there is no per-collection training step, but built for manuscript hands rather than for clean modern text. This is where Leo's own model, ATR-1, sits. It is a Transformer, like TrOCR and the general models, but tuned specifically for historical documents, handwritten and printed, and its design answers the weak points of the pipeline above:

• Acquisition and resolution. General models downsample the image they are given. ATR-1 reads it at high resolution, weighing the visual evidence against context, so the minims and worn letters the capture stage worked to preserve are still there when the model reads.
• Decoding and normalization. ATR-1 is trained to transcribe what is on the page rather than normalize it. It does not over-correct or silently modernize, so the long s stays a long s, an archaic spelling survives, and a macron standing for an omitted nasal is not silently resolved into modern letters. Modernization, correction and translation are available, but as separate Transformations, each written to its own tab.
• Failure detection. Any text-generating model can fall into a repetition loop on material far from its training, writing the same line over and over. ATR-1 checks its output for patterns like that: suspect output is hidden, retried automatically, and refunded if it still cannot succeed. The errors that do get through tend to be the recoverable kind, a wrong letter or word you check against the image shown beside the text.

It reads Latin-script material in any language written in that alphabet — English wills, French notarial records, Dutch registers, German parish books, and Latin among them — not any single language. Corrections users make feed back into training, so accuracy compounds across releases, and finished transcriptions export to TEI XML, Word, PDF, or HTML for downstream editions and catalogues. It is not perfect, and no honest HTR is; the point is that its errors tend to be the recoverable kind, and it does not smooth the page into modern prose.

ATR-1 is the default in Leo's model library, at 1 credit an image. The same menu holds the large general models from OpenAI, Google Gemini and Anthropic Claude, which you can run as a second reading of a hard page — each run lands in its own tab beside the image — and small, cheap models for clean, legible print such as a 19th-century book or a typescript. Those small models are for print, not handwriting.

Reading the Pipeline Critically

The value in understanding the HTR pipeline is not academic. It tells you where to look when a transcription disappoints. A garbled paragraph in the middle of a clean page usually points to layout analysis — the reading order was wrong, not the reading. Consistently mangled rare words point to an over-eager language model or a post-correction step normalizing what it should have preserved. A whole document that reads fluently but diverges from the source on names and dates points to a model erring on plausibility rather than evidence, and that is the pattern to fear most, because it hides.

Match the model to the material. Evaluate on held-out pages from your own collection, never on a public benchmark. Treat any language-model or post-correction layer as something to inspect rather than trust. And read the output against the page — because the transcription that looks most confident is not always the one that tells you the truth about what the writer actually wrote.

Frequently Asked Questions

How does HTR work?

HTR works as a pipeline that turns an image of handwriting into machine-readable text. It acquires the image, prepares it, finds the text lines on the page, runs a neural network that reads each line as a continuous sequence rather than letter by letter, decodes that sequence into characters, and exports the result. The reading step is the heart of it, but every stage carries its own error budget. In practice, the accuracy you get depends far more on how closely your material resembles what the model was trained on than on any headline benchmark number.

Why doesn't an HTR model read letter by letter?

Modern HTR models read a whole line at once and use context to resolve locally ambiguous shapes, rather than segmenting characters first. Older systems did try to cut lines into characters before classifying them, and that segmentation step dominated their errors — cut in the wrong place and every downstream classification inherited the mistake. Connectionist Temporal Classification, introduced by Graves and colleagues in 2006, let a model label unsegmented sequences directly, using a blank token and dynamic programming to align input to output. Every dominant architecture since works this way, much as a paleographer infers an unclear letter from the surrounding word.

Why do benchmark accuracy numbers not match what I get on my own documents?

Benchmark accuracy does not survive contact with a real archive. A model scoring single-digit character error rate on a clean dataset like IAM can score 30% or more on out-of-distribution material — a different hand, a different period, worse image quality, another language. Script, period, image quality, language, and individual writer all move the number, and almost no vendor publishes performance on representative samples of actual archive scans. When you read a single accuracy figure, the right question is never "how low is it?" but "on what material, and how close is that to mine?"

Why are general AI models risky for transcribing historical handwriting?

General multimodal models lean heavily on textual plausibility and downsample the high-resolution image that carries the real evidence. They have improved quickly, and read many later, regular hands well, but their surviving error is fluent normalization: a more probable spelling or word in place of the one on the page. A character error from an image-first engine announces itself and is recoverable against the page; a fluent substitution does not, and in a scholarly edition or family tree the error you cannot see does the damage. A model built for historical documents, such as Leo's ATR-1, is trained to keep what is on the page instead.

What is an HTR model?

An HTR model is the trained neural network at the heart of a handwritten text recognition system: it takes an image of a line or page of writing and outputs the text. Most current HTR models are either CNN–BLSTM networks trained with CTC or Transformer encoder–decoders, and what decides how well one reads your documents is less its architecture than what it was trained on. Some are trained per collection by the user; others, such as Leo's ATR-1, are trained in advance on historical documents and work zero-shot.