How to Search Handwritten Documents

This is a practical guide to how to search handwritten documents you have photographed yourself — the eight hundred JPEGs from last summer's archive trip that no query can currently reach. It covers what has to happen between a folder of images and a corpus you can search, verify, and cite: capture, transcription, quality control, and the difference between the text you quote and the text you index.

To search handwritten documents you have photographed yourself, you need three things a folder of JPEGs does not give you: a text layer, an index over that text, and a stable link between each hit and the page image it came from. An image is pixels, not characters — a keyword query has nothing to match against until transcription, whether by hand or by handwritten text recognition (HTR), produces machine-readable text. The practical sequence is: capture cleanly, transcribe faithfully, keep the diplomatic reading as the master text, add normalized or variant spellings as a separate labelled layer, then index and search — always displaying the hit against the image so you can verify the reading before you cite it.

That is the whole architecture. What follows are the decisions inside it, because each one constrains what you will be able to find two years from now.

Why your archive photos are not searchable, and what would make them so

Most researchers arrive at this problem after a productive archive trip. Eight hundred photographs, four boxes, three hands, and a growing suspicion that the letter mentioning the loan is in there somewhere. The instinct is to reach for the operating system's search bar. It returns nothing, because there is nothing to return.

The mechanics are worth understanding precisely, because they determine every downstream choice. Search over text works through an inverted index: text is tokenized and analyzed into terms, and an index — Lucene, Solr, or Elasticsearch in most implementations — maps each term to the documents and pages where it occurs, often with positions. A query is analyzed the same way, and matching terms return their postings. Without a text layer, there are no postings. This is why the IIIF Content Search API describes its function as searching OCR-generated or transcribed content associated with an image: the image itself is never the searchable object.

There are several places the text can live. A searchable PDF keeps the page raster visible and lays an invisible, position-aligned text layer beneath it — OCRmyPDF documents this "sandwich" approach explicitly. In XML-based workflows, ALTO stores layout information and recognized text; PAGE XML represents regions, lines, words, glyphs, reading order and content; hOCR encodes the same kind of information inside HTML. For scholarly work, the TEI Guidelines for representing primary sources let you hold a transcription alongside a facsimile, recording the position and relative size of elements on the written surface and linking them to the image.

For a single researcher building a personal corpus, you do not need all of this. You need one thing absolutely — text bound to a page identifier — and one thing nearly always: the image displayed beside the text at the moment you read a search result.

Stage one: capture that will still be legible to a model in five years

The text layer inherits every fault in the image. A photograph taken at an angle, in reading-room shadow, with the gutter curving away, produces a transcription with holes in it, and no amount of downstream tuning recovers them.

The specifics of reading-room photography — geometry, lighting, focus, handling — are a discipline of their own, and it is worth getting the capture method right before the trip rather than discovering the problem at your desk. Two points matter especially for searchability. First, shoot at high resolution and keep the originals; recognition models weigh fine visual evidence, and there is no way to add detail later. Second, photograph the reference — the box, the call number, the folio — as part of the sequence. A transcription you cannot locate in the archive is not a citable source, and the coordinates a footnote must contain have to be attached at capture or reconstructed painfully afterwards.

Name and group your files by archival unit at this stage. Box, folder, item, sequence. It takes ten minutes and it is the skeleton your index will hang on.

Stage two: producing the text layer

This stage decides whether the corpus is worth building, and it is where most of the confusion sits: OCR and HTR are not interchangeable technologies with different price tags. They approach the page differently. Conventional OCR segments text into small patches and classifies glyphs against a model of regular, modern type. HTR is typically line- or sequence-based, trained on image lines paired with accurate transcriptions, and recognizes continuous writing without first isolating individual characters. Romero and colleagues put the consequence bluntly in their survey of handwritten text recognition for historical documents: OCR products, however accurate on print, are very far from offering useful solutions to the HTR problem. A system can be excellent on a modern scan and produce nothing usable from a seventeenth-century secretary hand.

The distinction between OCR and HTR, and how each handles historical material, is worth reading properly if you are choosing a route for the first time. For corpus-building purposes, three routes are live.

Human transcription

Yours or a volunteer community's, this produces the highest-fidelity text and is the slowest. The Library of Congress's By the People programme has volunteers transcribe and then review pages precisely to improve search and discovery, and Transcribe Bentham demonstrates a reviewed scholarly workflow at scale — a published analysis examined 4,364 checked and approved transcripts submitted between October 2012 and June 2014. Those are project totals, not throughput rates you can plan against, and review labour rather than initial transcription is usually the constraint. For a personal corpus of several hundred pages, this route is realistic only for your most important documents.

Trainable specialist HTR

Transkribus, eScriptorium, OCR4all — these put a strong engine behind a training step. Transkribus's data-preparation guidance suggests something on the order of 15–30 representative pages for a consistent hand and at least 10,000 words per handwritten hand to train against. If your corpus is one scribe writing consistently for four hundred folios, that investment pays. If it is three boxes of correspondence in thirty hands, it does not: whether training your own model is worth it turns almost entirely on how homogeneous your material is.

Zero-shot HTR

This applies a ready model with no training step. It is where most individual researchers should start, if only because you can find out in an afternoon whether the material is readable at all before committing weeks to ground truth.

What none of these routes should do is silently improve the page. A model that resolves an abbreviation, modernizes a spelling, or repairs a scribe's slip has removed evidence from your corpus — and has done so invisibly. This is the specific danger with general chat models pressed into transcription service: their errors arrive as fluent, plausible prose rather than obvious garble, which is exactly what makes them hard to catch on a page you cannot easily read yourself. The failure mode is well enough documented that it deserves its own treatment; fluent-but-wrong output is the hardest error class to detect in historical transcription.

Where Leo fits in this stage

Leo occupies exactly this step and no more: you upload the images you captured, ATR-1 transcribes them, and you get text bound to each page inside a workspace where the image sits beside the transcription. There is no model to train first — ATR-1 is zero-shot, so a folder of mixed hands from four different scribes goes through the same way a homogeneous letterbook does. It reads Latin-script material of roughly the past five hundred years, whatever language is written in that alphabet: English wills, French notarial minutes, Dutch registers, German parish books, Spanish and Italian correspondence. Non-Latin scripts — Greek, Cyrillic, Hebrew, Arabic, Indic and East Asian systems — are out of scope. Printed matter is in scope too, including the early-modern founts where the long s, ligatures and typographic abbreviation defeat conventional OCR.

The design commitment that matters for a searchable corpus is source integrity: ATR-1 transcribes what is on the page, preserving strikethroughs, additions, marginalia, expansions and archaic orthography rather than smoothing them into modern prose. Your master text stays diplomatic. Interpretation happens in a separate layer — Transformations that Correct, Modernize, Translate or Summarize write to a new tab and never overwrite the base transcription — which is precisely the dual representation a defensible corpus needs.

On accuracy, the cleared figure: on a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, at ATR-1's release, Leo scored roughly 5% character error rate against Transkribus (Text Titan I) at about 13%, Claude Opus at 23.3%, Gemini 2.5 Pro at 24.8% and GPT-4.1 at 56.7% — 61% fewer errors than the next-best model, with the full comparison data published here. That is one corpus, one language, one moment in time. Your hands are not the Folger's hands, and the honest advice is the same as it would be for any tool: run a held-out sample of your own material before you commit a corpus to it. Leo's known weak spot is worth stating: pages that mix dominant printed structure with dense handwriting — pre-printed ledger and deed-book forms — where the model can favour the printed headers over the manuscript entries.

Stage three: quality control before indexing, not after

Errors in the text layer become invisible once they are in an index. A misread name does not announce itself; it simply never appears in a result set.

Judge quality by character error rate, which is character edit distance normalized by the reference length — CER = (S + D + I) / N, counting substitutions, deletions and insertions. Word error rate applies the same idea at word level. They are related but not convertible: "the cat sat" against "the cot sat" is one substitution in eleven characters, 9.1% CER, but one word in three, 33.3% WER. A fuller account of what these metrics do and do not measure is worth having before you set a threshold for yourself.

How much error can a searchable corpus tolerate? There is no universal answer, and anyone offering one is extrapolating. The most directly relevant experiment is Bazzo and colleagues' study of OCR errors and information retrieval, which injected word-error rates of 1%, 5%, 10%, 25% and 50% into a 103,000-document Portuguese newspaper collection across 72 runs: statistically significant impact began at 5% error, and at 10% all runs were significantly affected. That is one collection, one language, one retrieval setup. Earlier simulated-OCR work by Croft and colleagues found that document and query length matter as much as error rate — short documents with short queries degraded most, while long full-text documents were relatively robust. A twelve-page will and a two-hundred-page letterbook do not have the same error tolerance.

The practical implication: do not chase a global average. Sample pages against their images, and weight your review towards the tokens that carry the queries — personal names, place names, dates, sums. Those are the terms you will actually search, and they are the terms with no redundancy to protect them. A method for prioritizing high-stakes tokens during verification will save more retrieval than a general proofread.

Stage four: the diplomatic text and the findability layer

Here is the trap that catches careful researchers. You have a transcription with no errors in it. You search for "Sheffield" and find nothing, because the scribe wrote "Shefeild." A faithful transcription of a historical form is still unfindable through a modern query.

The resolution is not to correct the source. It is to keep two representations. The diplomatic transcription stays as written and is what you quote, cite, and display. A separate, explicitly labelled normalized or alias field carries modern and variant forms and is what you additionally index. Transkribus's own conventions guidance makes the same argument from the encoding side: choose a transcription policy and be consistent, and record abbreviation expansions as tagged properties rather than silently replacing the source reading.

Normalization is not free, and its evidence base is thinner than it looks. Bollmann's large-scale comparison of historical text normalization systems across eight languages reports maximum word accuracy above 97% for most datasets, with cSMTiser reaching 91.7% on Hungarian against 80.1% for a comparison system. A DH2018 study adapting VARD2 to seventeenth-century Dutch automatically respelled 62% of 340,000 variants in the 1657 Bible at 95% confidence, and 84% of variants in the 1637 edition at 75% confidence. These are word-accuracy figures, not retrieval-recall figures. How many additional historian queries they recover is genuinely unestablished.

Fuzzy search covers some of the remaining gap. Lucene's fuzzy query measures similarity by Damerau-Levenshtein edit distance and matches terms up to two edits; n-gram tokenization is another route, where shorter n-grams match more documents at lower precision and trigrams are a reasonable starting point. Treat fuzziness as a recall aid with a cost: wider matching enlarges the candidate set, admits false positives, and slows queries. Test your thresholds against pages you already know are relevant.

Leo's search operates on this principle at the individual-researcher scale — global fuzzy search across documents, transcriptions and annotations with adjustable sensitivity, plus filtering on the per-document metadata fields (Title, Creator, Date, Archive, Collection, Box, Folder, Identifier) you fill in as you go. It is string matching, not concept matching, which is why the metadata discipline at capture pays off.

Stage five: keeping the corpus usable beyond your own desktop

Two habits separate a corpus you can still work with in five years from a folder you eventually abandon.

Record the corpus policy as metadata, not memory. Language, period, hand, abbreviation policy, spelling policy, transcription conventions, and which pages have been human-verified. When a reviewer asks how you handled contractions, the answer should be a document, not a recollection. The same record supports disclosing machine assistance in a method note — the manuscript takes the citation; the transcription method belongs in the apparatus.

Export in a format that outlives the tool. TEI XML is the scholarly export for transcriptions bound to sources; Word, HTML and PDF cover circulation and drafting. Leo exports all four, plus images. What it does not produce is ALTO or PAGE XML, and it does not serve IIIF — if your corpus needs to feed an institutional repository with region-level highlighting, that is a downstream stage handled by other infrastructure, and worth confirming before you build.

The corpus is an argument about your sources

Everything above is, in the end, editorial work wearing technical clothes. Choosing to keep the diplomatic reading as master text is a claim about what the source is. Adding a normalized alias field is a claim about how you expect to look for it. Deciding that names and dates get verified while boilerplate does not is a claim about where your evidence actually lives.

That is why the corpus-building stage belongs inside the wider research workflow that runs from archive photo to citable source rather than sitting off to one side as data preparation. A searchable corpus does not just speed up the reading you were going to do anyway. It changes which questions are askable — you start noticing the third time a name appears across two boxes, the phrase that recurs in five unrelated depositions, the year everything in the series goes quiet. Those patterns were always in the paper. They were simply beyond the reach of a reader turning pages one at a time.

Build it carefully enough that when you find something, you can go back to the image, read the line yourself, and be certain.

Frequently Asked Questions

How do I search handwritten documents I photographed at the archive?

You need three things a folder of JPEGs cannot give you: a text layer, an index over that text, and a stable link between each search hit and the page image it came from. The sequence is: photograph cleanly and capture the box, folder and call number as part of the shoot; transcribe the pages by hand or with handwritten text recognition; check quality before indexing; then search, always displaying the result against the image so you can verify the reading before citing it. Until transcription produces machine-readable characters, a keyword query has nothing to match against.

What is the difference between OCR and HTR for historical documents?

OCR and HTR approach the page differently rather than being the same technology at different price points. Conventional OCR segments text into small patches and classifies glyphs against a model of regular, modern type. HTR is typically line- or sequence-based, trained on image lines paired with accurate transcriptions, and recognises continuous writing without first isolating individual characters. Romero and colleagues, surveying handwritten text recognition for historical documents, concluded that OCR products — however accurate on print — are very far from offering useful solutions to the HTR problem. A system can be excellent on a modern scan and produce nothing usable from a seventeenth-century secretary hand.

How accurate does a transcription need to be before search works?

There is no universal threshold, but the most directly relevant experiment offers a rough floor. Bazzo and colleagues injected word-error rates of 1%, 5%, 10%, 25% and 50% into a 103,000-document Portuguese newspaper collection across 72 runs: statistically significant impact on retrieval began at 5% error, and at 10% every run was affected. Earlier work by Croft and colleagues found document and query length matter as much as error rate — short documents with short queries degrade most, long full-text documents are relatively robust. Rather than chasing a global average, sample pages against their images and concentrate review on names, places, dates and sums.

Why can't I find a name in my transcription even though the transcription is correct?

Because a faithful transcription of a historical form is still unfindable through a modern query — you search "Sheffield" and the scribe wrote "Shefeild." The answer is not to correct the source. Keep two representations: the diplomatic transcription stays as written and is what you quote, cite and display; a separate, explicitly labelled normalised or alias field carries modern and variant spellings and is what you additionally index. Fuzzy search covers part of the remaining gap, matching terms within a small edit distance, but wider matching enlarges the candidate set, admits false positives and slows queries.

Do I need to train an HTR model on my own handwriting samples?

It depends almost entirely on how homogeneous your material is. Trainable systems such as Transkribus, eScriptorium and OCR4all put a strong engine behind a training step; Transkribus's guidance suggests roughly 15–30 representative pages for a consistent hand and at least 10,000 words per hand. If one scribe wrote four hundred folios consistently, that investment pays. Three boxes of correspondence in thirty hands, and it does not. Zero-shot recognition applies a ready model with no training step, which is where most individual researchers should start — you can find out in an afternoon whether the material is readable before committing weeks to ground truth.