Making Archives Searchable: A Planning Guide for Cultural Heritage Professionals

This is a practical guide to making archives searchable — deciding what level of search your holdings actually need, and building the pipeline to get there. It separates two things that are easily confused: helping a researcher find the right box, and helping them find a word inside the documents. Knowing which problem you are solving is the difference between a collection people can use and one that still frustrates them.

"Searchable" is not one thing. Before choosing any tool, an archive has to decide which of two problems it is solving: helping a user find the right box, file, or series through a descriptive finding aid, or helping a user find a word or phrase inside the documents themselves. The first is metadata searchability, produced by descriptive cataloguing to standards like DACS and encoded in EAD. The second is full-text searchability, produced by running page images through OCR or handwritten text recognition. They use different tools, cost different amounts, and answer different questions. Getting the two confused is the most common reason a "digitised" collection still frustrates the people trying to use it.

This guide walks through the decision and the pipeline behind it, so you can plan the level of searchability your collection actually needs — and avoid paying for one you don't.

The distinction that governs everything: description vs. full text

A finding aid lets a researcher discover that your institution holds, say, the papers of a nineteenth-century merchant, arranged in twelve boxes, with correspondence in boxes three and four. That is a real and valuable form of searchability. It is also the only level most archives can afford to produce across their entire holdings, and the argument for working at that level rather than exhaustively item by item was made canonically in Greene and Meissner's More Product, Less Process — the observation that item-level processing is unsustainably slow relative to the scale of accessioned material.

What a finding aid does not do is let a researcher find every occurrence of a person's name inside those letters. For that you need the text itself: a machine-readable transcription indexed word by word. This is the level people usually imagine when they say "searchable," and it is far more expensive to produce, because every page has to be turned into text.

So the first planning question is not "which software?" but "searchable at what level?" Many collections are best served by good aggregate description and nothing more. Others — heavily used sources, a flagship correspondence series, records where the contents are the point — justify full-text treatment. Most institutions end up doing both, at different depths for different material. This is part of the wider set of choices covered in archives, digitization, and metadata workflows, and it pays to make the decision deliberately rather than by default.

The full-text pipeline, stage by stage

If you decide a body of material warrants full-text search, the work moves through five reasonably standard stages. Each constrains the next, which is why the whole thing is best understood as a single archival digitization workflow rather than a set of unrelated tasks.

1. Digitisation

You produce image derivatives from the physical or microfilm originals. Everything downstream is limited by what happens here: microfilm of nineteenth-century newsprint is a notorious constraint, and no text-recognition model recovers detail the image never captured. Leo, it is worth saying plainly, begins after this stage — it works from images you supply, not from a scanning rig.

2. Text recognition

This is where the print-versus-handwriting distinction bites, and where the wrong tool quietly wastes the most effort. OCR (Optical Character Recognition) converts images of printed text into characters; HTR (Handwritten Text Recognition) does the same for manuscript hands. They are different technologies, and the difference matters more than most planning documents admit — the distinction between HTR and OCR is worth understanding before you commit a budget to either.

3. Description and metadata

In parallel, you describe the material — DACS in the US, ISAD(G) internationally, increasingly the ICA's Records in Contexts — and encode it, typically in EAD for finding aids, wrapped in METS for the digital object, with PREMIS recording preservation events. How these descriptive, encoding, and preservation layers fit together without overloading any single schema is its own subject, treated in the guide to archival metadata standards.

4. Indexing and search

A search engine — Solr, Elasticsearch, Blacklight — builds an inverted index over your recognised text. Fuzzy and n-gram matching lets it tolerate recognition errors, which matters more than it sounds (see below). If your images and text are exposed through IIIF, the IIIF Content Search API 2.0, released in 2022, lets a cross-institutional searcher query your text and highlight the hit region on the page image itself.

5. Interoperability and delivery

OAI-PMH exposes your metadata to aggregators such as Europeana and DPLA; IIIF delivers the images and text; authority identifiers link your people and places to records held elsewhere.

The stages Leo occupies are the second and, in part, the third and fourth: turning page images into faithful text, holding that text alongside its image in an organised workspace, and letting you search across it. The capture at stage one and the OAIS-grade packaging and IIIF delivery at the far end are handled by other systems. It is worth being precise about that boundary rather than blurring it.

Where the money and the frustration go: reading the page

Stage two is where most searchability projects succeed or fail, so it deserves the most attention.

For printed material, off-the-shelf OCR is fast, cheap, and — on clean, modern, high-contrast type — very good, with commercial engines reaching well above 95% character accuracy on good scans. The trouble is that production corpora are not clean modern type. Historical print breaks the assumptions OCR is built on: early-modern founts and blackletter, the long s read as an f, ligatures split or dropped, the macron standing for an omitted nasal simply discarded, uneven inking and show-through read as character evidence, archaic spelling "corrected" into something the printer never set. Empirical studies routinely find character-accuracy gaps of ten to thirty percentage points between clean modern print and eighteenth- or nineteenth-century historical print, with larger gaps still for early-modern type and Fraktur. Roslyn Holley's analysis of the National Library of Australia's newspaper programme found baseline OCR character accuracy averaging around 70–80%, with the worst titles under 40%.

For handwriting, conventional OCR does not work at all; you need HTR. And here the standard tools carry a cost that planning often overlooks: specialist HTR systems typically expect you to train or fine-tune a model on your own material before they perform well on it. Transkribus documentation recommends on the order of 25–50 manually transcribed pages as a minimum starting set for a given hand, with real gains up to several hundred. That ground truth has to be produced by hand first — which, for an archive with many hands across many collections, is a substantial and recurring commitment before a single new page becomes searchable. Public pre-trained models exist for common hands, but they thin out fast for unusual or rapid ones.

There is a persistent temptation to shortcut all of this by pasting a page into a general-purpose chatbot. Resist it for primary transcription. Large language models produce fluent, confident, plausible output — and on a degraded or difficult hand, that fluency is precisely the danger: they invent and omit text and present the fabrication as smoothly as the truth. A specialist transcription error is usually a wrong character you can catch against the image; an LLM's error is a well-formed sentence that was never on the page. For a searchable corpus that researchers will cite, the second kind of error is far more corrosive, because it is far harder to see.

The accuracy target depends on what "searchable" means

One of the more useful findings in this field is that search tolerates more error than a scholarly edition does. Holley's work and the studies after it show that keyword search can succeed even at fairly high miss rates, because users tolerate imperfect recall as long as they can visually verify a hit against the page image. Where full recognition accuracy is genuinely out of reach, keyword spotting — matching a query term against the image without a full transcription — can make a collection discoverable at a lower quality threshold.

The practical consequence: the right accuracy target is set by the downstream use. If the goal is a critical edition, character-level fidelity matters and the source integrity of the transcription is paramount. If the goal is "let a family historian find a surname somewhere in ten thousand pages," a noisier index that surfaces candidate pages for visual confirmation may be entirely adequate. Decide which you are building before you decide how much accuracy to pay for.

A related point, often missed: you do not have to wait for complete text before you publish anything. Partial coverage is publishable. Incremental HTR pipelines and IIIF Content Search let an institution release searchable content as it is produced and re-run better models later. A collection that is 40% searchable today is more useful than one that is 0% searchable while you wait for a complete transcription.

Where a zero-shot transcription model changes the calculation

The training bottleneck described above is the reason many archives never get past image-only "digitisation" for their handwritten holdings. Producing ground truth for every hand is exactly the labour that MPLP warned was unsustainable at scale. This is the specific stage where it is worth knowing that Leo exists.

Leo's transcription engine, ATR-1, is a specialist model for Latin-script material that runs zero-shot — out of the box, with no per-collection model to train first. The constraint is the writing system, not the language: it reads any language written in the Latin alphabet — English wills, French notarial records, Dutch registers, German parish books, Spanish and Italian correspondence — and handles both manuscript hands and printed matter, including the early-modern typography that trips conventional OCR. (Leo's own public corpus, ExLatinis, is a Latin-language project, but Latin is one case among the many vernaculars the model reads, not the point of it.) What it does not read is non-Latin scripts — Greek, Cyrillic, Hebrew, Arabic, and the East Asian and Indic writing systems.

On a randomised 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, ATR-1 scored roughly a 5% character error rate at release — about 61% fewer errors than the next-best model tested, with Transkribus's Text Titan I around 13% and the general LLMs ranging from the low twenties to well over fifty percent; the full benchmark data is public. The design principle behind that figure is source integrity: ATR-1 transcribes what is on the page, preserving strikethroughs, marginalia, expansions, and archaic spelling rather than smoothing them into modern prose. Around the model sits a document workspace — folders, per-document metadata on a fixed Dublin-Core-adjacent field set, image shown beside transcription, fuzzy search across everything, and export to TEI, Word, PDF, or HTML — which covers the transcribe-organise-search stretch of the pipeline without the model-training step. For an academic or archival project, the Leo Transcription Grant offers substantial free credits in exchange for publishing the resulting transcriptions openly; if your material is chiefly historical print, note the grant does not cover it, and the free or paid plans are the route instead.

None of this removes the need for verification. Any machine transcription — Leo's included — is a draft to be checked against the image, and the closer your target is to a citable edition, the more checking it needs. What a zero-shot model changes is the starting cost: it moves an archive from "produce hundreds of ground-truth pages before anything is searchable" to "get a usable draft on the first page and correct from there."

Deciding what to make searchable, and how deeply

Pulling the threads together, a workable plan for an archive looks less like "digitise everything and OCR it" and more like a set of deliberate choices:

• Triage by use, not by shelf order. Aggregate description for the whole; full-text treatment reserved for material where the contents are what people search.
• Match the recognition tool to the material. OCR for clean print, HTR for handwriting and historical typography, and an honest accounting of the training cost before committing to a tool that demands it.
• Set the accuracy target from the downstream use. A finding aid, a searchable index, and a critical edition are three different jobs with three different tolerances.
• Publish incrementally. Release what is searchable now; improve it as better models and more corrections arrive.

The goal is not a uniform, fully transcribed collection — that standard defeats most institutions before they start. The goal is that a researcher who needs something in your holdings can find it at the level their question requires, and confirm it against the page. An archive that gets that right has done the real work of access: it has closed the distance between a document existing and a document being read.

Frequently Asked Questions

How do you make archives searchable?

Making archives searchable starts with deciding which of two problems you are solving: helping a user find the right box, file, or series through a descriptive finding aid, or helping them find a word or phrase inside the documents themselves. The first is metadata searchability, produced by descriptive cataloguing to standards like DACS and encoded in EAD. The second is full-text searchability, produced by running page images through OCR for print or handwritten text recognition for manuscript. They use different tools, cost different amounts, and answer different questions. Choose the level your collection actually needs before choosing any software.

What is the difference between OCR and HTR?

OCR (Optical Character Recognition) converts images of printed text into machine-readable characters; HTR (Handwritten Text Recognition) does the same for manuscript hands. They are different technologies, and matching the wrong one to your material quietly wastes effort. Off-the-shelf OCR is fast, cheap, and very good on clean modern print, but conventional OCR does not work at all on handwriting. Historical typography — early-modern founts, blackletter, the long s — also breaks the assumptions OCR is built on, causing large accuracy gaps compared with clean modern type. Understand which technology your material demands before committing a budget to either.

Why not use ChatGPT to transcribe historical documents?

Avoid general-purpose chatbots for primary transcription. Large language models produce fluent, confident, plausible output, and on a degraded or difficult hand that fluency is precisely the danger: they invent and omit text while presenting the fabrication as smoothly as the truth. A specialist transcription error is usually a wrong character you can catch against the image; an LLM's error is a well-formed sentence that was never on the page. For a searchable corpus that researchers will cite, the second kind of error is far more corrosive, because it is far harder to see. Use a specialist transcription model instead.

How accurate does transcription need to be for full-text search?

Search tolerates more error than a scholarly edition does, so the right accuracy target is set by the downstream use. Keyword search can succeed even at fairly high miss rates, because users tolerate imperfect recall as long as they can visually verify a hit against the page image. If the goal is "let a family historian find a surname somewhere in ten thousand pages," a noisier index that surfaces candidate pages for confirmation may be adequate. If the goal is a critical edition, character-level fidelity matters and source integrity is paramount. Decide which job you are building before deciding how much accuracy to pay for.

Does Leo's ATR-1 need training on my collection before it works?

No. ATR-1 runs zero-shot — out of the box, with no per-collection model to train first — which removes the usual bottleneck where specialist HTR systems expect you to hand-transcribe dozens or hundreds of ground-truth pages before they perform. The constraint is the writing system, not the language: it reads any language written in the Latin alphabet, including English, French, Dutch, German, Spanish, and Italian, and handles both manuscript hands and printed matter, including early-modern typography. It does not read non-Latin scripts such as Greek, Cyrillic, Hebrew, Arabic, or East Asian and Indic writing systems. Any machine transcription remains a draft to be checked against the image.