This is a practical guide to making archives searchable — deciding what level of search your holdings actually need, and building the pipeline to get there. It separates two things that are easily confused: helping a researcher find the right box, and helping them find a word inside the documents. Knowing which problem you are solving is the difference between a collection people can use and one that still frustrates them.

"Searchable" is not one thing. Before choosing any tool, an archive has to decide which of two problems it is solving: helping a user find the right box, file, or series through a descriptive finding aid, or helping a user find a word or phrase inside the documents themselves. The first is metadata searchability, produced by descriptive cataloguing to standards like DACS and encoded in EAD. The second is full-text searchability, produced by running page images through OCR or handwritten text recognition. They use different tools, cost different amounts, and answer different questions. Getting the two confused is the most common reason a "digitised" collection still frustrates the people trying to use it.

This guide walks through the decision and the pipeline behind it, so you can plan the level of searchability your collection actually needs — and avoid paying for one you don't.

The distinction that governs everything: description vs. full text

A finding aid lets a researcher discover that your institution holds, say, the papers of a nineteenth-century merchant, arranged in twelve boxes, with correspondence in boxes three and four. That is a real and valuable form of searchability. It is also the only level most archives can afford to produce across their entire holdings, and the argument for working at that level rather than exhaustively item by item was made canonically in Greene and Meissner's More Product, Less Process — the observation that item-level processing is unsustainably slow relative to the scale of accessioned material.

What a finding aid does not do is let a researcher find every occurrence of a person's name inside those letters. For that you need the text itself: a machine-readable transcription indexed word by word. This is the level people usually imagine when they say "searchable," and it is far more expensive to produce, because every page has to be turned into text.

So the first planning question is not "which software?" but "searchable at what level?" Many collections are best served by good aggregate description and nothing more. Others — heavily used sources, a flagship correspondence series, records where the contents are the point — justify full-text treatment. Most institutions end up doing both, at different depths for different material. This is part of the wider set of choices covered in archives, digitization, and metadata workflows, and it pays to make the decision deliberately rather than by default.

If you decide a body of material warrants full-text search, the work moves through five reasonably standard stages. Each constrains the next, which is why the whole thing is best understood as a single archival digitization workflow rather than a set of unrelated tasks.

You produce image derivatives from the physical or microfilm originals. Everything downstream is limited by what happens here: microfilm of nineteenth-century newsprint is a notorious constraint, and no text-recognition model recovers detail the image never captured. Leo, it is worth saying plainly, begins after this stage — it works from images you supply, not from a scanning rig.

This is where the print-versus-handwriting distinction bites, and where the wrong tool quietly wastes the most effort. OCR (Optical Character Recognition) converts images of printed text into characters; HTR (Handwritten Text Recognition) does the same for manuscript hands. They are different technologies, and the difference matters more than most planning documents admit — the distinction between HTR and OCR is worth understanding before you commit a budget to either.

In parallel, you describe the material — DACS in the US, ISAD(G) internationally, increasingly the ICA's Records in Contexts — and encode it, typically in EAD for finding aids, wrapped in METS for the digital object, with PREMIS recording preservation events. How these descriptive, encoding, and preservation layers fit together without overloading any single schema is its own subject, treated in the guide to archival metadata standards.

A search engine — Solr, Elasticsearch, Blacklight — builds an inverted index over your recognised text. Fuzzy and n-gram matching lets it tolerate recognition errors, which matters more than it sounds (see below). If your images and text are exposed through IIIF, the IIIF Content Search API 2.0, released in 2022, lets a cross-institutional searcher query your text and highlight the hit region on the page image itself.

OAI-PMH exposes your metadata to aggregators such as Europeana and DPLA; IIIF delivers the images and text; authority identifiers link your people and places to records held elsewhere.

The stages Leo occupies are the second and, in part, the third and fourth: turning page images into faithful text, holding that text alongside its image in an organised workspace, and letting you search across it. The capture at stage one and the OAIS-grade packaging and IIIF delivery at the far end are handled by other systems. It is worth being precise about that boundary rather than blurring it.

Where the money and the frustration go: reading the page

Stage two is where most searchability projects succeed or fail, so it deserves the most attention.

For printed material, off-the-shelf OCR is fast, cheap, and — on clean, modern, high-contrast type — very good, with commercial engines reaching well above 95% character accuracy on good scans. The trouble is that production corpora are not clean modern type. Historical print breaks the assumptions OCR is built on: early-modern founts and blackletter, the long s read as an f, ligatures split or dropped, the macron standing for an omitted nasal simply discarded, uneven inking and show-through read as character evidence, archaic spelling "corrected" into something the printer never set. Empirical studies routinely find character-accuracy gaps of ten to thirty percentage points between clean modern print and eighteenth- or nineteenth-century historical print, with larger gaps still for early-modern type and Fraktur. Roslyn Holley's analysis of the National Library of Australia's newspaper programme found baseline OCR character accuracy averaging around 70–80%, with the worst titles under 40%.

For handwriting, conventional OCR does not work at all; you need HTR. And here the standard tools carry a cost that planning often overlooks: specialist HTR systems typically expect you to train or fine-tune a model on your own material before they perform well on it. Transkribus documentation recommends on the order of 25–50 manually transcribed pages as a minimum starting set for a given hand, with real gains up to several hundred. That ground truth has to be produced by hand first — which, for an archive with many hands across many collections, is a substantial and recurring commitment before a single new page becomes searchable. Public pre-trained models exist for common hands, but they thin out fast for unusual or rapid ones.

There is a persistent temptation to shortcut all of this by pasting a page into a general-purpose chatbot. Resist it for primary transcription. Large language models produce fluent, confident, plausible output — and on a degraded or difficult hand, that fluency is precisely the danger: they invent and omit text and present the fabrication as smoothly as the truth. A specialist transcription error is usually a wrong character you can catch against the image; an LLM's error is a well-formed sentence that was never on the page. For a searchable corpus that researchers will cite, the second kind of error is far more corrosive, because it is far harder to see.

The accuracy target depends on what "searchable" means

One of the more useful findings in this field is that search tolerates more error than a scholarly edition does. Holley's work and the studies after it show that keyword search can succeed even at fairly high miss rates, because users tolerate imperfect recall as long as they can visually verify a hit against the page image. Where full recognition accuracy is genuinely out of reach, keyword spotting — matching a query term against the image without a full transcription — can make a collection discoverable at a lower quality threshold.

The practical consequence: the right accuracy target is set by the downstream use. If the goal is a critical edition, character-level fidelity matters and the source integrity of the transcription is paramount. If the goal is "let a family historian find a surname somewhere in ten thousand pages," a noisier index that surfaces candidate pages for visual confirmation may be entirely adequate. Decide which you are building before you decide how much accuracy to pay for.

A related point, often missed: you do not have to wait for complete text before you publish anything. Partial coverage is publishable. Incremental HTR pipelines and IIIF Content Search let an institution release searchable content as it is produced and re-run better models later. A collection that is 40% searchable today is more useful than one that is 0% searchable while you wait for a complete transcription.

Where a zero-shot transcription model changes the calculation

The training bottleneck described above is the reason many archives never get past image-only "digitisation" for their handwritten holdings. Producing ground truth for every hand is exactly the labour that MPLP warned was unsustainable at scale. This is the specific stage where it is worth knowing that Leo exists.

Leo's transcription engine, ATR-1, is a specialist model for Latin-script material that runs zero-shot — out of the box, with no per-collection model to train first. The constraint is the writing system, not the language: it reads any language written in the Latin alphabet — English wills, French notarial records, Dutch registers, German parish books, Spanish and Italian correspondence — and handles both manuscript hands and printed matter, including the early-modern typography that trips conventional OCR. (Leo's own public corpus, ExLatinis, is a Latin-language project, but Latin is one case among the many vernaculars the model reads, not the point of it.) What it does not read is non-Latin scripts — Greek, Cyrillic, Hebrew, Arabic, and the East Asian and Indic writing systems.

On a randomised 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, ATR-1 scored roughly a 5% character error rate at release — about 61% fewer errors than the next-best model tested, with Transkribus's Text Titan I around 13% and the general LLMs ranging from the low twenties to well over fifty percent; the full benchmark data is public. The design principle behind that figure is source integrity: ATR-1 transcribes what is on the page, preserving strikethroughs, marginalia, expansions, and archaic spelling rather than smoothing them into modern prose. Around the model sits a document workspace — folders, per-document metadata on a fixed Dublin-Core-adjacent field set, image shown beside transcription, fuzzy search across everything, and export to TEI, Word, PDF, or HTML — which covers the transcribe-organise-search stretch of the pipeline without the model-training step. For an academic or archival project, the Leo Transcription Grant offers substantial free credits in exchange for publishing the resulting transcriptions openly; if your material is chiefly historical print, note the grant does not cover it, and the free or paid plans are the route instead.

None of this removes the need for verification. Any machine transcription — Leo's included — is a draft to be checked against the image, and the closer your target is to a citable edition, the more checking it needs. What a zero-shot model changes is the starting cost: it moves an archive from "produce hundreds of ground-truth pages before anything is searchable" to "get a usable draft on the first page and correct from there."

Deciding what to make searchable, and how deeply

Pulling the threads together, a workable plan for an archive looks less like "digitise everything and OCR it" and more like a set of deliberate choices:

Triage by use, not by shelf order. Aggregate description for the whole; full-text treatment reserved for material where the contents are what people search.

Match the recognition tool to the material. OCR for clean print, HTR for handwriting and historical typography, and an honest accounting of the training cost before committing to a tool that demands it.

Set the accuracy target from the downstream use. A finding aid, a searchable index, and a critical edition are three different jobs with three different tolerances.

Publish incrementally. Release what is searchable now; improve it as better models and more corrections arrive.

The goal is not a uniform, fully transcribed collection — that standard defeats most institutions before they start. The goal is that a researcher who needs something in your holdings can find it at the level their question requires, and confirm it against the page. An archive that gets that right has done the real work of access: it has closed the distance between a document existing and a document being read.

Making archives searchable starts with deciding which of two problems you are solving: helping a user find the right box, file, or series through a descriptive finding aid, or helping them find a word or phrase inside the documents themselves. The first is metadata searchability, produced by descriptive cataloguing to standards like DACS and encoded in EAD. The second is full-text searchability, produced by running page images through OCR for print or handwritten text recognition for manuscript. They use different tools, cost different amounts, and answer different questions. Choose the level your collection actually needs before choosing any software.

What is the difference between OCR and HTR?

OCR (Optical Character Recognition) converts images of printed text into machine-readable characters; HTR (Handwritten Text Recognition) does the same for manuscript hands. They are different technologies, and matching the wrong one to your material quietly wastes effort. Off-the-shelf OCR is fast, cheap, and very good on clean modern print, but conventional OCR does not work at all on handwriting. Historical typography — early-modern founts, blackletter, the long s — also breaks the assumptions OCR is built on, causing large accuracy gaps compared with clean modern type. Understand which technology your material demands before committing a budget to either.

Why not use ChatGPT to transcribe historical documents?

Avoid general-purpose chatbots for primary transcription. Large language models produce fluent, confident, plausible output, and on a degraded or difficult hand that fluency is precisely the danger: they invent and omit text while presenting the fabrication as smoothly as the truth. A specialist transcription error is usually a wrong character you can catch against the image; an LLM's error is a well-formed sentence that was never on the page. For a searchable corpus that researchers will cite, the second kind of error is far more corrosive, because it is far harder to see. Use a specialist transcription model instead.

How accurate does transcription need to be for full-text search?

Search tolerates more error than a scholarly edition does, so the right accuracy target is set by the downstream use. Keyword search can succeed even at fairly high miss rates, because users tolerate imperfect recall as long as they can visually verify a hit against the page image. If the goal is "let a family historian find a surname somewhere in ten thousand pages," a noisier index that surfaces candidate pages for confirmation may be adequate. If the goal is a critical edition, character-level fidelity matters and source integrity is paramount. Decide which job you are building before deciding how much accuracy to pay for.

Does Leo's ATR-1 need training on my collection before it works?

No. ATR-1 runs zero-shot — out of the box, with no per-collection model to train first — which removes the usual bottleneck where specialist HTR systems expect you to hand-transcribe dozens or hundreds of ground-truth pages before they perform. The constraint is the writing system, not the language: it reads any language written in the Latin alphabet, including English, French, Dutch, German, Spanish, and Italian, and handles both manuscript hands and printed matter, including early-modern typography. It does not read non-Latin scripts such as Greek, Cyrillic, Hebrew, Arabic, or East Asian and Indic writing systems. Any machine transcription remains a draft to be checked against the image.

Separates metadata searchability from full-text search, then walks through the pipeline — digitisation, OCR vs HTR, indexing — so you can plan the right level for your holdings.

Making Archives Searchable: A Planning Guide

Digitizing a collection is only half the job; making it searchable and well-described is the rest. This hub covers workflows from images to transcribed, organized records and where Leo fits in a wider digitization pipeline.

Archives, Digitization & Metadata Workflows

Making Archives Searchable: A Planning Guide for Cultural Heritage Professionals

This is a working guide to the archival metadata standards that govern transcribed documents — what each one describes, which layer it belongs to, and how to combine them without asking one standard to do another's job. For cultural heritage professionals making holdings findable and durable, getting these layers straight is the difference between a record that lasts and one that quietly loses information.

Archival metadata is not a single standard but a layered ecosystem, and transcribed documents sit at the intersection of several layers at once. The essential distinction is functional. Description records what a document is — its provenance, creator, place in a collection. Transcription encoding records what it says — the text, with its deletions and expansions. Preservation metadata records how the digital object is maintained over time. No one schema handles all three. Choose standards by the concern you are actually addressing, and expect to combine descriptive standards like DACS or EAD3, an encoding standard like TEI P5 for the transcription itself, and PREMIS for preservation — each doing the job it was designed for.

This guide walks through those layers in the order they matter for a transcribed document, names the standard that owns each one, and flags the misconceptions that most often send projects down the wrong path. It sits within the broader practice of digitizing and describing collections, where metadata choices constrain — and are constrained by — every other step in the workflow.

Why "one standard for everything" is the wrong mental model

The most persistent error in describing transcribed material is expecting a single schema to cover it end to end. It cannot, because the standards were built by different communities to solve different problems. The finding-aid tradition (ICA, SAA) describes archival hierarchy: fonds, series, file, item. The text-encoding tradition (the TEI Consortium) captures editorial phenomena down to the character. The preservation community (Library of Congress, ISO) tracks fixity, events, and provenance of the digital file. These are complementary layers, not competing options.

Two consequences follow. First, a mature description of a transcribed document routinely uses three or four standards at once — and that is correct practice, not redundancy. Second, most of the recurring confusion in the field comes from treating a standard as if it belonged to a layer it does not. The sections below are organised by layer for exactly this reason.

Layer one: describing the document (descriptive standards)

Descriptive standards answer what is this, who made it, where does it sit in the collection. They govern the finding aid and the catalogue record.

ISAD(G) — the ICA's General International Standard Archival Description (2nd ed., 2000) defines 26 elements across seven information areas. ISAAR(CPF) (2nd ed., 2003) is its companion for authority records describing the creators of records — corporate bodies, persons, families.

DACS — the Society of American Archivists' Describing Archives: A Content Standard, the US content standard aligned with ISAD(G) and ISAAR(CPF); the 2022 version is current.

RAD — Library and Archives Canada's Rules for Archival Description, the Canadian equivalent.

Records in Contexts (RiC) — the ICA's successor conceptual model. RiC-CM v1.0 was published in 2023, with the RiC-O ontology reaching v1.1 in May 2025. RiC reconceives archival description as linked data rather than a hierarchy of finding aids.

A word of caution on RiC. It has, at the conceptual level, superseded ISAD(G) and ISAAR(CPF), and legacy descriptions built on those standards remain perfectly valid. But production-level uptake is thin: the deployments that exist are pilots — Archives Portal Europe, national-archive research projects — and no industry-wide adoption measurement is available. Treat RiC as the direction of travel, not as a requirement your institution has fallen behind on.

Layer two: encoding the description (structural encoding standards)

Where the descriptive standards above define content, encoding standards define the machine-readable structure that carries it.

EAD3 (v1.1.2, released June 2023, maintained by the Library of Congress with the SAA) is the XML finding-aid format derived from ISAD(G). EAC-CPF 2.0 (2022) is its authority-record counterpart, encoding creators.

MARC21 / MARCXML and MODS (v3.8) serve the bibliographic side; MODS is the richer XML schema.

METS binds descriptive, structural, and administrative metadata to the actual files — a wrapper rather than a description standard in its own right.

On EAD2002-to-EAD3 migration: no consolidated statistic exists. Repository disclosures suggest large national institutions have migrated while smaller repositories and some aggregators remain on EAD2002. If you are a small institution still on EAD2002, you are in ordinary company.

Layer three: encoding the transcription itself (TEI, ALTO, PAGE)

This is the layer that most directly concerns transcribed documents, and the one where the description-versus-transcription distinction earns its keep. EAD describes the collection; TEI encodes what the page actually says. They are complementary, not interchangeable — a point worth stating plainly, because conflating them is one of the most common mistakes in the field.

TEI P5 — the Text Encoding Initiative's P5 Guidelines — is the scholarly standard for the transcription text. It is an XML vocabulary for editorial phenomena at character level: deletions, additions, expansions of abbreviations, unclear readings, and manuscript description through the `<msDesc>` element. If you are producing a scholarly edition of a transcribed manuscript, TEI is where the editorial intelligence lives. Its cost is a genuine learning curve: the schema is large, projects typically use only a subset, and inter-edition inconsistency is a well-known consequence. Tooling like oXygen mitigates this but does not remove it.

ALTO XML and PAGE XML are a different animal, and the confusion here is worth correcting directly: neither is a description standard. ALTO (Library of Congress) is a layout format holding word- and line-level bounding-box coordinates alongside recognised text — it is OCR/HTR output, not a description of archival material. PAGE XML (PRImA Research, University of Salford) plays the equivalent role for HTR ground truth and layout export. Both answer where on the image is this text and what does it read, not what is this document.

The practical upshot: a single transcribed page can legitimately generate a TEI edition (for scholarship), an ALTO or PAGE file (as machine output preserving coordinates), and an EAD entry (in the finding aid) — three standards for one page, each serving a different consumer. No consolidated registry tracks which projects use TEI editions versus machine outputs for the same material, so do not assume one implies the other.

Where the transcription comes from — and why the source text matters here

Before any of this encoding applies, the transcription has to exist and has to be faithful. This is where the choice of transcription method has downstream metadata consequences that are easy to underestimate. A metadata layer is only as trustworthy as the text it describes, and the difference between OCR and HTR is not academic when the source is a manuscript hand or historical print — the distinction between the two determines what actually lands in your TEI `<supplied>` and `<abbr>` elements.

The specific risk is silent normalisation. General-purpose OCR is engineered for clean modern type; on early-modern material it tends to read the long s as f, split or drop ligatures, discard scribal and typographic abbreviation marks, and "correct" archaic orthography that was never an error. General-purpose LLMs introduce a subtler and more dangerous failure: they produce fluent, plausible readings that are wrong, and plausibility is precisely what makes a fabricated transcription hard to catch in review. Either failure mode corrupts your metadata at the source — you end up encoding, describing, and preserving a text that does not match the page.

This is where a purpose-built transcription tool earns its place in the pipeline. Leo's ATR-1 model is trained on images of historical documents to transcribe what is on the page rather than smooth it into modern prose: it preserves strikethroughs, marginal additions, editorial expansions, and archaic spelling rather than resolving them silently — which is exactly the material your TEI encoding needs to represent faithfully. On a randomised 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, ATR-1 scored roughly 5% character error rate — 61% fewer errors than the next-best model tested — with the full comparison across Transkribus, Claude, Gemini, and GPT documented in the benchmark. Leo exports to TEI (XML) directly, alongside Word, PDF, and HTML, so the faithful base transcription carries forward into the encoding layer rather than being retyped. It does not, however, produce ALTO or PAGE output, nor does it handle the IIIF or preservation layers below — those remain the province of the tools built for them. As users correct transcriptions, those corrections feed back into the model, so accuracy improves across releases. The point stands regardless of method: the text entering your metadata stack should be verified against the page.

Layer four: preservation metadata (OAIS and PREMIS)

Preservation is a distinct concern with its own standards, and one misconception here has real consequences: a transcription derivative needs preservation metadata of its own. When a transcription is a separately managed digital object, it is itself an Archival Information Package — it requires metadata describing its representation, the events that produced it (the HTR run, a fixity check), and its provenance.

OAIS (ISO 14721, third edition ISO 14721:2025) is a reference model, not a metadata schema. It defines roles (Producer, Management, Consumer) and information packages (SIP, AIP, DIP). You implement its concepts; you do not "fill in OAIS fields." The 2025 edition introduces preservation-watch concepts, though vendor conformance documentation has not yet caught up.

PREMIS — the Preservation Metadata: Implementation Strategies Data Dictionary, v3.0 (2015, still current) — is the schema that implements preservation metadata in practice, covering objects, events, agents, and rights.

MIX (NISO Metadata for Images in XML, v2.0) holds technical metadata for the image files themselves.

For a transcription pipeline, the practical rule is: record the provenance of the transcription as an event. Who or what produced it, when, from which source image, and against which model or process — this is exactly the information a future user needs to judge whether to trust the text.

Layer five: controlled vocabularies and dates

Interoperability rises sharply when names, places, subjects, and dates are drawn from shared authorities rather than free text.

Names and subjects: LCNAF (over 11 million name authority records), VIAF (aggregating national-library authority files), and LCSH for subjects.

Places, concepts, and artists: the Getty vocabularies — AAT (over 500,000 terms), TGN (over 2.7 million places), and ULAN. Known coverage gaps for non-Western, pre-modern, and indigenous material are real and not formally documented.

Languages: ISO 639, which is not one standard but several parts — ISO 639-1 (two-letter), 639-2 (three-letter), 639-3 (all languages), 639-5 (families). Pick the part your use case requires.

Dates: for historical material this matters more than it first appears. ISO 8601 handles ranges and some approximation, but the cultural-heritage community typically adopts EDTF — the Extended Date/Time Format (LC, 2019) — which extends ISO 8601 with qualifiers for uncertain and approximate dates: the `?` for uncertainty, `~` for approximation, and masked digits for unknown values. For a will "dated 159[?]" or a letter "circa 1640," EDTF lets you record the uncertainty as data rather than losing it to a guess.

The final layer moves description and images outward, to users and aggregators.

IIIF — the International Image Interoperability Framework — defines APIs for image delivery, presentation manifests, and content search (Image, Presentation, and Content Search APIs, all at v3.0 / 2.0). A frequent misconception is that IIIF stores metadata; it does not. It delivers images and references your descriptive metadata, which still lives in EAD, MODS, or TEI. IIIF is well established — the consortium reports over 130 institutions implementing it — though Content Search API uptake specifically is uneven.

Aggregator profiles: the DPLA Metadata Application Profile and Europeana's EDM, harvested via OAI-PMH. DPLA aggregates over 53 million items; Europeana over 58 million.

Here lies the most consequential misconception of all: Dublin Core is not full description. The 15-element Dublin Core set is a deliberate minimum for cross-domain discovery and harvesting — exactly right for feeding an aggregator, wrong as your primary descriptive standard. When EAD3 or MODS is flattened to Dublin Core for harvesting, archival-specific granularity collapses, typically into a single `dc:description`. The qualitative loss is well documented; no published study quantifies it. The rule: describe richly in EAD/MODS/TEI, and map down to Dublin Core for discovery — never the reverse.

A single transcribed manuscript, fully handled, touches every layer: a DACS-compliant description encoded in EAD3; a TEI P5 edition of the transcription with its expansions and deletions marked; possibly an ALTO or PAGE file preserving the machine output and coordinates; PREMIS metadata documenting the transcription as a preserved object with its own provenance; authority-controlled names and EDTF-qualified dates; and IIIF delivery with a Dublin Core crosswalk for the aggregators. That is not over-engineering. It is what it takes to make a transcribed document findable, citable, trustworthy, and durable.

Most institutions will not implement all six layers at full depth, and they do not need to. Small repositories often implement the descriptive and discovery layers well and rely on national or aggregator infrastructure for preservation — a reasonable division of labour. The discipline that matters is knowing which layer a given standard belongs to, so that you never ask Dublin Core to do TEI's job, or expect IIIF to be your catalogue. Get the layers straight, keep the transcription faithful to the page, and the standards stop competing and start doing the work each was built for.

What are the archival metadata standards for transcribed documents?

Transcribed documents draw on a layered set of standards rather than one schema, because each layer answers a different question. Descriptive standards — ISAD(G), DACS, RAD, or the newer Records in Contexts — record what a document is and where it sits in a collection. Encoding standards like EAD3 and MODS carry that description in machine-readable form. TEI P5 encodes the transcription text itself, while ALTO and PAGE hold machine output and coordinates. PREMIS handles preservation, controlled vocabularies govern names and dates, and IIIF plus Dublin Core manage delivery and discovery. Each does the job it was built for.

What is the difference between description and transcription in archival metadata?

Description records what a document is — its provenance, creator, and place in a collection — while transcription encoding records what it says, the actual text with its deletions, additions, and expanded abbreviations. These are complementary layers, not interchangeable ones. EAD describes the collection; TEI P5 encodes what the page reads at character level. Conflating them is one of the most common mistakes in the field. A single transcribed page can legitimately have both: an EAD entry in the finding aid and a TEI edition of its text, each serving a different consumer.

Is Dublin Core enough to describe an archival collection?

No — Dublin Core is a deliberate minimum for cross-domain discovery and harvesting, not a primary descriptive standard. Its 15-element set is exactly right for feeding an aggregator but wrong as your main description. When richer formats like EAD3 or MODS are flattened to Dublin Core, archival-specific granularity collapses, often into a single `dc:description` field. The working rule is to describe richly in EAD, MODS, or TEI and map down to Dublin Core for discovery — never build up from Dublin Core as your foundation. Doing the reverse loses information that the fuller standards were designed to preserve.

Do ALTO and PAGE XML describe archival documents?

No — neither ALTO nor PAGE is a description standard. ALTO, maintained by the Library of Congress, is a layout format holding word- and line-level bounding-box coordinates alongside recognised text; it is OCR/HTR output. PAGE XML, from PRImA Research at the University of Salford, plays the equivalent role for HTR ground truth and layout export. Both answer where on the image is this text and what does it read — not what is this document. That descriptive question belongs to EAD, MODS, or DACS. Treating a machine-output format as a description standard is a common and correctable mistake.

Does a transcription need its own preservation metadata?

Yes — when a transcription is a separately managed digital object, it is itself an Archival Information Package and needs preservation metadata of its own. It requires records describing its representation, the events that produced it — such as the HTR run and fixity checks — and its provenance. PREMIS is the schema that implements this in practice, covering objects, events, agents, and rights, while OAIS provides the reference model behind it. The practical rule for a transcription pipeline is to record its provenance as an event: who or what produced it, when, from which source image, and against which model — the information a future user needs to judge whether to trust the text.

A layered guide to the archival metadata standards that govern transcribed documents - from DACS and EAD3 to TEI P5, PREMIS, and IIIF - and how to combine them correctly.

Archival Metadata Standards for Transcriptions

Archival Metadata Standards for Transcribed Documents: A Layered Guide

This is a working guide to the archival digitization workflow — the full sequence from selection through access, and where each stage constrains the next. It is written for cultural heritage professionals who need to plan a program against real materials, real standards, and real costs, not a rate sheet. The aim is to help you design a chain of dependencies that holds, so a decision made early does not surface as a permanent defect later.

An archival digitization workflow is a gated sequence: selection and planning, preparation and handling, image capture, quality control, image processing, metadata and description, text recognition, storage and preservation, and finally access and publication. Each stage constrains the next. Capture cannot recover what preparation damaged, and text recognition cannot read what capture failed to resolve. The most common mistake is treating scanning as the whole job. Scanning produces image files; digitization produces described, preserved, discoverable records. This guide walks the full pipeline and points out where the gates actually bind.

Most of the failures I see in the field trace back to a decision made two stages earlier and never revisited. So it is worth understanding the workflow as a chain of dependencies rather than a checklist. This piece sits within our broader work on archives, digitization, and metadata workflows, and it aims to give a working professional something to plan against.

Before an object reaches a scanner, someone has decided it is worth digitizing and to what standard. This is where you commit to a target. The target governs cost, equipment, and schedule for everything that follows.

Two imaging standards dominate cultural-heritage practice. In North America, the FADGI Technical Guidelines define a four-star system, where 4-star is the strictest tier — roughly 600 ppi for reflective text, 16 bits per channel, an ICC-managed color workflow, verified against physical targets. In Europe, the Metamorfoze Preservation Imaging Guidelines define three levels — Full, Light, and Extra Light — around a single governing principle: all visible information in the original must be visible in the master.

Institution size tends to sort against these tiers. National libraries and large archives run FADGI 4-star or Metamorfoze Full programs with formal QA and dedicated infrastructure. Mid-size repositories often target 3-star or Light and rely on vendor processing. Small historical societies frequently default to 2-star or vendor-conformant capture, managing metadata in AtoM or Omeka S and delegating preservation to a regional consortium.

Be honest about cost at this stage, because the published figures will not settle it for you. University capture rates run around $1.00 per image for 300 ppi grayscale and $2.00 for 600 ppi color at one public digitization center; vendor page scanning sits in the $0.05–$0.25 range. There is no peer-reviewed, cross-institutional cost benchmark with comparable methodology — the numbers you find are project-specific or vendor-quoted. Plan against your own pilot, not someone else's rate sheet.

Preparation is conservation work, not clerical work. Fragile bindings, tight gutters, brittle paper, and light-sensitive media each dictate how — and whether — an item can be captured. This stage sets the physical constraints capture must respect, and getting it wrong is the one failure the entire workflow cannot undo.

Capture hardware follows the material. Book and planetary scanners (ATIZ, Bookeye, Indus) handle bound volumes where opening angle is constrained. Copy-stand DSLR or mirrorless rigs with macro lenses handle oversize and special-collections material. Flatbeds remain standard for loose paper at small scale, and sheet-fed production scanners drive high-volume backfile conversion.

Whatever the rig, the discipline is the same: capture software writes to raw TIFF in checksum-stamped ingest directories, and you image your targets at the start and end of each session. A color target such as an X-Rite ColorChecker or a Golden Thread target verifies color accuracy; a resolution target verifies MTF and sampling efficiency. Those target images are your evidence that the session met its tier. Without them, QC has nothing to measure against.

FADGI and Metamorfoze both require inspection of a defined sample against pass/fail gates, typically expressed as a percentage or an AQL formula. The quantitative checks are specific: modulation transfer function from slanted-edge targets, color accuracy against chart-based ΔE, tonal response, noise, geometric distortion, and visual completeness. FADGI provides OpenDice and AutoSFR for MTF measurement.

The tolerances are concrete. Metamorfoze v2.0 sets white-balance ΔE at or below 3 for Full and Light, and color accuracy at a mean ΔE of 3 or less (max 7) for Full. QC is the gate before publication for a reason: a color cast or a soft focus caught here is a reshoot; caught after the master is archived and the original reshelved, it is a permanent defect in the record.

Processing derives from the master, never the reverse. This is the deliberate split at the center of a sound workflow: the master file is uncompressed or lossless — TIFF 6.0 or lossless JP2, 16 bits per channel, color-managed — and every access derivative (JPEG, lossy JP2, PDF/A) is produced from it. Deskew, cropping, and color correction happen on copies. The master stays untouched.

The format trade-offs are qualitative, not settled by a single benchmark. Uncompressed TIFF is lossless, universally supported, and large — a 600 ppi 24-bit scan of a letter-size page can exceed 100 MB. Lossless JPEG2000 offers comparable fidelity at roughly 20–40% smaller files, with more implementation complexity. PDF/A bundles text and image and is ISO-standardized for long-term preservation. FADGI and Metamorfoze both accept TIFF and lossless JP2 as masters; the choice is yours to defend.

No single metadata standard covers everything, and the belief that one does is a reliable source of stalled projects. Metadata is layered because it answers different questions, and the layers are usually bundled in a wrapper such as METS.

Descriptive — what is it? Dublin Core (a 15-element core), MODS (richer bibliographic), EAD3 (hierarchical finding aids), and DACS or ISAD(G) as content rules.

Structural — how do the parts fit? METS as the wrapper, ALTO XML for OCR layout and text per region, PAGE XML for richer page-level zones.

Technical — what is the file? MIX, EXIF, capture-device logs.

Preservation — what happened to it, and under what rights? The PREMIS Data Dictionary v3, organized around Objects, Events, Rights, Agents, and now Intellectual Entities.

Larger institutions typically combine ArchivesSpace for finding aids with a DAM such as Preservica or Rosetta for digital objects. Smaller ones lean on AtoM as a single system or Omeka S for cultural-object collections. A researcher's tool like Tropy organizes photographs taken on an archival visit — useful, but not an institutional repository, and it does not describe to these standards.

Here the workflow turns an image into something you can search, and here the assumptions that carried you through capture quietly stop holding. Two distinct technologies live under this stage, and conflating them causes real damage.

OCR (optical character recognition) classifies glyph patches against learned representations, assuming consistent, clean glyph shapes. On clean modern print, top engines — Tesseract LSTM, ABBYY FineReader, Google Cloud Vision — reach character error rates around 1–2%. That is genuinely strong, and if your collection is modern typescript, conventional OCR is the right and cheap answer.

HTR (handwritten text recognition) uses sequence models trained on ground truth, and the unit of training is the hand, not the corpus. That distinction matters more than any marketing figure. Our fuller treatment of the difference between HTR and OCR is worth reading if your collection mixes the two, as most do.

Two things break the naive plan. First, historical print is not modern print: the long s read as f, ligatures split or dropped, blackletter and Antiqua on the same page, ink bleed-through and scanning noise read as character evidence. On 19th-century newspapers, general OCR degrades into the 8–20% CER range depending on engine and condition — a far cry from the 1–2% on clean modern text. Second, handwriting is largely unusable to conventional OCR without model training, and specialist HTR platforms are not plug-and-play: Transkribus guidance places usable searchable text in the 5–10% CER band and publication-ready below roughly 5%, but reaching those numbers on a given hand typically means transcribing 15–30 accurate pages per hand as a training minimum, with diminishing returns past 50–100 pages and weak cross-hand generalization. For a collection with many scribal hands, that training cost is not marginal. It is the project.

There is a further trap, and it is the dangerous one. Reaching for a general chatbot to read a manuscript feels efficient, and a 2025 benchmark did find generalist models competitive with specialist systems on some corpora in zero-shot — but the same study, evaluated on a single set of corpora and still requiring human verification, found no generalist model reliably surpassed a well-trained specialist on the hardest scripts, with documented fabrication on degraded input. The mechanism is worth naming: a large language model is trained to produce fluent text, so given an unfamiliar or noisy image it will generate plausible, well-formed prose that is not what the page says. OCR errs on characters, which you can catch. An LLM errs with fluent fabrication, which reads perfectly and is wrong. For an archive whose entire value is fidelity to the source, that is the failure mode to design against.

Where a specialist model fits without a training project

This is the stage where an archive's plan most often stalls, and it is the stage Leo is built for. Leo's transcription model, ATR-1, reads Latin-script material — handwritten or printed — from roughly the past five centuries, in whatever language the Latin alphabet was used to write it: English wills, French notarial records, Dutch registers, German parish books, and Latin among them, not only Latin. It runs zero-shot, out of the box, with no per-hand training set to build first. For a collection of mixed hands, that removes the 15-to-30-pages-per-hand tax that makes specialist HTR a project in its own right.

The design commitment that matters at this stage is source integrity: ATR-1 transcribes what is on the page rather than smoothing it into modern, plausible prose. It preserves strikethroughs, marginal additions, editorial expansions, and archaic orthography — the long s stays a long s, u/v and i/j interchange survives, an abbreviation mark is not silently resolved. On a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, at ATR-1's release it recorded roughly a 5% character error rate — 61% fewer errors than the next-best model tested (Transkribus/Text Titan I at about 13%, and general LLMs considerably higher), per the published benchmark. That is a manuscript figure; it is not a claim about printed material, where the argument is mechanical rather than numerical — a model trained on images of historical documents weighs visual evidence against context instead of matching glyphs to a modern type model.

Around the model sits a workflow: upload (including straight from a phone camera in the reading room), transcribe, edit in a rich-text editor that keeps the original image beside the transcription, organize into folders, search with adjustable fuzzy matching, record structured metadata, and export to Word, PDF, HTML, or TEI XML. A separate AI layer can translate, summarize, or extract named entities — each writing to a new tab so the base transcription stays untouched. Where Leo does not go is equally worth stating plainly: it begins at upload, not capture, and it does not produce IIIF, OAIS packages, or EAD/METS/PREMIS — TEI XML is its scholarly export. It sits after capture and before your preservation and access infrastructure, and the corrections users make in the app feed back into training, so accuracy improves across releases.

For material eligible under an open license, academics and archives can apply for a Leo Transcription Grant of free credits; for printed-matter projects, which the grant does not cover, the free tier or a paid plan is the route in.

Digitization is not a one-time project, and this is the stage that proves it. The OAIS reference model (ISO 14721:2025) frames the archive's obligation as six mandatory responsibilities and three information packages — the SIP you ingest, the AIP you preserve, the DIP you disseminate. Underneath that sit continuous operational commitments: fixity checking with checksums, redundancy on a 3-2-1 or geographic pattern, format watch and migration, and storage refresh. The NDSA Levels of Digital Preservation give a graduated maturity framework to measure yourself against. Production systems that implement the OAIS model include Preservica, Rosetta, Archivematica, RODA, and the Internet Archive's stack.

The final stage makes the work usable, and here IIIF — the International Image Interoperability Framework — has become the de facto interoperability layer. Its open APIs let institutions serve high-resolution image tiles and structured digital objects, with viewers such as Mirador and the Universal Viewer rendering them for deep zoom, annotation, and cross-institution comparison. The IIIF Consortium, formed in 2015, counts around 71 member institutions as of 2025 — though membership counts what institutions have joined, not what share of the world's holdings is actually exposed through IIIF endpoints, which remains uncounted.

The discipline in all of this is remembering that the stages are gated, not parallel. A metadata schema chosen after capture forces rework; a QC standard set too loosely surfaces as a defect no derivative can repair; a text-recognition plan that assumes clean type meets a collection of secretary hands and stops dead. The strongest workflows are the ones where each stage is designed knowing what the next one needs — and where the person planning capture has already read a sample of the actual hands, seen the actual foxing, and priced the actual transcription. Digitization rewards that kind of looking ahead more than almost any craft I know.

What are the stages of an archival digitization workflow?

An archival digitization workflow is a gated sequence of nine stages: selection and planning, preparation and handling, image capture, quality control, image processing, metadata and description, text recognition, storage and preservation, and access and publication. Each stage constrains the next — capture cannot recover what preparation damaged, and text recognition cannot read what capture failed to resolve. The common mistake is treating scanning as the whole job. Scanning produces image files; digitization produces described, preserved, discoverable records. Design each stage knowing what the next one needs, because a decision made early can surface as a permanent defect later.

What is the difference between FADGI and Metamorfoze imaging standards?

FADGI and Metamorfoze are the two imaging standards that dominate cultural-heritage practice. FADGI, used in North America, defines a four-star system where 4-star is the strictest tier — roughly 600 ppi for reflective text, 16 bits per channel, an ICC-managed color workflow, verified against physical targets. Metamorfoze, used in Europe, defines three levels — Full, Light, and Extra Light — around a single governing principle: all visible information in the original must be visible in the master. Both require inspection of a defined sample against pass/fail gates and both accept TIFF and lossless JP2 as master formats.

OCR (optical character recognition) classifies glyph patches against learned representations, assuming consistent, clean glyph shapes; on clean modern print, top engines reach character error rates around 1–2%. HTR (handwritten text recognition) uses sequence models trained on ground truth, and the unit of training is the hand, not the corpus. Handwriting is largely unusable to conventional OCR without model training. Specialist HTR platforms are not plug-and-play either: reaching usable accuracy on a given hand typically means transcribing 15–30 accurate pages per hand as a training minimum, with weak cross-hand generalization. For collections with many scribal hands, that training cost is the project.

Why is using a general AI chatbot to transcribe manuscripts risky?

Using a general chatbot to read a manuscript is the dangerous trap in text recognition because of how these models fail. A large language model is trained to produce fluent text, so given an unfamiliar or noisy image it generates plausible, well-formed prose that is not what the page says. OCR errs on characters, which you can catch; an LLM errs with fluent fabrication, which reads perfectly and is wrong. A 2025 benchmark found generalist models competitive on some corpora in zero-shot, but no generalist model reliably surpassed a well-trained specialist on the hardest scripts, with documented fabrication on degraded input. For an archive whose value is fidelity to the source, that is the failure mode to design against.

Why should the master file never be edited during image processing?

The master file should never be edited because processing must derive from the master, never the reverse — this is the deliberate split at the center of a sound workflow. The master is uncompressed or lossless (TIFF 6.0 or lossless JP2, 16 bits per channel, color-managed), and every access derivative such as JPEG, lossy JP2, or PDF/A is produced from it. Deskew, cropping, and color correction happen on copies while the master stays untouched. This protects the archival record: a defect introduced by editing a master cannot be undone once the original is reshelved, so keeping it pristine preserves the option to regenerate derivatives later.

Nine gated stages from selection through access, where each decision constrains the next. Covers FADGI, Metamorfoze, QC, OCR vs HTR, and preservation planning.

Archival Digitization Workflow: A Stage-by-Stage Guide

Building an Archival Digitization Workflow: A Stage-by-Stage Guide