Topic Modelling Historical Corpora: When It Helps, and When It Misleads

This is a working guide to topic modelling historical corpora — what latent Dirichlet allocation and its successors can legitimately tell you about a body of manuscript or printed sources, and where the method quietly fails. It matters because the failures are invisible in the output: a noisy text layer, an undocumented preprocessing choice, or a document unit chosen by accident all produce topics that look perfectly clean on the page.

Topic modelling is an unsupervised method that infers clusters of co-occurring words across a corpus and reports each document as a mixture of those clusters. For historians it is useful as an exploratory instrument. It can point you toward where in a large, unread body of text a particular vocabulary concentrates, and how that concentration shifts across time or genre.

It misleads when its output is treated as evidence rather than as a finding aid — when a list of high-probability words is read as a theme the sources themselves assert, when the underlying text is too noisy or too short to support the inference, or when the preprocessing choices that determined the result go undocumented. The discipline is not in running the model. It is in knowing what the model is entitled to tell you.

What a topic actually is

Start with the formal definition, because most misreadings begin with a loose one. In latent Dirichlet allocation (LDA), the standard generative model, a topic is a probability distribution over a fixed vocabulary. It is not a theme. It has no name until you give it one. The label "MIDWIFERY" over a list beginning birth, deld, safe, morn, receivd, calld is your interpretive act, not the model's output.

The generative story LDA assumes, as Blei sets out in Communications of the ACM, runs like this: a fixed number of topics k exists for the corpus and is specified in advance; each document is a mixture of those topics; each word in a document is drawn from one of them.

Two assumptions do a great deal of work, and both are false of historical documents. The "bag of words" assumption holds that word order within a document does not matter. Exchangeability holds that the order of documents in the corpus does not matter either. A model that cannot see word order cannot see syntax, negation, irony, quotation, or the difference between a petitioner's claim and a clerk's summary of it. A model that treats document order as irrelevant cannot, by itself, see change over time; any chronological reading you produce comes from you sorting the document-topic weights by date afterwards.

This is not a defect to be apologised for. It is the price of the abstraction, and it defines the questions the method can answer.

The variants worth knowing

Several extensions relax specific assumptions. Choosing among them is a methodological decision, not a matter of taste.

Structural topic models (stm, Roberts, Stewart and Tingley, Journal of Statistical Software 2019) keep the LDA backbone but let document-level covariates — author, year, genre, place of publication — influence topic prevalence and content. For diachronic or regression-style questions, where you want to ask whether a topic's prominence changes with date while holding genre constant, stm is the right tool rather than LDA plus a spreadsheet.

Non-negative matrix factorization (NMF, as implemented in scikit-learn) factorises the term-document matrix directly rather than positing a generative process. It matters here for one empirical reason discussed below: it tends to be more stable than LDA on noisy text.

Polylingual LDA (Mimno et al., EMNLP 2009) models tuples of topically aligned documents in different languages with shared topic distributions. That is the honest answer to a corpus running across Latin, French and Dutch, rather than pretending a single bag-of-words vocabulary can absorb all three.

Embedding-based methods — BERTopic, which combines transformer embeddings, dimensionality reduction and class-based TF-IDF, and Top2Vec, which clusters joint word/document embeddings — replace the bag-of-words representation entirely. A comparative study on Twitter data by Egger and Yu in Frontiers in Sociology found BERTopic preferable on topic quality, with Top2Vec showing more topic evolution in dynamic settings. Be cautious carrying that finding into your own work: how pretrained encoders behave on historical orthography is still an open research question, not settled practice.

For readers who do not write code, MALLET remains the digital-humanities default, largely because the Programming Historian walkthrough made it the standard entry point. Voyant Tools offers a browser-based Topics panel at the lowest floor, though it gives corpus-level exploration rather than a configurable model.

When topic modelling genuinely helps

Three conditions mark the cases where the method earns its place.

The corpus is too large to read and roughly homogeneous in genre. Topic modelling is a way of allocating your attention. Cameron Blevins's work on the diary of Martha Ballard — twenty-seven years of entries from 1785 to 1812, modelled in MALLET at thirty topics — remains the clearest demonstration. The topics did not make an argument. They told him which entries clustered around midwifery, which around illness, which around gardening, and let him then read those entries closely. The model was a routing layer over ten thousand entries.

Orthography is unstable and you need to see past it. Ballard spells "daughter" fourteen recorded ways. A keyword search fails on that; a co-occurrence model can group daught, dagt and dat if they keep the same company. This is one of the few respects in which historical text is easier for the method than modern text is.

Your question is about distribution, not meaning. Where does this vocabulary concentrate? Which years, which authors, which sub-series? Those are questions about proportion, and proportion is exactly what the document-topic weights encode. "What did contemporaries mean by commonwealth" is not that question, and no topic model will answer it.

When it misleads

Coherence is not correctness

The most durable finding in the evaluation literature is that the model your metrics prefer may not be the model a human can read. Chang et al.'s "Reading Tea Leaves" introduced the word-intrusion test: insert a random word into a topic's top terms and ask annotators to spot it. On one low-precision topic, 87.5% of subjects flagged "taxis" as the intruder among otherwise political words. Their broader result is the one to internalise — traditional held-out likelihood was negatively correlated with the measures of topic quality they developed, and LDA outperformed pLSI and CTM on intrusion precision despite CTM's better likelihood.

Automated coherence scores improve on likelihood but do not replace human judgement. Lau, Newman and Baldwin sampled 600 topics with an average of 11.7 additional annotations each and found topic-level Pearson correlations between human word-intrusion and automatic NPMI/PMI scores in the range 0.554–0.648, against human inter-annotator agreement of 0.735–0.770. That is a useful correlation. It is not a substitute for looking. Report coherence if you like; do not let a single scalar stand in for having read the topics.

Neither stability nor reliability has been well validated for the newer neural topic models. Hoyle, Goel and Resnik found human annotators failing to meaningfully distinguish among them under certain evaluation setups. Novelty is not evidence.

Text quality is a precondition, not a polish step

This is where topic modelling on historical corpora most often goes quietly wrong, because the damage is invisible in the output. A model fed noisy transcription still produces clean-looking topics.

Mutuvi et al. (2018) tested this directly on roughly 12 million characters of English and French across four centuries, running fifty models per setting for k = 2–8. Average term stability of the top-10 terms fell from 0.265 on gold-standard text to 0.252 on raw OCR for LDA, and from 0.414 to 0.383 for NMF; mean topic coherence fell from 0.3622 to 0.3529 for LDA and from 0.4748 to 0.4720 for NMF. Two things follow. NMF was the more stable model throughout, which is a reason to prefer it on noisy input. And degradation is real but graded — the literature gives paired-condition comparisons, not a threshold, so nobody can tell you the error rate at which your model becomes untrustworthy. You have to judge that on your own material.

The Hall, Mernitz and Rensch study in Digital Scholarship in the Humanities modelled 1,797 German history textbooks from 1871–1918 with OCR applied and no manual correction, and assessors judged roughly two-thirds of the German topics excellent and over three-quarters good. The authors' own warning is the part worth carrying: individually excellent topics do not entail overall model quality. Good topics can coexist with a bad model, and a corpus of uncorrected text can still throw off plausible-looking clusters.

For handwritten sources the problem is more acute. You are not correcting a print-OCR baseline; you are generating the text layer from nothing, and the quality of that layer sets a ceiling on everything downstream. This is the point at which the choice of transcription method becomes methodological rather than logistical.

General-purpose chat models are the most common shortcut and the worst fit here. They tend to smooth archaic spelling into modern forms and to produce fluent readings of illegible passages, which is precisely the error mode a topic model cannot survive. Normalised orthography destroys the co-occurrence signal you were relying on, and fabricated passages inject vocabulary the source never contained. Garbled output is recoverable; plausible output is not.

Leo exists for this stage. Its transcription model, ATR-1, reads Latin-script manuscript and printed material — whatever the language on the page, English, French, German, Dutch, Spanish, Italian and Latin among them — and is trained to transcribe what is on the page rather than to normalise it, preserving archaic spelling, strikethroughs, marginalia and expansions. It runs zero-shot, with no per-corpus model training, and the transcriptions live in a workspace with folders, metadata and full-corpus search, exporting to TEI, Word, HTML or PDF for whatever pipeline you feed next. Its AI Transformations — modernisation, translation, entity extraction — each write to a separate tab, so a normalised layer never overwrites the diplomatic base text. For topic modelling that separation matters more than it might seem: you want the option to model both, and to know which one you modelled.

Preprocessing is the analysis

Stopword lists, lemmatisation, minimum document frequency, token length, the choice of k — these determine the result. Hall et al. report lemmatisation for historical languages as a persistent source of error in their German corpus. May, Cotterell and Van Durme found lemmatisation measurably improved interpretability in one configuration on morphologically rich Russian text, while concluding that further investigation was needed. Neither finding licenses a default. Both mean the choice must be recorded.

Nor does the method travel uniformly across languages. Without language-specific morphology handling, a bag-of-words representation splits inflectional variants into separate vocabulary items and dilutes every topic they belong to — which is precisely the problem polylingual LDA was introduced to make tractable, with word-alignment precision reported above 50% for Spanish, French and Italian at T = 400 and 800 on EuroParl and Wikipedia.

Short documents break the assumption

LDA assumes each document contains several topics. Diary entries, register lines, court minutes and marginal notes often contain one, or none — the same word rarely recurs within a single short document, so the co-occurrence signal the model depends on is absent. Albalawi, Yeap and Ghezala's comparison of LDA, BTM and NMF on 20-newsgroups (20,000 documents, average length 28 words) across k = 5, 10, 20, 50 supports no universal length threshold. The practical implication is that the unit of analysis is itself a decision: model per entry, per page, per month, or per volume, and expect different topics from each. Say which you chose.

A working protocol

The evidence converges on a short discipline, and it is not onerous.

1. Fix the text layer first. Know how your corpus was transcribed and how good it is. Sample and check a set of pages against the images before modelling anything.
2. Decide the document unit deliberately and state it.
3. Vary k and the random seed. Run the same configuration several times. Topics that survive re-seeding are candidates; topics that appear once are artefacts.
4. Hold out sub-corpora. If a topic only exists when a particular archive or decade is included, that is worth knowing before it becomes a claim.
5. Do intrusion-style checking on the topics you intend to use, in the manner of Chang et al. and Lau et al. It takes an afternoon and catches the topics you were going to over-read.
6. Close-read the documents with the highest weight on any topic before it enters an argument. This is the step Benjamin Schmidt's "Words Alone" and Ted Underwood's primer both insist on, and the one most often skipped.
7. Document every preprocessing choice in your methods note — stopwords, lemmatiser, frequency cutoffs, k, seeds, software version. Hall et al. identify the absence of exactly this reporting as what makes published topic-model results hard to trust or replicate.

What the model is for

Topic modelling belongs to the same family of instruments as a card catalogue or a well-made index: it tells you where to look. Treated that way, it is genuinely valuable on corpora no one could read entire, and it survives orthographic chaos better than string search does. Treated as an oracle — as a machine that reads the sources so you do not have to — it produces exactly what it was built to produce, a plausible arrangement of words, and offers no signal at all as to whether that arrangement means anything.

The distinction holds across the wider question of what computational methods can and cannot contribute to historical scholarship: the machine narrows the field; the historian makes the claim. A topic model that sends you to forty documents you would never otherwise have opened has done good work. What happens next — the reading, the context, the argument — is still the part that only you can do, and still the part your reviewers will judge.

Frequently Asked Questions

How do you do topic modelling on historical corpora?

Start with the text layer, not the model. Check how the corpus was transcribed and sample pages against the images before running anything. Then decide the document unit deliberately — per entry, per page, per month, per volume — because each produces different topics. Vary the number of topics k and the random seed across several runs; topics that survive re-seeding are candidates, topics that appear once are artefacts. Hold out sub-corpora to test whether a topic depends on one archive or decade. Do intrusion-style checking, close-read the highest-weighted documents before making any claim, and record every preprocessing choice in your methods note.

Does OCR or transcription quality affect topic modelling results?

Yes, and the damage is invisible in the output — a model fed noisy transcription still produces clean-looking topics. Mutuvi et al. tested this on roughly 12 million characters of English and French across four centuries and found term stability and topic coherence both fell measurably when moving from gold-standard text to raw OCR, for LDA and NMF alike. Degradation is graded rather than sudden, and the literature offers no threshold below which a model becomes untrustworthy, so you must judge it on your own material. For handwritten sources the stakes are higher: you are generating the text layer from nothing, and its quality caps everything downstream.

Should I use LDA, NMF, or BERTopic for historical texts?

It depends on the question and the state of your text. Standard LDA treats each document as a mixture of topics drawn from a fixed vocabulary. Structural topic models keep that backbone but let covariates like author, year or genre influence topic prevalence, which suits diachronic and regression-style questions. NMF factorises the term-document matrix directly and has proved more stable than LDA on noisy input, a reason to prefer it when transcription is imperfect. Polylingual LDA handles corpora spanning several languages. Embedding-based methods such as BERTopic and Top2Vec replace bag-of-words entirely, but how pretrained encoders behave on historical orthography remains an open question.

Can you topic model short documents like diary entries or court minutes?

Cautiously, and with the limitation stated. LDA assumes each document contains several topics; diary entries, register lines, court minutes and marginal notes often contain one or none. The same word rarely recurs within a short document, so the co-occurrence signal the model depends on is largely absent. Comparative work on LDA, BTM and NMF supports no universal length threshold, which means the unit of analysis is itself an analytical decision. Modelling per entry, per page, per month or per volume will give you different topics from the same corpus. Choose deliberately and say which you chose.

Is topic coherence a reliable measure of topic model quality?

No — coherence is useful but cannot replace human reading. Chang et al.'s word-intrusion work found that traditional held-out likelihood was negatively correlated with the measures of topic quality they developed, and that LDA beat models with better likelihood on intrusion precision. Lau, Newman and Baldwin found topic-level correlations between human intrusion judgements and automatic NPMI/PMI scores in the range 0.554–0.648, against human inter-annotator agreement of 0.735–0.770. That is a helpful correlation, not a substitute for looking. Report coherence if you wish, but do not let one scalar stand in for having read the topics.