Select all (Ctrl+A or Cmd+A), then copy. Back to the recipe.
# Building a Topic Index for a Set of Documents
## 1. What this is
You have a folder of documents. You want to answer questions about them without
reading every page every time. A topic index solves this. It is a lookup table
that says, for each topic, which pages of which documents mention it.
The index does not store what the documents say. It stores where to look. When
a question arrives, you look up the topic, get a list of pages, open those pages,
and answer from the original text. The index is a map, not a copy.
## 2. Definitions
**Collection.** The folder of documents you are indexing. Everything below is
scoped to one collection.
**Document.** One file in the collection. Usually a PDF or a spreadsheet.
**Page.** The unit of location. For a PDF it is a page number. For a spreadsheet
it is a tab name.
**Page image.** A rendered picture of one PDF page, or an HTML or CSV export of
one spreadsheet tab. You index from these, not from the original file, so that
a page number in the index always points to one specific image.
**Topic.** A short label naming something a reader might search for. Examples:
`revenue`, `pricing: by tier`, `termination clause`, `vendor: Acme`. A topic is
never a fact. `revenue` is a topic. `revenue was $49M` is not.
**Vocabulary.** The fixed list of topic labels you allow. Every topic in the index
comes from this list or follows the pattern `category: name` for named things.
**Document ID.** A short code you assign to each document so the index can refer
to it without repeating the file path.
## 3. What you will produce
Three kinds of JSON file in one folder.
### The registry: `_meta.json`
One file. Maps each document ID to the document.
```json
{
"collection": "Acme Contracts",
"updated": "2026-11-02",
"docs": {
"D001": {"path": "Master Services Agreement.pdf", "type": "pdf", "pages": 42},
"D002": {"path": "Pricing Model.xlsx", "type": "xlsx", "tabs": 5},
"D003": {"path": "Offer Letters/", "type": "group", "count": 31, "samples": ["D004","D005","D006"]}
}
}
```
### One per-document index: `<ID>.json`
One file per document. Maps each page to its topics.
For a PDF, keys are page numbers:
```json
{"id":"D001","pg":{"1":["parties","effective date"],"2-3":["scope of services"],"17":["termination clause","notice period"]}}
```
For a spreadsheet, keys are tab names:
```json
{"id":"D002","tb":{"Summary":["pricing: by tier"],"Assumptions":["pricing: discounts","volume tiers"]}}
```
For a group of similar documents, there is no page map. Just the topics the
group covers and which members you sampled:
```json
{"id":"D003","topics":["compensation","start date","at-will employment"],"samples":["D004","D005","D006"]}
```
### The master index: `_master.json`
One file. Every topic, and every place it appears.
```json
{
"updated": "2026-11-02",
"topics": {
"termination clause": [{"d":"D001","p":"17"},{"d":"D003","g":true}],
"pricing: by tier": [{"d":"D002","t":"Summary"}]
}
}
```
A location has a document ID `d` and one of: `p` for a page or page range,
`t` for a tab name, or `g: true` meaning "somewhere in this group."
## 4. The process
### Step 1. Inventory
List every file in the collection. Skip operating-system metadata files. For
each file record its relative path, file type, and page count or tab count.
### Step 2. Decide deep or shallow
For each document ask: does every page cover something different, or would a
few samples tell me what the whole set contains?
- Deep: each page is different. Contracts, reports, presentations, financial
statements. You will index every page.
- Shallow: the documents are repetitive. Thirty offer letters, forty
near-identical vendor agreements. You will index the set as one group, sample
three or four members, and index only those samples page by page.
### Step 3. Assign IDs and write the registry
Give every deep document and every shallow group an ID. Sampled members of a
group get their own IDs immediately after the group's ID. Write `_meta.json`
now, before reading any content, so the ID list is fixed.
### Step 4. Convert to page images
Render every deep document, and every sampled member of a shallow group, to one
image per page or one export per tab. Name them by page number.
### Step 5. Index each document
For each page image, write down the topics a reader would search for to find
that page. Apply three rules:
1. Use labels from the vocabulary. If a page covers a named entity, use the
pattern `category: name`.
2. Never put content in a label. Before saving, scan every label for numbers,
percentages, quotes, or clause summaries. If you find one, replace it with
the plain topic.
3. Include both the broad topic and the specific one. A page about revenue by
region gets both `revenue` and `revenue: by region`.
Merge adjacent pages into a range only when their topic lists are identical.
Save one compact JSON file per document.
### Step 6. Build the master index
Read every per-document file. For each topic on each page, add a location to
that topic's list in `_master.json`. Remove duplicates. Sort topics
alphabetically. Always rebuild this file from scratch from the per-document
files. Never edit it by hand.
### Step 7. Validate
Do not call the index done until all of these pass:
- Every ID in `_meta.json` that is not a group has a matching `<ID>.json` file,
and there are no extra files.
- No two documents share an ID.
- Every topic label is in the vocabulary or matches `category: name`.
- No label contains a dollar sign, percent sign, or quoted text.
- Every location in `_master.json` points to a document that exists in the
registry and a page that exists in that document.
## 5. Adding documents later
Inventory only the new files. Assign them the next free IDs. Add them to the
registry. Index them. Rebuild `_master.json` from all per-document files.
Nothing already indexed needs to be touched.
## 6. Using the index
To answer a question: open `_master.json`, find the relevant topics, collect
the locations, resolve each document ID to a path through `_meta.json`, open
those page images, and answer from what they say. Cite the file name and page
number. Never answer from the index alone.