Recipes

Open as plain text to copy the whole recipe Then select all and copy.

1. What this is

You have a folder of documents. You want to answer questions about them without reading every page every time. A topic index solves this. It is a lookup table that says, for each topic, which pages of which documents mention it.

The index does not store what the documents say. It stores where to look. When a question arrives, you look up the topic, get a list of pages, open those pages, and answer from the original text. The index is a map, not a copy.

2. Definitions

Collection. The folder of documents you are indexing. Everything below is scoped to one collection.

Document. One file in the collection. Usually a PDF or a spreadsheet.

Page. The unit of location. For a PDF it is a page number. For a spreadsheet it is a tab name.

Page image. A rendered picture of one PDF page, or an HTML or CSV export of one spreadsheet tab. You index from these, not from the original file, so that a page number in the index always points to one specific image.

Topic. A short label naming something a reader might search for. Examples: revenue, pricing: by tier, termination clause, vendor: Acme. A topic is never a fact. revenue is a topic. revenue was $49M is not.

Vocabulary. The fixed list of topic labels you allow. Every topic in the index comes from this list or follows the pattern category: name for named things.

Document ID. A short code you assign to each document so the index can refer to it without repeating the file path.

3. What you will produce

Three kinds of JSON file in one folder.

The registry: _meta.json

One file. Maps each document ID to the document.

{
  "collection": "Acme Contracts",
  "updated": "2026-11-02",
  "docs": {
    "D001": {"path": "Master Services Agreement.pdf", "type": "pdf", "pages": 42},
    "D002": {"path": "Pricing Model.xlsx", "type": "xlsx", "tabs": 5},
    "D003": {"path": "Offer Letters/", "type": "group", "count": 31, "samples": ["D004","D005","D006"]}
  }
}

One per-document index: <ID>.json

One file per document. Maps each page to its topics.

For a PDF, keys are page numbers:

{"id":"D001","pg":{"1":["parties","effective date"],"2-3":["scope of services"],"17":["termination clause","notice period"]}}

For a spreadsheet, keys are tab names:

{"id":"D002","tb":{"Summary":["pricing: by tier"],"Assumptions":["pricing: discounts","volume tiers"]}}

For a group of similar documents, there is no page map. Just the topics the group covers and which members you sampled:

{"id":"D003","topics":["compensation","start date","at-will employment"],"samples":["D004","D005","D006"]}

The master index: _master.json

One file. Every topic, and every place it appears.

{
  "updated": "2026-11-02",
  "topics": {
    "termination clause": [{"d":"D001","p":"17"},{"d":"D003","g":true}],
    "pricing: by tier":   [{"d":"D002","t":"Summary"}]
  }
}

A location has a document ID d and one of: p for a page or page range, t for a tab name, or g: true meaning "somewhere in this group."

4. The process

Step 1. Inventory

List every file in the collection. Skip operating-system metadata files. For each file record its relative path, file type, and page count or tab count.

Step 2. Decide deep or shallow

For each document ask: does every page cover something different, or would a few samples tell me what the whole set contains?

• Deep: each page is different. Contracts, reports, presentations, financial statements. You will index every page.
• Shallow: the documents are repetitive. Thirty offer letters, forty near-identical vendor agreements. You will index the set as one group, sample three or four members, and index only those samples page by page.

Step 3. Assign IDs and write the registry

Give every deep document and every shallow group an ID. Sampled members of a group get their own IDs immediately after the group's ID. Write _meta.json now, before reading any content, so the ID list is fixed.

Step 4. Convert to page images

Render every deep document, and every sampled member of a shallow group, to one image per page or one export per tab. Name them by page number.

Step 5. Index each document

For each page image, write down the topics a reader would search for to find that page. Apply three rules:

1. Use labels from the vocabulary. If a page covers a named entity, use the pattern category: name.
2. Never put content in a label. Before saving, scan every label for numbers, percentages, quotes, or clause summaries. If you find one, replace it with the plain topic.
3. Include both the broad topic and the specific one. A page about revenue by region gets both revenue and revenue: by region.

Merge adjacent pages into a range only when their topic lists are identical. Save one compact JSON file per document.

Step 6. Build the master index

Read every per-document file. For each topic on each page, add a location to that topic's list in _master.json. Remove duplicates. Sort topics alphabetically. Always rebuild this file from scratch from the per-document files. Never edit it by hand.

Step 7. Validate

Do not call the index done until all of these pass:

• Every ID in _meta.json that is not a group has a matching <ID>.json file, and there are no extra files.
• No two documents share an ID.
• Every topic label is in the vocabulary or matches category: name.
• No label contains a dollar sign, percent sign, or quoted text.
• Every location in _master.json points to a document that exists in the registry and a page that exists in that document.

5. Adding documents later

Inventory only the new files. Assign them the next free IDs. Add them to the registry. Index them. Rebuild _master.json from all per-document files. Nothing already indexed needs to be touched.

6. Using the index

To answer a question: open _master.json, find the relevant topics, collect the locations, resolve each document ID to a path through _meta.json, open those page images, and answer from what they say. Cite the file name and page number. Never answer from the index alone.