How to chunk wiki pages
A wiki page arrives with its structure already written down: a heading tree, an infobox, tables and lists. Fixed-size chunking throws all of that away and then pays to reconstruct it. Split on the headings instead, tell every chunk where it came from, and retrieve the section when a paragraph matches.
Structural, heading-aware chunking plus contextual enrichment. One section or subsection per chunk. Prepend the page title and heading path before embedding. Flatten infoboxes and tables to text. Fall back to paragraph splitting only when a section runs past about 500 tokens. Index paragraphs, retrieve sections.
Contents
01Why wiki pages are a special case
Most chunking advice assumes the input is undifferentiated prose and the splitter has to guess where the ideas begin and end. A wiki page is the opposite. It was written by people who were forced to declare its structure: nested headings in MediaWiki's == syntax or Markdown's #, a table of contents generated from those headings, an infobox summarising the key facts, and tables and lists for anything tabular.
Fixed-size or naive recursive splitting ignores all of that. It cuts a table mid-row, separates a subsection from its heading, and produces chunks like "He was born in 1942" with no way of knowing who "he" is, because the page title was two thousand tokens ago. The recall versus faithfulness tension is at its worst here, and it is entirely self-inflicted.
02Split on the heading hierarchy
The core move is to split along heading boundaries rather than character counts. Parse the page into a tree and make each leaf section a chunk.
Page: Apollo 11
├── # Overview
├── ## Mission summary
└── ## Lunar module operations
├── ### Descent phase → chunk 1
└── ### Surface activities → chunk 2
This works for wikis for two reasons.
- Topic boundaries are preserved. A section under a heading is usually about one idea or one event. The author has already done the segmentation that a splitter would otherwise be guessing at, so each vector is about one thing, which is what search wants.
- Tables and lists stay whole. An infobox, a specification table or a bulleted list lives inside one section, so it is never cut mid-row. A half table is worse than no table, because the model reads column headers that no longer line up with the values.
A side benefit: with structural boundaries there is no reason to add chunk overlap. Overlap exists to repair cuts that land mid-thought, and headings do not cut mid-thought.
03Fallback: long sections
Wiki sections vary wildly in length, from a single sentence to three thousand words. A History section on a well-tended page can be longer than some entire articles. Pure structural splitting would make that one chunk, and its vector would be the diluted average of fifty years of events.
So structural splitting needs a fallback. When a section exceeds a budget, around 500 tokens is a sensible default, split inside it recursively. Split on double line breaks first so paragraphs stay intact, then on single line breaks, then on sentences, and only as a last resort on a token count. Every piece produced this way keeps the same heading path as the section it came from, which is the next point.
04Heading path injection
An isolated wiki chunk has lost its scope. A section under Engine specs on the Falcon 9 page does not contain the words "Falcon 9" anywhere. A query about Falcon 9's engines will match pages that happen to say Falcon 9 and engines in the same paragraph, and miss the section that actually answers it.
The fix is cheap: prepend the page title and the full heading path to the top of every chunk before embedding it.
[DOCUMENT]: SpaceX Falcon 9 [PATH]: Falcon 9 > First stage > Engine specs [TEXT]: Uses 9 Merlin 1D engines arranged in an Octaweb pattern generating 1.71 million pounds of thrust at sea level...
The vector now carries the entity name and the section's place in the page, so the match is on Falcon 9 engines rather than on some engines. The same prefix resolves the "he was born in 1942" problem, because the document line names the person. And it helps a keyword index for free, since the page title is now literally in the text.
Anthropic's contextual retrieval pattern asks a model to write a short preface for every chunk. On a wiki page the preface is already written: it is the title and the heading path. Use the structure first and reach for a generated preface only for sections where the path alone is not enough.
05Infoboxes and tables
Wiki pages put their most-asked-about facts in the infobox: launch date, crew, manufacturer, population, dates of birth and death. In raw wikitext or HTML that is a tangle of template syntax and table markup, and an embedding of it is mostly noise.
Convert tables and infoboxes to a clean textual form before chunking. Two forms work well.
- Key-value lines for infoboxes:
Launch date: July 16, 1969, one per line. Each line is a self-contained fact and the vector captures the relationship between the key and the value. - Markdown tables for real tables, with the header row kept. If a table is too long for one chunk, repeat the header row at the top of every piece so the columns stay labelled.
Give the infobox its own chunk with the heading path Page > Infobox. It is the single most valuable chunk on the page for factual lookups.
06Index paragraphs, retrieve sections
Everything above produces good chunks. The last step is to stop using the same chunk for search and for the model, following the decoupling pattern.
| Layer | Unit | Size | Role |
|---|---|---|---|
| Child | Paragraph or subsection, with the heading path injected | 150 to 300 tokens | Vector search. Sharp, specific, highly searchable. |
| Parent | The whole ## section | Whatever the section is | Model context. When a child matches, hand the model its entire parent section. |
Each child carries its parent's section id. On a hit, look up the parent and send that, and deduplicate so that three matching paragraphs from one section produce one copy of the section in the prompt. The model gets the full background of the section, including the paragraphs around the one that matched, and the search still ran on small, precise vectors.
07Pipeline checklist
- Parser. Use a syntax-aware parser that gives you the heading tree, not a regex over the text. WikiExtractor or mwparserfromhell for wikitext, BeautifulSoup for rendered HTML, a Markdown parser for Markdown wikis.
- Chunking unit. One section or subsection per chunk.
- Fallback. Recursive paragraph-first splitting when a section exceeds about 500 tokens.
- Enrichment. Page title plus heading path prepended to every chunk before embedding.
- Tables. Infoboxes to key-value lines, tables to Markdown with headers kept.
- Metadata. Attach the wiki title, heading path, URL and section id to every vector record, so the answer can cite the exact section and the retrieval step can fetch the parent.
- Retrieval. Children in the index, parents in the document store, deduplicate on section id.
08The same shape in the Purana corpus
The Purana texts used in the Purana question-answering system are not wiki pages, but they have the same property: the structure is already written down. Each Purana is divided into chapters with verse numbers, and the GRETIL e-texts carry those markers, five different reference conventions across seventeen files. The parsing work was larger than the generation work, and it was worth it, because the chapter turned out to be the right retrieval unit. Indexing at the structural boundary rather than at a token count is what made the chapter cards possible, and the card's title and Purana name play exactly the role the heading path plays here.
09Takeaways
- Use the structure the author gave you. Headings are a segmentation someone already did by hand.
- Every chunk should know its page and its path. Prepend the title and heading path before embedding.
- Flatten the infobox. Key-value lines turn the most-asked facts into the most-findable chunk.
- Fall back, do not default, to size-based splitting. Only inside sections that are too long, paragraph-first.
- Search the paragraph, read the section. Decoupled parent-child retrieval closes the loop.