The chunk you embed is not the chunk you retrieve
Search wants small, focused text. The language model wants rich, surrounding text. If one chunk has to do both jobs, it does both badly. The fix is to decouple the text you index from the text you hand to the model.
Embed a small, focused or contextualised version of the text to produce the vector record. When it matches, retrieve the full surrounding chunk for the model. The vector and the context are two different strings that share an id.
Contents
01The conflict inside every RAG chunk
A retrieval-augmented generation system does two very different things with the same piece of text. First it turns the text into a vector so a query can find it. Then it pastes the text into a prompt so a language model can answer from it. Those two uses want opposite things.
- What search wants. Small, concise snippets about one thing. A vector made from a focused snippet sits tightly in one region of embedding space, so a specific question lands on it precisely.
- What generation wants. The whole paragraph, the parent section, the table header, the definitions two paragraphs up. A model answers accurately when it can see the surrounding explanation, and hallucinates when it is handed a sentence torn out of its setting.
Most pipelines pick one chunk size and use that chunk for both. That is the mistake this note is about. For the metric-by-metric picture of what happens as chunks grow, see the companion note on chunk size.
02Why one chunk size is always wrong
If the indexed text and the retrieved text are the same string, chunk size is a single dial that has to serve two masters.
| Chunk size | Search | Generation |
|---|---|---|
| Small, about 100 tokens | Precise. Vectors are sharp and specific queries hit. | Poor. The model gets chopped-up sentences with no setting. |
| Large, about 1,000 tokens | Diluted. One vector averages many topics and misses fine-grained queries. | Rich. The model sees whole explanations. |
Tuning the dial finds the least bad compromise, not a good setting. Every chunking benchmark that reports "512 tokens worked best" is reporting the point where the two failures balance.
An embedding is a lossy summary of its input. The more you put in, the more the vector becomes an average, and an average of three topics is close to none of them. A language model has the opposite property: the more relevant context it sees, the better it does, up to its window.
03Decoupling: two strings, one id
The way out is to stop pretending there is one chunk. Each record in the index has two texts:
- The indexed text, which is what gets embedded. Small, focused, possibly rewritten.
- The retrieved text, which is what the model reads. Large, raw, surrounding.
They are linked by an id. Search runs over the indexed texts. When a vector matches, the system follows the id to a document store and returns the retrieved text instead. The vector database never has to hold the big text at all.
There are three common ways to produce the indexed text. They differ in what the small string is: a piece of the big one, the big one with a preface, or a rewrite of it.
04Pattern 1: small-to-big, or parent-document retrieval
Split each document twice. Cut it into large parent sections of around 1,000 tokens, then cut each parent into small child chunks of around 128 tokens. Embed only the children. Each child carries its parent's id in its metadata.
- Search step. The query matches a small child vector. Because the child is about one thing, the match is precise, down to the line.
- Retrieval step. The system reads the child's parent id, fetches the whole parent section from the document store, and hands the parent to the model.
This is the simplest of the three because no model is involved at index time. It is pure bookkeeping. If several children of the same parent match, deduplicate on the parent id so the model does not see the same section three times.
Precision comes from the small vector pinpointing the exact line. Faithfulness comes from the model reading the entire surrounding explanation. Neither has to give anything up.
05Pattern 2: contextualised embeddings
Small chunks lose meaning when isolated. A chunk that says "It grew by 15% in Q3" is nearly useless on its own. The vector has no idea what "it" is, which company, or which year's third quarter. A query asking about that company's quarterly revenue growth will not land on it.
Anthropic's contextual retrieval pattern fixes this at index time. Before embedding each chunk, pass the chunk and the whole document to a model and ask it to write a short preface, roughly 50 to 100 tokens, that says what the chunk is about within the broader document. The chunk above might become:
This chunk is from ACME Corp's 2023 annual filing, in the section on North American revenue. The figure refers to revenue growth over the previous quarter. It grew by 15% in Q3.
- Search step. Embed the generated context plus the raw chunk. The vector now carries the company name, the document title and the core theme, so queries phrased in those terms find it.
- Retrieval step. Return either the raw chunk or the contextualised version to the model. Both work. The preface is usually harmless to include.
The cost is one model call per chunk at index time. With prompt caching the document is sent once and each chunk's call is cheap, which is what makes this affordable over a large corpus. The same preface also helps a keyword index, since the company name is now literally in the text.
06Pattern 3: summary and question indexes
For dense technical material, long reports or legal text, embedding the raw prose produces noisy vectors. The language is formal and repetitive, and the thing a user asks about is often buried in a subordinate clause.
Here the indexed text is not a piece of the document at all. Run a light model over each section and have it write a two-sentence summary, or a list of questions the section answers. Embed only that.
- Search step. The query is matched against the summary's vector, or the generated questions' vectors. A question-to-question match is much tighter than a question-to-prose match, because both sides are written in the same register.
- Retrieval step. When a summary matches, fetch the full underlying section it was written from and give that to the model. The summary itself is never what the model answers from.
A close cousin is HyDE, hypothetical document embeddings, which does the rewrite on the query side instead: the model writes a hypothetical answer to the user's question and that is embedded and searched. Both are the same idea, moving the two sides of the match into the same register before comparing them.
When the indexed text is generated, the prompt that generates it decides what the index can find. Ask for summaries and you find topics. Ask for the questions a reader would bring and you find intents. Changing that prompt changes recall without touching the embedding model.
07Which one to use
| Pattern | Indexed text | Index-time cost | Best when |
|---|---|---|---|
| Parent-document | A small piece of the section | None | Documents are well structured and sections stand alone. |
| Contextualised | Preface plus the raw chunk | One short model call per chunk | Chunks are full of pronouns, abbreviations and references to things defined elsewhere. |
| Summary or questions | A rewrite of the section | One model call per section | The prose is dense or in a different register, or language, from the questions. |
They stack. A common production shape is parent-document retrieval with contextualised children, so the vector is sharp and self-describing and the model still reads the full section. Start with parent-document because it is free, then add a generated layer when you can measure a recall gap that it would close.
08Where this shows up in the Purana RAG
The clearest case of pattern 3 on this site is the Purana question-answering system. The corpus is Sanskrit and the users ask in Telugu. Embedding the Sanskrit verses and searching them with Telugu questions did not work, for the same reason a dense technical section does not match a casual question: the two sides are in different registers, and here also in different languages.
So the indexed text and the retrieved text were split. For each chapter, a model wrote a Telugu card, a summary plus the questions a reader would bring to that chapter, and only the card was embedded. When a card matches, the retrieval step fetches the original Sanskrit verses of that chapter and sends those to the answering model, which reads Sanskrit well. The card summary rides along for orientation, but the verses are what the model answers from and cites.
Two things from that project are worth carrying back to the general pattern. Indexing at chapter level, the granularity of questions, made the index fifty times smaller than verse level and matched how people ask. And rewriting the card prompt to require three life-situation questions per card doubled hit@1 on moral questions, with no change to the embedding model. The prompt that writes the indexed text was the most effective retrieval parameter in the system.
09Takeaways
- One chunk cannot serve both search and generation. Small is right for the vector, large is right for the model, and any single size is a compromise.
- Store two strings per record, linked by an id. Embed the small one. Return the large one.
- Start with parent-document retrieval. It costs nothing at index time and fixes most of the problem.
- Add generated context when chunks cannot stand alone. A short preface makes a vector self-describing and helps keyword search too.
- Index summaries or questions when the prose is in the wrong register. Match question to question, then read the source.
- Treat the generating prompt as a retrieval parameter. It decides what the index can find.