Chunk size: recall falls, faithfulness rises
Grow a chunk from 128 tokens to 2,000 and two things happen at once. Search gets worse, because the vector now represents an average of many topics. The answer gets better, because the model finally sees whole paragraphs. Here is why both are true, and why the fix is not to pick a number in the middle.
Chunk size sets the granularity of the vector. Small chunks give sharp vectors and high recall but fragmented context. Large chunks give rich context and faithful answers but diluted vectors. One size cannot win both, so decouple the text you index from the text you retrieve.
Contents
01Why chunk size matters at all
A dense embedding model takes a block of text and compresses it into one fixed-length vector, typically 768 or 1,536 floating-point numbers. The vector is the same size whether the block is one sentence or three pages. That single fact is the root of the whole trade-off.
- Small chunks, around 128 tokens. The block is about one thing: a rule, a figure, a definition. The vector is sharp and uncluttered, and cosine similarity lines it up cleanly with a specific user question.
- Large chunks, around 2,000 tokens. The block weaves together several topics, transitions and asides. The vector has to represent all of them, so it ends up as an average. Each distinct signal is weakened. This is vector dilution.
So the chunk size you choose is not a tokenisation detail. It directly sets how fine-grained your search can be, and how much setting the model gets to read.
02What happens from 128 to 2,000 tokens
Two retrieval-augmented generation metrics move in opposite directions as chunks grow.
| Metric | At 128 tokens | → | At 2,000 tokens |
|---|---|---|---|
| Recall (does the right passage come back?) | High | declines | Lower |
| Faithfulness (is the answer grounded in the source?) | Low | rises | High |
Recall is a property of the search step and is hurt by big chunks. Faithfulness is a property of the generation step and is helped by them. The two sections below say why.
03Recall declines: vector dilution
Take a 2,000-token chunk from a customer policy document. Somewhere inside it is the sentence "Refund processing takes 3 to 5 business days." That fact is about one percent of the chunk. The other ninety-nine percent covers cancellation terms, shipping rules and account setup.
The embedding of that chunk represents its average topic, which is something like policy management in general. A user asking about the refund timeline produces a query vector that is sharp and specific. It is now being compared against a blurred vector, and a different chunk that happens to mention refunds in passing may score higher. The right passage drops out of the top results.
There is a second effect. Search works best when the variance inside a chunk is low. A 128-token chunk has almost no off-topic text, so the noise ratio is minimal and fine-grained queries hit reliably. A 2,000-token chunk carries high internal variance, and the search has no way to see past it.
The more topics a chunk contains, the less its vector says about any one of them. Recall for specific questions falls as chunks grow.
04Faithfulness rises: no more splintered context
Now look at the same documents from the model's side. A 128-token chunk is often cut mid-thought. It can begin halfway through a sentence, or open with a pronoun whose referent was in the previous chunk: "It increased by 10%", with no statement of what increased or compared with when.
When the model is handed a set of fragments like that, it has to guess at the connections between them. The guessing comes from parametric memory, not from the source, and that is exactly what a faithfulness metric penalises. The answer reads fluently and is partly made up.
A 2,000-token chunk gives the model complete paragraphs, the section header, the surrounding constraints and the local definitions. It has the full picture, so it can synthesise an answer that stays strictly inside the source text. Faithfulness rises, up to the point where the chunk is so large that the relevant sentence is buried in material the model has to wade through, and where a handful of chunks fills the context window with padding.
A model hallucinates to fill gaps. Bigger chunks leave fewer gaps. Faithfulness rises with chunk size, until the window is full of fluff.
05Side by side
| Dimension | 128-token chunks | 2,000-token chunks |
|---|---|---|
| Search precision and recall | Very high. Pinpoints specific facts, numbers and definitions. | Lower. Dilution hides fine detail inside broad themes. |
| Answer faithfulness | Lower. Fragments with missing background trigger hallucination. | High. Keeps sentence logic, pronoun referents and conditional rules. |
| Context window cost | Efficient. Only targeted snippets reach the prompt. | Expensive. A few chunks fill the window, much of it irrelevant. |
| Typical fit | FAQ lookup, precise entity search, code snippet matching. | Multi-step reasoning, thematic summaries, narrative analysis. |
06The way out is not a compromise size
Because neither extreme is good on its own, the tempting move is to tune the dial to the middle. Many teams land somewhere around 300 to 500 tokens, and that is a reasonable place to balance the two failures. But it is a balance of failures. Recall is a bit worse than it could be and faithfulness is a bit worse than it could be, and no amount of tuning changes that, because one string is still doing two jobs.
Production systems escape the dial by decoupling the indexed text from the retrieved text. The vector is made from one string and the model reads another.
- Small-to-big, or parent-document retrieval. Index 128-token child chunks so that search has sharp vectors and high recall. When a child matches, fetch its 1,000 to 2,000-token parent block and give that to the model, so that generation has the whole picture and high faithfulness.
- Contextual chunks. Keep chunks in the 300 to 500 range, but before embedding each one, prepend a short generated summary of where it sits in the document. The vector now knows what "it" refers to, which recovers the recall that mid-sized chunks lose, without shrinking what the model reads.
Both patterns let you move recall and faithfulness independently, which is the whole point. The previous note walks through each of them, plus a third based on summary and question indexes, in more detail.
07Takeaways
- A vector is a fixed-size average of its input. The more topics go in, the less it says about any one of them.
- Recall falls as chunks grow. Specific questions stop matching diluted vectors.
- Faithfulness rises as chunks grow. Whole paragraphs leave the model fewer gaps to fill from memory.
- A middle size balances two failures. It does not remove either.
- Decouple the two texts. Index small for recall, retrieve big for faithfulness, and tune each on its own.