Semantic chunking made my retrieval evals worse

01What actually changed

"Semantic chunking" usually means this: embed each sentence, measure the distance between consecutive sentence embeddings, and cut wherever the distance spikes above some percentile. The promise is that cuts land at topic shifts rather than at arbitrary token counts.

The promise is real, but the switch is not one change. Compared with fixed 512-token chunks and a 64-token stride, it changes:

• Length. From every chunk at 512 tokens to a spread from a single sentence to a page.
• Overlap. From 64 shared tokens at every boundary to none.
• Boundary placement. From positions your evaluation set may have been built on to positions it has never seen.
• A new tuning parameter. The distance percentile, which has no right value across different kinds of text.

Each of those can lower a retrieval metric by itself. The regression you see is their sum, and the job is to find which terms are large.

02Cause 1: variable length and embedding drift

Fixed chunking guarantees every vector represents the same amount of text. Semantic chunking produces chunks of 50 tokens and chunks of 1,200 tokens in the same index, and most embedding models do not behave consistently across that range.

Very short chunks lack semantic density: a single sentence's vector is sharp but carries little, and often sits close to many unrelated single sentences. Very long chunks suffer the dilution described in the chunk size note: the vector averages several topics and a specific query no longer lands on it. Both tails score worse than the 512-token middle you came from, and both tails are now populated.

There is a quieter effect too. Similarity scores from chunks of different lengths are not on quite the same scale, so a ranking that mixes a 60-token chunk and a 900-token chunk is comparing numbers that were not meant to be compared. Rerankers partly absorb this; plain cosine ranking does not.

03Cause 2: the overlap went away

A fixed splitter almost always carries an overlap, and that overlap was quietly doing a job: any fact or definition that straddled a boundary appeared whole in at least one chunk. A semantic splitter cuts cleanly at the distance spike and shares nothing across it.

Cutting at a topic shift should mean fewer straddling facts, and often it does. But distance spikes are noisy. A splitter that reacts to a change in sentence style, a list item, or a quotation will cut in the middle of an explanation, and now neither side contains enough to match the query. The fixed chunker had the same bad cuts; the overlap hid them.

04Cause 3: tiny chunks starve the top-k budget

This one is often the whole story, and it is an artefact of how the metric is defined rather than of search quality.

If the semantic chunker produces many tiny chunks, retrieving five of them returns a fraction of the text that five fixed chunks returned. Any metric that asks "is the answer in the retrieved context" is now being asked with a sixth of the context. The downstream model is starved, and so is the evaluator. The ranking may be just as good or better; the budget it is given is smaller.

05Cause 4: one global distance threshold

The splitter cuts when the distance between consecutive sentences exceeds, say, the 95th percentile. That percentile is computed over whatever text the splitter sees, and one value cannot be right for a corpus that mixes code, dense legal clauses, tables and narrative prose.

In dense technical text every sentence introduces something new, so distances are uniformly high and the splitter over-segments into fragments. In smooth narrative prose distances are uniformly low, so the splitter under-segments into page-long chunks. The same threshold produces both tails of the length histogram, which is why cause 1 and cause 4 usually appear together.

06Cause 5: the ground truth was labelled on the old boundaries

This is the cause to rule out first, because if it applies, the metric is not measuring what you think.

Evaluation sets are built from the chunks that existed when they were built. Synthetic question generation ran over 512-token chunks and recorded which chunk the answer came from. Human annotators were shown 512-token passages and asked to write questions those passages answer. Either way, the ground truth carries the old boundaries inside it, in two forms.

• The relevance label is a chunk id. If the metric counts a hit when the retrieved chunk id equals the labelled chunk id, then re-chunking invalidates every label, and the score is noise. Relevance has to be defined on something stable: the document, the section, or the answer span's character offsets.
• The questions need the old context. A question written from a 512-token passage may rely on two or three sentences that were together in that passage and are now in different semantic chunks. The new chunker is being graded against a test that was shaped by the old one.

07Isolating the cause

eval dropped after switching to semantic chunks
 │
 ├─ 0. is relevance labelled by chunk id?            → relabel on spans before anything else
 │
 ├─ 1. plot the chunk length histogram               → tails below 100 or above 800 tokens?
 │
 ├─ 2. re-score on a fixed token budget, not top-k   → does the gap close?
 │
 ├─ 3. inspect queries that flipped pass → fail      → is the answer split across a boundary?
 │
 └─ 4. add min/max bounds, then overlap, one at a time → which one recovers the score?

Step 0: check what the label is

Open the evaluation set and look at the relevance field. If it is a chunk id from the old index, stop and relabel. Map each labelled chunk to its source document and the character span of the answer, and define a hit as "a retrieved chunk overlaps the labelled span" or, more loosely, "a retrieved chunk is from the labelled document and section". Only then are the two chunkers being scored on the same question.

Step 1: the length histogram

Count tokens per chunk for the semantic index and plot the distribution. Look for the two tails: a spike of chunks under 100 tokens, and a tail above 800. A heavy low tail points to causes 3 and 4. A heavy high tail points to causes 1 and 4. A histogram that looks like a reasonable hump around a few hundred tokens largely rules out length as the problem and sends you to step 3.

Step 2: score on a token budget

Instead of retrieving the top 5 chunks for both chunkers, retrieve the top chunks until a fixed budget is filled, say 2,000 tokens, and score both on that. This removes the budget asymmetry of cause 3.

• If the semantic chunker's score recovers to or above the fixed chunker's, the regression was context starvation from tiny chunks. The ranking was fine; the top-k metric was unfair to it.
• If the gap stays, the ranking itself is worse, and the cause is among 1, 2 and 4.

Step 3: inspect the flipped queries

Filter the evaluation run to the queries that passed under fixed chunking and failed under semantic chunking. There are usually fewer than you expect, and they are concentrated. For each, take the ground-truth answer span and find where it sits in the semantically chunked document.

• If the span straddles the boundary between chunk N and N+1, this is cause 2, aggressive splitting with no overlap.
• If the span sits whole inside one chunk that is very long, this is cause 1, dilution.
• If the span sits whole inside one chunk that is very short and that chunk ranked low, this is cause 1 on the other tail, a sentence with too little in it to match.
• If the span is intact and well-sized and the chunk still ranked below unrelated ones, look at the query: it may depend on neighbouring sentences that the old passage included, which is cause 5.

Step 4: controlled experiments

Change one thing at a time and re-score after each.

1. Add a minimum chunk size of about 200 tokens, merging adjacent small chunks until they reach it. If the score recovers, it was tiny chunks.
2. Add a maximum of about 600 tokens, splitting long semantic chunks at paragraph breaks. If the score recovers, it was dilution.
3. Add a 10 to 15 percent overlap at the semantic boundaries. If the score recovers now, after bounds did not fix it, the splitter was cutting through explanations.

The order matters. Bounds first, because they are cheaper to keep than overlap, and because overlap applied to tiny chunks only duplicates tiny chunks.

08Fixes, by cause

One honest possibility remains after all of this: semantic chunking is simply not better for your corpus. On text that already has structure, headings do the job with none of the variance, and on uniform prose a fixed size with a small overlap is a strong baseline that is hard to beat. The chunker is a hypothesis, and the evaluation set, once it is fair, is allowed to reject it.

09Takeaways

1. The switch changes length, overlap and boundaries at once. Separate them before you tune anything.
2. Check the label first. Relevance defined by chunk id is invalidated by any re-chunking.
3. Score on a token budget as well as top-k. Tiny chunks lose on top-k even when the ranking is better.
4. Read the flipped queries. Where the answer span sits in the new chunks names the cause.
5. Bounds before overlap, parent-child before either. Min and max chunk sizes fix most of it; parent-child fixes boundary splits without duplicating text.
6. One global percentile cannot serve mixed text. Use structure where there is structure.
7. Be willing to keep the fixed chunker. It is a strong baseline, and a fair evaluation may say so.