Which RAG component broke? Why an end-to-end score cannot tell you
An answer-correctness score of 70 percent tells you that three answers in ten are wrong. It does not tell you whether the retriever never found the passage, the reranker buried it, the assembler truncated it, or the model had it in front of it and ignored it. Each of those has a different fix, and the end-to-end number is identical for all of them. This note names the stages that can fail, gives each its own measurement, and shows how to attribute every wrong answer to the stage that caused it.
A RAG answer passes through query understanding, retrieval, reranking, context assembly and generation, and any of the five can lose it. The end-to-end score is roughly the product of their individual success rates, so it cannot be inverted to find the culprit. Measure each stage against the one before it: did the gold passage enter the candidate set, survive reranking, survive assembly, and get used by the model. Two oracle experiments separate the halves: give the model the gold context and score generation alone; give a judge the retrieved context and the gold answer and score retrieval sufficiency alone. Then run every failing case through a short decision tree that stops at the first stage where the gold was lost.
Contents
01Five stages, five ways to lose the answer
| Stage | What it does | How it fails | What you see |
|---|---|---|---|
| Query understanding | Rewrites the question, resolves references to earlier turns, extracts filters such as dates or product names, decides whether to retrieve at all | Rewrites away the key term; extracts a wrong filter that excludes everything relevant; misses that "it" refers to last turn's product | Retrieval looks healthy on the rewritten query and still returns the wrong passages |
| Retrieval | Finds candidate chunks by vector, lexical or hybrid search under the filters | The gold chunk is not in the top-k: embedding mismatch, a chunk that is too long or too short, a filter applied too early, the document never indexed | Gold absent from candidates |
| Reranking | Reorders candidates with a stronger model and cuts to the final few | Gold was in the candidates and is pushed below the cut; the reranker prefers a near-duplicate or a passage that merely mentions the entity | Gold present before reranking, absent after |
| Context assembly | Deduplicates, expands children to parents, orders, fits to a token budget | Truncation drops the gold; parent expansion dilutes it; two near-identical passages fill the budget; the order buries it in the middle | Gold present after reranking, absent or mangled in the final prompt |
| Generation | Answers from the assembled context | Ignores the context and answers from memory; answers from the wrong passage; is faithful but incomplete; right content in the wrong format | Gold present in the prompt, answer still wrong |
Each row's "what you see" is defined in terms of the previous row. That is the whole trick: a stage is judged on what it did with what it was given, not on the final answer.
02Why the end-to-end score cannot be inverted
Suppose each stage passes the gold through 90 percent of the time. End to end you get about 0.95, roughly 59 percent. Now suppose retrieval drops to 70 percent and everything else is perfect: the end-to-end score is 70 percent. Or generation drops to 70 percent and the rest is perfect: also 70 percent. One number, two completely different problems, two completely different fixes. The first wants a better embedding model or chunking; the second wants a better prompt or model.
Three further reasons the aggregate hides the cause.
- Stages mask each other. A strong reranker can rescue a weak retriever, so the retriever's problems do not show in the final score until the day the reranker's cut is tightened. A generous model can answer correctly from memory when retrieval fails, so retrieval looks fine until a question arrives that memory cannot answer.
- Failures compound in one direction. A loss at any stage is final; nothing downstream can recover a passage that was never retrieved. So an early-stage failure rate puts a ceiling on every later stage's apparent performance, and improving generation cannot move the score past that ceiling.
- The score is noisy. An end-to-end judge on a few hundred questions has a confidence interval of several points. A change that fixes one stage and slightly hurts another can look like no change at all.
03What you need labelled
Per-stage measurement needs more than question-and-answer pairs. For each evaluation question you want:
- The gold answer, as before.
- The gold evidence: which document, and which span within it, contains the answer. Label it on something stable, the document id and a character range or a section id, not a chunk id, since chunk ids change whenever chunking changes. The semantic chunking note covers why.
- The expected filters, where the system extracts any: the date range, the product, the tenant.
- For multi-turn questions, the resolved standalone form, so query rewriting can be scored.
Gold evidence is the expensive label and the one that unlocks everything else. A practical way to get it is to generate questions from passages, so the passage is known by construction, and then have people check that the question is natural and the passage really answers it.
04A metric per stage
| Stage | Metric | Computed how |
|---|---|---|
| Query understanding | Rewrite fidelity; filter accuracy | Judge compares the rewrite to the resolved standalone question; filters compared field by field to expected |
| Retrieval | Recall@k on gold evidence; MRR | A candidate counts as a hit if its span overlaps the gold span; k is the number passed to the reranker |
| Reranking | Gold-in-top-n after reranking, conditioned on gold-in-candidates; nDCG@n before versus after | Only cases where retrieval succeeded count; n is the number passed to assembly |
| Context assembly | Gold-in-prompt, conditioned on gold-in-top-n; context precision | Does the final prompt text contain the gold span; what fraction of prompt tokens come from relevant passages |
| Generation | Faithfulness; answer correctness, conditioned on gold-in-prompt | Judge checks every claim in the answer is supported by the prompt; correctness against gold, only on cases where the prompt contained the evidence |
The conditioning is what makes these stage metrics rather than restatements of the end-to-end score. Reranking is only scored on cases the retriever got right; generation only on cases where the evidence was in the prompt. Each stage's number then answers one question: given a fair chance, how often did this stage do its job.
05Two oracle experiments
Conditioning on logs gives you attribution for the system as it is. Two controlled experiments give you the ceiling of each half, which tells you where effort will pay.
Gold-context oracle: how good is generation on its own?
Bypass retrieval entirely. For every question, put the gold evidence passage in the prompt and generate. The resulting correctness is the best your generation stage can do with perfect retrieval. If it is 95 percent and the full system scores 70, retrieval is where the thirty points are. If it is 75 percent, generation is losing most of them and a better retriever will barely move the final score.
Retrieved-context sufficiency: how good is retrieval on its own?
Take the context the real pipeline assembled, and instead of generating, ask a judge: does this context contain enough to produce the gold answer? This scores retrieval, reranking and assembly together, independent of the model's ability to use what it was given. A sufficiency rate well above the final correctness means the model is being handed answers it fails to use.
full system correctness 70% gold-context oracle 95% generation is capable when given the evidence retrieved-context sufficiency 72% the evidence is reaching the prompt 72% of the time → almost all loss is upstream of generation full system correctness 70% gold-context oracle 74% generation fails even with the evidence retrieved-context sufficiency 93% → the loss is in generation; fix prompt or model first
06Attributing each failure
With gold evidence labelled and every stage's intermediate output logged, each wrong answer can be walked down a short tree. Stop at the first stage where the gold was lost.
answer wrong │ ├─ was the rewritten query or extracted filter wrong? → query understanding │ ├─ was the gold span in the retrieval candidates? no → retrieval │ ├─ was it in the top-n after reranking? no → reranking │ ├─ was it in the final prompt, intact? no → context assembly │ └─ it was in the prompt → generation ├─ answer unsupported by the prompt → ignored context, answered from memory ├─ answer supported by a wrong passage → used the wrong evidence; often an assembly ordering issue └─ answer supported and incomplete → partial extraction
Run every failing case through this and you get a histogram of failures by stage. That histogram, not the end-to-end score, is what tells you what to work on. It also tells you what not to work on: if four percent of failures are reranking, a better reranker is not this quarter's project.
07Reading the result
Some patterns recur often enough to name.
- Retrieval dominates on specific, rare questions and generation dominates on broad ones. Split the histogram by question type before deciding.
- Assembly failures cluster at the token budget. If gold-in-prompt is well below gold-in-top-n, the budget or the parent expansion is the cause, and both are cheap to change.
- Query understanding failures are invisible in single-turn evaluation sets. They appear in multi-turn sets and in sets with filters. If your set has neither, the stage is unmeasured, not healthy.
- Generation failures split into two kinds that need different fixes. Ignoring the context is a prompting and model problem; using the wrong passage is usually an assembly problem wearing a generation costume.
08Takeaways
- Five stages, each judged on what it did with what it received, not on the final answer.
- The end-to-end score is a product and cannot be inverted; the same number has different causes with different fixes.
- Label gold evidence, on document and span, not chunk id. It is the label that unlocks per-stage measurement.
- Condition each stage's metric on the previous stage's success. Reranking is scored only where retrieval found the gold.
- Run the two oracles. Gold-context correctness bounds generation; retrieved-context sufficiency bounds the retrieval half.
- Walk every failure down the tree and work on the stage with the tallest bar.