Design a retrieval-augmented generation pipeline

01The design, in the six-step framework

1. Requirements

Functional.

• Answer natural-language questions over a private, changing corpus (internal documents, product manuals, a support knowledge base) using a language model, with every answer grounded in retrieved passages and citing them.
• Documents are added, updated and deleted continuously; an update is reflected in answers within an hour; a deletion is never cited again after it is applied.
• Respect document permissions: a user only gets answers grounded in documents they may read.
• Out of scope: training the language model, the chat UI, the corpus's authoring tools.

Non-functional.

• Corpus: 50 million documents, 500 million chunks; 200 questions per second at peak.
• Latency: first token within 1.5 s at p95, with retrieval under 300 ms of that.
• Quality: measured by an evaluation set; answer must not assert anything the retrieved passages do not support, and must say "not found" when they do not.
• Cost: retrieval cheap enough that most of the per-question cost is generation.

2. Core entities

• Document: doc_id, source, version, permissions (ACL), updated_at, content hash.
• Chunk: chunk_id, doc_id, version, position, text, headings path, token count, embedding, keyword terms.
• Index entry: chunk_id → vector (ANN index) and chunk_id → terms (inverted index), both tagged with doc version and ACL.
• Query: query_id, user, question, conversation context, retrieved chunk ids with scores, prompt, answer, citations, latency.
• Evaluation case: question, expected grounding, graded answer; used to score every pipeline change.

3. API

• POST /ask {question, conversation_id, user} → streamed answer with citations [{chunk_id, doc_id, quote span}] and a grounding flag; not_found when retrieval returns nothing relevant.
• POST /documents {doc_id, content, acl, version} and DELETE /documents/{doc_id} → 202; the ingest path, idempotent by (doc_id, version).
• GET /documents/{id}/status → indexed version, chunk count.
• Internal: POST /retrieve {query embedding, terms, acl filter, k} → candidates; POST /rerank {question, candidates} → scores.

4. High-level design

Ingest and query:

1. A document change lands on the queue; a worker fetches the content, splits it into chunks along headings and paragraphs (about 300 tokens with overlap), attaches the heading path and metadata, embeds each chunk, and writes the vector, the keyword terms and the text, all tagged with the document version and ACL.
2. Old chunks of that document version are marked deleted in the same commit, so an updated document never has two versions retrievable at once.
3. A question is embedded and sent to both indexes in parallel with an ACL filter (user's groups); each returns 50 candidates; the lists are fused (reciprocal rank fusion).
4. A cross-encoder reranks the fused 100 against the question and the top 8 to 10 are selected, deduplicated by document, and their text fetched.
5. The orchestrator builds the prompt (instructions, passages with ids, the question, conversation context), streams the model's answer, and validates that every citation id exists in the passage set before returning it.

5. Deep dives

Why hybrid retrieval and a reranker, rather than one vector search?

Embedding search finds passages that mean the same thing as the question but misses exact tokens (a product code, an error string, a person's name) that a keyword index finds trivially; the reverse is also true. Running both and fusing the rankings costs one extra 30 ms call and raises recall measurably on any realistic evaluation set. The first-stage retrievers are optimised for speed over 500 million items and are only approximately right; a cross-encoder that reads question and passage together is far more accurate but costs 1 ms per pair, so it is applied to 100 candidates, not 500 million. This is the same funnel shape as recommendation ranking, and the numbers in the budget (300 ms retrieval) follow directly from it.

How does chunking decide quality, and how are updates and deletes made safe?

Chunks too small lose the context that makes a passage answer a question; too large and the embedding averages over several topics and the prompt fills with noise. Splitting on document structure (headings, paragraphs) at about 300 tokens with a one-sentence overlap, and prefixing each chunk with its heading path, is the workable default; the evaluation set decides any change. Every chunk carries (doc_id, version), and ingest is idempotent on that pair: re-processing a version produces the same chunk ids and overwrites. Deletion is a tombstone applied to the indexes immediately (filtered out at query time) and physically removed at the next index compaction; the query orchestrator also checks the chunk store's live flag before using a passage, so a chunk deleted a second ago cannot be cited even if an index replica is lagging.

How are permissions enforced so that retrieval never leaks?

The ACL is stored on every chunk as a list of allowed group ids, and the filter is applied inside the retrieval call (pre-filtering in the ANN index and the keyword index), not after: post-filtering 50 results down to the 3 the user may read destroys recall and leaks existence through timing. When a document's ACL changes, its chunks are re-tagged by the ingest path as a new version. The prompt includes only passages that passed the filter, and the answer's citations are validated against that same set, so the model cannot cite something it was not given. For sensitive corpora, the chunk text itself is fetched with the user's credentials at query time, so the index holds vectors and ids and nothing readable.

How do we know the answers are grounded, and keep them that way?

An evaluation set of a few hundred questions with known relevant passages measures retrieval recall at k and, with an automated grader, answer faithfulness (every claim supported by a cited passage) and abstention (saying not found when nothing relevant exists). Every change to chunking, embedding model, fusion weights or prompt runs against it before deployment, as a batch job. In production, a sample of answers is graded continuously, and the citation validator rejects answers whose citations do not exist. The prompt instructs the model to answer only from the passages and to say so when they do not cover the question; the abstention rate is a tracked metric, because a system that never abstains is making things up.

6. What a strong answer sounds like

• Lay out the two pipelines, ingest and query, and put a latency number on every query stage summing to the budget.
• Justify hybrid retrieval plus reranking with the funnel argument, and chunking choices with an evaluation set, not opinion.
• Make versions and tombstones explicit so updates and deletes are safe, and pre-filter by ACL inside retrieval.
• Describe grounding as measured, with citation validation and an abstention path.
• Expected follow-ups: multi-turn questions (rewrite the question with context before retrieval); caching (identical questions and embeddings); cost control (smaller models for routing, fewer passages when confidence is high); structured sources (tables, SQL) alongside text.
• Common mistake: embedding whole documents, filtering permissions after retrieval, or skipping evaluation and tuning by anecdote.

02Where this question was reported

• Exponent · ML SD: 'retrieval-augmented generation pipeline for a large-scale search product'
• PracHub · PM: grounding misinformation moderation with a RAG pipeline
• Prepfully · Candidate reports: RAG pipeline for a large-scale search product; eval, LLM-as-judge reliability.