Retrieval-Augmented Generation (RAG)
The RAG pipeline end to end: ingestion, chunking, embeddings, hybrid retrieval, reranking, context assembly, generation with citations, advanced patterns (GraphRAG, agentic RAG) and failure modes.
For a full worked design see Enterprise RAG Knowledge Assistant.
Why RAG
LLMs do not know your private or recent data. RAG retrieves relevant content at query time and puts it in the prompt, so answers are grounded, citeable, up to date and permission-aware, without retraining.
Pipeline
flowchart LR
subgraph Offline ingestion
D[Documents] --> P[Parse & clean] --> CH[Chunk + metadata + ACLs] --> E[Embed] --> IX[(Hybrid index)]
end
subgraph Online query
Q[Question] --> RW[Rewrite / expand] --> R[Retrieve top-k<br/>BM25 + vector + filters]
R --> RR[Rerank] --> CTX[Assemble context<br/>with source IDs]
CTX --> LLM[Generate with citations] --> CHK[Grounding check] --> A[Answer]
end
IX --> RDesign decisions
| Decision | Options | Guidance |
|---|---|---|
| Chunking | Fixed size, structure-aware (headings), semantic, parent-child | Structure-aware 300–800 tokens with overlap; store parent sections for expansion |
| Retrieval | Vector only, keyword only, hybrid | Hybrid (BM25 + vectors) with reciprocal rank fusion usually wins |
| Reranking | None, cross-encoder, LLM reranker | Cross-encoder reranking of top 30–100 gives large precision gains |
| k (chunks in context) | 3–20 | Fewer, better chunks beat many mediocre ones |
| Query transformation | Rewrite follow-ups, multi-query, HyDE | Rewrite conversational questions into standalone queries |
| Metadata filters | Date, product, region, ACL | Always apply permission filters inside retrieval |
Advanced patterns
- Agentic RAG: the model decides when and what to retrieve, possibly iteratively, using search as a tool.
- GraphRAG: build a knowledge graph of entities/relationships for multi-hop questions and corpus-level summaries (Microsoft Research GraphRAG).
- Contextual retrieval: prepend chunk-specific context (document title, section summary) before embedding to improve recall.
- Structured + unstructured: route to SQL (text-to-SQL with guardrails) for numeric questions and to RAG for document questions.
Failure modes and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Correct document exists but answer says "not found" | Poor recall (chunking, embedding, query mismatch) | Hybrid search, query rewriting, better chunking, evaluate recall@k |
| Answer cites wrong source | Context noise | Reranking, fewer chunks, citation instructions |
| Outdated answers | Stale index, duplicate versions | Incremental sync, version metadata, deduplicate |
| Leaked restricted content | No ACL filtering | Permission-aware retrieval |
| Slow answers | Large context, sequential steps | Rerank to fewer chunks, parallelize, smaller models for rewriting |
Interview questions and answers
RAG or long context windows?
Long context lets you include whole documents for small corpora, but cost and latency grow with tokens and quality can drop for information buried in the middle. RAG scales to large corpora, enforces permissions and keeps costs bounded. They combine well: retrieve the right documents, then include larger sections.
How do you measure retrieval quality?
Labelled question → relevant chunk sets; compute recall@k, precision@k, MRR or nDCG; track separately from generation quality.