Your RAG chatbot gives confident wrong answers 15% of the time. How do you diagnose and improve it?
Short answer
Separate retrieval failures from generation failures using an evaluation set: measure whether the right chunks were retrieved (recall@k) and whether answers are faithful to retrieved context. Fix retrieval with better parsing, chunking, hybrid search, query rewriting and reranking; fix generation with grounding instructions, citations, refusal behaviour and output grounding checks; then gate changes with evals.
Detailed answer
- Build or expand the golden set from failing conversations with labelled relevant sources and correct answers.
- Classify failures:
- Retrieval miss (relevant chunk not in top-k) → parsing/chunking issues, vocabulary mismatch, missing documents, stale index.
- Retrieval noise (wrong chunks ranked high) → add reranker, metadata filters, fewer chunks.
- Generation unfaithful despite good context → stronger instructions, citations required, lower temperature, smaller context, better model.
- Question unanswerable → teach "not found" behaviour.
- Improve and measure each change on the eval set; deploy with canary and online feedback.
flowchart LR F[Failing answers] --> L[Label: retrieval miss / noise /<br/>unfaithful / unanswerable] L --> R[Fix retrieval:<br/>chunking, hybrid, rerank] L --> G[Fix generation:<br/>prompt, citations, model] R & G --> E[Re-run eval set] --> D[Deploy if metrics improve]
What the interviewer expects
Systematic diagnosis splitting retrieval and generation, with evaluation as the backbone.
Common mistakes to avoid
Switching to a bigger model without measuring retrieval; tuning prompts on a handful of anecdotes.