Your RAG chatbot gives confident wrong answers 15% of the time. How do you diagnose and improve it?

Short answer

Separate retrieval failures from generation failures using an evaluation set: measure whether the right chunks were retrieved (recall@k) and whether answers are faithful to retrieved context. Fix retrieval with better parsing, chunking, hybrid search, query rewriting and reranking; fix generation with grounding instructions, citations, refusal behaviour and output grounding checks; then gate changes with evals.

Detailed answer

  1. Build or expand the golden set from failing conversations with labelled relevant sources and correct answers.
  2. Classify failures:
    • Retrieval miss (relevant chunk not in top-k) → parsing/chunking issues, vocabulary mismatch, missing documents, stale index.
    • Retrieval noise (wrong chunks ranked high) → add reranker, metadata filters, fewer chunks.
    • Generation unfaithful despite good context → stronger instructions, citations required, lower temperature, smaller context, better model.
    • Question unanswerable → teach "not found" behaviour.
  3. Improve and measure each change on the eval set; deploy with canary and online feedback.
flowchart LR
  F[Failing answers] --> L[Label: retrieval miss / noise /<br/>unfaithful / unanswerable]
  L --> R[Fix retrieval:<br/>chunking, hybrid, rerank]
  L --> G[Fix generation:<br/>prompt, citations, model]
  R & G --> E[Re-run eval set] --> D[Deploy if metrics improve]

What the interviewer expects

Systematic diagnosis splitting retrieval and generation, with evaluation as the backbone.

Common mistakes to avoid

Switching to a bigger model without measuring retrieval; tuning prompts on a handful of anecdotes.