Model and LLM Application Evaluation

Offline and online evaluation, golden datasets, metrics for classification, retrieval and generation, LLM-as-judge with calibration, A/B tests and evaluation in CI/CD.

Why evaluation is the core discipline

Without evaluation, every prompt, model or retrieval change is a guess. Treat evals like tests: versioned datasets, metrics, thresholds and regression gates.

What to measure

System Metrics
Classical ML classifier Precision, recall, F1, ROC-AUC/PR-AUC, calibration, cost-weighted metrics, fairness across groups
Retrieval Recall@k, precision@k, MRR, nDCG
RAG generation Faithfulness/groundedness, answer relevance, answer correctness, citation accuracy
Agents Task success rate, steps, tool-call accuracy, cost and latency per task, safety violations
Production User feedback, deflection/resolution rate, escalation rate, latency, cost per successful outcome

Building a golden dataset

  1. Collect real questions/tasks (logs, SMEs), including edge cases and adversarial inputs.
  2. Label expected answers, key facts, relevant sources, or rubrics.
  3. Stratify by category and difficulty; keep a held-out set.
  4. Grow it from production failures (every bad answer becomes a test case).

LLM-as-judge

Use a strong model with a precise rubric to score outputs at scale. Calibrate against human labels (agreement rate), use pairwise comparisons for subtle differences, avoid judging with the same model family when possible, and spot-check regularly.

flowchart LR
  DS[(Golden dataset)] --> RUN[Run system version N]
  RUN --> OUT[Outputs + traces]
  OUT --> J[Automated metrics +<br/>LLM judge + rules]
  J --> REP[Scorecard vs baseline]
  REP --> GATE{Thresholds met?}
  GATE -- yes --> DEPLOY[Canary deploy]
  GATE -- no --> FIX[Investigate & fix]
  DEPLOY --> ONLINE[Online metrics<br/>A/B, feedback]
  ONLINE --> DS

Tools

Ragas, DeepEval, promptfoo, OpenAI Evals, LangSmith, Langfuse, Arize Phoenix, Braintrust; managed evaluation in Bedrock, Vertex AI and Azure AI Foundry.

Interview questions and answers

How do you know a new model version is better for your use case?

Run both on the golden dataset with the same prompts and retrieval, compare scorecards (quality, latency, cost), review disagreements manually, then canary in production with online metrics before full rollout.