Model and LLM Application Evaluation
Offline and online evaluation, golden datasets, metrics for classification, retrieval and generation, LLM-as-judge with calibration, A/B tests and evaluation in CI/CD.
Why evaluation is the core discipline
Without evaluation, every prompt, model or retrieval change is a guess. Treat evals like tests: versioned datasets, metrics, thresholds and regression gates.
What to measure
| System | Metrics |
|---|---|
| Classical ML classifier | Precision, recall, F1, ROC-AUC/PR-AUC, calibration, cost-weighted metrics, fairness across groups |
| Retrieval | Recall@k, precision@k, MRR, nDCG |
| RAG generation | Faithfulness/groundedness, answer relevance, answer correctness, citation accuracy |
| Agents | Task success rate, steps, tool-call accuracy, cost and latency per task, safety violations |
| Production | User feedback, deflection/resolution rate, escalation rate, latency, cost per successful outcome |
Building a golden dataset
- Collect real questions/tasks (logs, SMEs), including edge cases and adversarial inputs.
- Label expected answers, key facts, relevant sources, or rubrics.
- Stratify by category and difficulty; keep a held-out set.
- Grow it from production failures (every bad answer becomes a test case).
LLM-as-judge
Use a strong model with a precise rubric to score outputs at scale. Calibrate against human labels (agreement rate), use pairwise comparisons for subtle differences, avoid judging with the same model family when possible, and spot-check regularly.
flowchart LR
DS[(Golden dataset)] --> RUN[Run system version N]
RUN --> OUT[Outputs + traces]
OUT --> J[Automated metrics +<br/>LLM judge + rules]
J --> REP[Scorecard vs baseline]
REP --> GATE{Thresholds met?}
GATE -- yes --> DEPLOY[Canary deploy]
GATE -- no --> FIX[Investigate & fix]
DEPLOY --> ONLINE[Online metrics<br/>A/B, feedback]
ONLINE --> DSTools
Ragas, DeepEval, promptfoo, OpenAI Evals, LangSmith, Langfuse, Arize Phoenix, Braintrust; managed evaluation in Bedrock, Vertex AI and Azure AI Foundry.
Interview questions and answers
How do you know a new model version is better for your use case?
Run both on the golden dataset with the same prompts and retrieval, compare scorecards (quality, latency, cost), review disagreements manually, then canary in production with online metrics before full rollout.