How do you measure the business value of a GenAI customer support assistant?
Short answer
Define outcome metrics before launch (containment/deflection rate, resolution rate, average handle time for agents, customer satisfaction, escalation quality), measure them against a control group or baseline, include full costs (models, platform, content maintenance, oversight), and monitor quality and safety incidents alongside value.
Detailed answer
| Metric | Description |
|---|---|
| Containment rate | Conversations resolved without human agent |
| Resolution accuracy | Correct answers (sampled review + evals) |
| CSAT / NPS | Compared with human-handled baseline |
| Agent handle time | For agent-assist variants |
| Cost per resolved contact | Total cost / resolved contacts |
| Escalation quality | Context passed to humans |
Use A/B tests or phased rollout by segment to attribute impact; report value net of costs and risks.
What the interviewer expects
Outcome orientation, rigorous attribution, full cost accounting.
Common mistakes to avoid
Reporting usage (messages sent) as value.