AI Observability
Tracing LLM calls, retrieval and tool steps; tracking tokens, cost, latency and quality; drift monitoring for ML models; and OpenTelemetry semantic conventions for GenAI.
What to capture
| Signal | Examples |
|---|---|
| Traces | Each step: query rewrite, retrieval (results and scores), rerank, LLM call (model, prompt version, tokens), tool calls, guardrail decisions |
| Metrics | Latency (time to first token, total), tokens in/out, cost per request/feature/team, error and refusal rates, cache hit rates |
| Quality | User feedback, online LLM-judge scores on samples, grounding failures, escalation rates |
| Safety | Injection detections, blocked outputs, PII redactions |
| ML model health | Feature drift (PSI, KL divergence), prediction drift, performance on delayed labels |
OpenTelemetry has GenAI semantic conventions (attributes for model, tokens, operations) so LLM telemetry can flow into standard observability backends. Specialized tools: Langfuse, Arize Phoenix, LangSmith, Helicone, Datadog LLM Observability, cloud-native tracing in Bedrock/Vertex/Foundry.
flowchart LR APP[AI application<br/>OTel GenAI instrumentation] --> COL[OTel collector<br/>redact PII] COL --> TR[(Traces)] COL --> MT[(Metrics: tokens, cost, latency)] TR --> EVAL[Sampled online evals] EVAL --> DASH[Quality & cost dashboards] MT --> DASH DASH --> ALERT[Alerts: cost spikes,<br/>quality drops, safety events]
Privacy
Prompts and outputs may contain personal data. Redact before storage, restrict access, set retention, and honour data subject requests.
Interview questions and answers
The cost of an AI feature doubled overnight. How would observability help?
Cost metrics broken down by feature, model, prompt version and user/tenant show where tokens increased (e.g. a prompt change added large context, a retry loop, an abusive user, or cache misses after a deployment), enabling a targeted fix and alerts for the future.