AI Observability

Tracing LLM calls, retrieval and tool steps; tracking tokens, cost, latency and quality; drift monitoring for ML models; and OpenTelemetry semantic conventions for GenAI.

What to capture

Signal Examples
Traces Each step: query rewrite, retrieval (results and scores), rerank, LLM call (model, prompt version, tokens), tool calls, guardrail decisions
Metrics Latency (time to first token, total), tokens in/out, cost per request/feature/team, error and refusal rates, cache hit rates
Quality User feedback, online LLM-judge scores on samples, grounding failures, escalation rates
Safety Injection detections, blocked outputs, PII redactions
ML model health Feature drift (PSI, KL divergence), prediction drift, performance on delayed labels

OpenTelemetry has GenAI semantic conventions (attributes for model, tokens, operations) so LLM telemetry can flow into standard observability backends. Specialized tools: Langfuse, Arize Phoenix, LangSmith, Helicone, Datadog LLM Observability, cloud-native tracing in Bedrock/Vertex/Foundry.

flowchart LR
  APP[AI application<br/>OTel GenAI instrumentation] --> COL[OTel collector<br/>redact PII]
  COL --> TR[(Traces)]
  COL --> MT[(Metrics: tokens, cost, latency)]
  TR --> EVAL[Sampled online evals]
  EVAL --> DASH[Quality & cost dashboards]
  MT --> DASH
  DASH --> ALERT[Alerts: cost spikes,<br/>quality drops, safety events]

Privacy

Prompts and outputs may contain personal data. Redact before storage, restrict access, set retention, and honour data subject requests.

Interview questions and answers

The cost of an AI feature doubled overnight. How would observability help?

Cost metrics broken down by feature, model, prompt version and user/tenant show where tokens increased (e.g. a prompt change added large context, a retry loop, an abusive user, or cache misses after a deployment), enabling a targeted fix and alerts for the future.