Observability
Logs, metrics, traces and profiles; OpenTelemetry; SLO-based alerting; correlation across services; and controlling observability cost.
What it is
Observability is the ability to understand a system's internal state from its outputs, so you can answer new questions about production behaviour without shipping new code. Monitoring checks known failure conditions; observability helps debug unknown ones.
| Signal | Answers | Examples |
|---|---|---|
| Metrics | Is something wrong? How much? Trends | Request rate, error rate, duration (RED); utilization, saturation, errors (USE) |
| Logs | What exactly happened in this event? | Structured JSON with trace_id, user/tenant ID (no secrets) |
| Traces | Where did time go across services? | Spans per hop with attributes |
| Profiles | Which code consumes CPU/memory? | Continuous profiling (async-profiler, Pyroscope) |
| Events/changes | What changed recently? | Deployments, config changes, feature flags |
Architecture
flowchart LR
subgraph Services
S1[Service A<br/>OTel SDK] --> COL
S2[Service B<br/>OTel SDK] --> COL
S3[Service C<br/>OTel agent] --> COL
end
COL[OpenTelemetry Collector<br/>batch, sample, redact] --> M[(Metrics store<br/>Prometheus / Mimir / CloudWatch)]
COL --> L[(Log store<br/>Loki / OpenSearch / Cloud Logging)]
COL --> T[(Trace store<br/>Tempo / Jaeger / X-Ray)]
M & L & T --> G[Dashboards + SLOs<br/>Grafana / vendor APM]
G --> AL[Alerting / on-call]SLO-based alerting
Alert on symptoms users feel (error rate, latency against SLO) rather than causes (CPU 80%). Use burn-rate alerts: page when the error budget is being consumed fast enough to exhaust it within hours; ticket when slow burn threatens it within days (Google SRE Workbook approach).
Real-world example
A checkout incident: the SLO dashboard shows the checkout success rate dropping. Exemplars link a latency spike to traces showing 2-second waits on the tax service. The tax service's logs (joined by trace_id) show TLS handshake errors after a certificate rotation that went out 10 minutes earlier. Time to diagnose: minutes, because signals are correlated and changes are recorded.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Faster incident detection and diagnosis | Cost grows quickly with log volume and metric cardinality |
| Data for capacity and performance work | PII leakage risk in logs |
| Enables SLOs and error budgets | Instrumentation effort and consistency across teams |
Technology options
OpenTelemetry (vendor-neutral standard for traces, metrics, logs), Prometheus + Grafana stack (Mimir, Loki, Tempo), Elastic/OpenSearch, cloud-native (CloudWatch + X-Ray/Application Signals, Azure Monitor + Application Insights, Google Cloud Observability), commercial APM (Datadog, Dynatrace, New Relic, Honeycomb, Splunk).
Architect-level considerations
[!EA] Standardize on OpenTelemetry, common attribute names (service.name, deployment.environment, tenant), trace context propagation (W3C Trace Context), and log redaction rules.
[!CTO] Observability is often 5% to 15% of infrastructure spend at scale. Manage it like any other cost: sampling, retention tiers, dropping debug logs in production, cardinality budgets.
Interview questions and answers
What would you put in every log line?
Timestamp, level, service, environment, version, trace and span IDs, request ID, tenant/user identifiers where allowed (pseudonymized), event name and key business identifiers, never secrets or full card numbers.
Head-based vs tail-based sampling?
Head-based decides at the start of a trace (cheap, may miss rare errors). Tail-based decides after the trace completes (keeps errors and slow traces) but requires buffering in a collector tier.
How do you avoid alert fatigue?
Page only on user-impacting SLO burn, route the rest to tickets, deduplicate, include runbook links, and review every page in on-call retrospectives.