You are building a new platform of 30 services. What is your observability design?
Short answer
Standardize on OpenTelemetry for traces, metrics and logs with consistent resource attributes and W3C trace context propagation, structured JSON logs with trace IDs, RED/USE metrics and SLOs per critical journey with burn-rate alerting, an OTel collector tier for sampling and redaction, and dashboards and runbooks owned by teams.
Detailed answer
- Instrumentation: OTel SDK/agents in a shared starter; auto-instrument HTTP, DB, messaging.
- Conventions: service.name, deployment.environment, version, tenant (where allowed).
- Pipeline: OTel collectors (agent + gateway) → metrics (Prometheus/Mimir or cloud), logs (Loki/OpenSearch/cloud), traces (Tempo/Jaeger/cloud APM).
- SLOs: availability and latency for top journeys; burn-rate alerts; error budgets.
- Cost: tail sampling for traces, log levels, retention tiers.
- Privacy: redact PII at the collector.
What the interviewer expects
Vendor-neutral standards, correlation across signals, SLO-based alerting and cost awareness.
Common mistakes to avoid
Unstructured logs, per-team tooling sprawl, alerting on every metric.