After a release, p99 latency of a critical API jumped from 200 ms to 3 s, but averages look fine. How do you investigate?

Short answer

Correlate with the release via change tracking, compare traces of slow vs fast requests to find where time goes, check for new downstream calls, query plan changes, lock contention, GC pauses, pool exhaustion and retries, then roll back or feature-flag the change while fixing the root cause.

Detailed answer

  1. Mitigate first: if the release correlates, roll back or disable the feature flag; protect users.
  2. Scope: which endpoints, tenants, regions, instances? Tail-only issues often affect a subset of inputs or nodes.
  3. Traces: compare slow and fast traces; look for new spans, retries, or time waiting for connections.
  4. Dependencies: new N+1 queries, missing index (EXPLAIN ANALYZE), a new synchronous call, cache key change causing misses.
  5. Runtime: GC logs (pause spikes), thread/connection pool saturation metrics, CPU throttling in containers.
  6. Retries/timeouts: misconfigured retries can multiply latency.
  7. Fix and verify with a load test reproducing the tail; add a regression performance test.

Real-world example

A common case: a change added an ORM relationship that triggers lazy loading for a subset of customers with many items, causing hundreds of queries only for large accounts, invisible in averages.

Follow-up questions

  • Why look at p99 and not average? (Tail affects many users in fan-out; averages hide it.)
  • How do you prevent recurrence? (Performance budgets in CI, canary analysis on p99.)

What the interviewer expects

Structured, evidence-driven debugging with mitigation first and knowledge of common causes.

Common mistakes to avoid

Scaling up instances without understanding cause; tuning JVM flags before looking at traces and queries.