Latency and Throughput
Percentiles, tail latency, Little's Law, the relationship between utilization and queueing delay, and practical techniques to cut latency and raise throughput.
What it is
- Latency: time to complete one operation (for example, server time for an API call). Measure distributions: p50, p95, p99, p99.9.
- Throughput: operations completed per unit time (RPS, messages/s, MB/s).
They interact through queueing. Little's Law: average items in system = arrival rate × average time in system (L = λW). If an API receives 1,000 RPS and each request takes 200 ms, about 200 requests are in flight; with a pool of 100 threads, requests queue and latency rises.
Why the tail matters
If a page makes 50 backend calls in parallel and each has a p99 of 100 ms, the chance that at least one call hits its p99 is 1 − 0.99⁵⁰ ≈ 39%. Fan-out turns tail latency into typical latency, which is why large systems use hedged requests, tight timeouts and fewer synchronous hops (Dean and Barroso, "The Tail at Scale", 2013).
As utilization approaches 100%, queueing delay grows sharply (non-linearly). Running a service at 90% CPU usually means poor latency; 50% to 70% is a common target for latency-sensitive tiers.
flowchart LR
subgraph Critical path
C[Client] -->|DNS + TLS + RTT| E[Edge/CDN]
E --> G[Gateway]
G --> S[Service]
S -->|cache hit 1 ms| R[(Cache)]
S -->|miss 5-20 ms| D[(Database)]
end
S -. async .-> Q[[Queue for<br/>non-critical work]]Techniques
Lower latency
- Move data closer: CDN, edge compute, regional replicas.
- Cache aggressively; precompute (materialized views, feeds).
- Remove synchronous hops; parallelize independent calls; batch round trips.
- Connection reuse (HTTP/2, keep-alive), TLS session resumption, efficient serialization (Protobuf).
- Fix the database: indexes, avoiding N+1 queries, right isolation level.
Higher throughput
- Horizontal scaling and partitioning.
- Batching and pipelining (Kafka producer batching, bulk DB inserts).
- Asynchronous, non-blocking I/O or virtual threads for I/O-bound services.
- Avoid shared locks and hot keys.
Advantages and disadvantages of common techniques
| Technique | Helps | Costs |
|---|---|---|
| Batching | Throughput | Adds latency per item |
| Caching | Latency and throughput | Staleness, invalidation complexity |
| Async processing | Perceived latency | Eventual consistency, more moving parts |
| Hedged requests | Tail latency | Extra load (cap at a few %) |
| Compression | Bandwidth/latency on slow links | CPU |
Technology options
APM and tracing (OpenTelemetry, Jaeger, Grafana Tempo, Datadog, Dynatrace, New Relic, AWS X-Ray, Application Insights, Cloud Trace), profilers (async-profiler, JFR), load testing (k6, Gatling), CDNs, Redis, gRPC.
Architect-level considerations
[!ARCHITECT] Set latency SLOs as percentiles at a stated throughput. Draw the critical path and assign a latency budget per hop. Keep non-critical work off the critical path.
[!DEVELOPER] Never report averages alone. Use histograms; check for coordinated omission in load tests (tools like wrk2 and Gatling handle it better than naive clients).
Interview questions and answers
Your p50 is 40 ms but p99 is 2 s. What do you investigate?
GC pauses, lock contention, cold caches, slow queries for specific inputs, retries hitting timeouts, noisy neighbours, a slow dependency or shard, and queueing at saturated pools. Compare traces of slow vs fast requests to find what differs.
How do you increase throughput of a Kafka consumer group?
Add partitions and consumers (up to one consumer per partition), process in batches, make processing asynchronous within ordering constraints, tune fetch sizes, and remove slow synchronous calls from the processing path.
What is Little's Law useful for in practice?
Sizing pools and concurrency: required concurrency ≈ throughput × latency. For 2,000 RPS at 50 ms you need about 100 concurrent workers or connections, plus headroom.