Rate Limiting

Token bucket, leaky bucket, fixed and sliding windows; distributed rate limiting with Redis; quotas per tenant; and returning proper 429 responses.

What it is

Rate limiting controls how many requests a client (user, API key, IP, tenant) may make in a time window. It protects services from overload and abuse, enforces commercial quotas, and limits the cost of attacks such as credential stuffing.

Algorithms

Algorithm How Pros Cons
Token bucket Bucket refills at rate r up to capacity b; each request takes a token Allows bursts up to b, smooth average Two parameters to tune
Leaky bucket Requests queue and leave at a fixed rate Smooth output Adds latency; queue can fill
Fixed window counter Count per calendar window (e.g. per minute) Simple, cheap Burst of 2× at window boundaries
Sliding window log Store timestamps of requests Exact Memory heavy
Sliding window counter Weighted mix of current and previous window counts Good accuracy, cheap Approximation
flowchart LR
  C[Client] --> GW[API gateway]
  GW --> RL{Rate limiter<br/>key = api_key + route}
  RL -- tokens available --> S[Service]
  RL -- exhausted --> E[429 Too Many Requests<br/>Retry-After header]
  RL <--> R[(Redis<br/>atomic Lua / counters)]

Distributed implementation (token bucket in Redis)

-- KEYS[1] = bucket key; ARGV = capacity, refill_per_sec, now_ms, cost
local cap, rate, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local allowed = tokens >= cost
if allowed then tokens = tokens - cost end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000) * 2)
return allowed and 1 or 0

Running the logic as one Lua script makes it atomic across concurrent gateway instances.

Real-world example

A public API offers plans: Free (10 requests/s, 10K/day), Pro (100 requests/s, 1M/day). The gateway enforces per-key token buckets for burst control and daily quotas for billing, returns 429 with Retry-After and rate-limit headers, and applies a stricter limit on /login per IP and per account to slow credential stuffing.

Advantages and disadvantages

Advantages Disadvantages
Protects capacity and fairness between tenants Legitimate spikes get rejected
Mitigates brute force and scraping Distributed counters add latency and a dependency
Enables monetized plans Choosing keys is hard behind NAT/shared IPs

Technology options

API gateways (AWS API Gateway usage plans, Azure API Management policies, Apigee quotas, Kong, Envoy rate limit service), WAF rate-based rules (AWS WAF, Cloud Armor, Azure WAF), Redis-based libraries (Bucket4j for Java, resilience4j RateLimiter for in-process limits).

Architect-level considerations

[!ARCHITECT] Layer limits: coarse IP/bot limits at the edge (WAF), per-key/tenant limits at the gateway, and concurrency limits inside services. Decide fail-open vs fail-closed if the limiter's store is down (fail-open with local fallback limits is common for availability).

[!DEVELOPER] Clients should honour Retry-After and back off with jitter.

Interview questions and answers

Design a rate limiter for 1 million API keys across 20 gateway nodes.

Token bucket per key in Redis Cluster using an atomic Lua script, keys sharded by API key, short TTLs to evict idle buckets; local in-memory pre-check to absorb obvious abuse; fail-open with conservative local limits if Redis is unreachable; return 429 with headers.

Why is a fixed window problematic?

A client can send the full limit at the end of one window and again at the start of the next, doubling the burst rate. Sliding windows or token buckets smooth this.

Rate limiting vs throttling vs load shedding?

Rate limiting enforces client quotas; throttling slows requests (queues or delays) rather than rejecting; load shedding rejects based on server health and priority.