Scalability

Vertical vs horizontal scaling, stateless services, scaling the data tier, autoscaling and the bottlenecks that actually stop systems from growing.

What it is

Scalability is the ability to handle more load (users, requests, data) by adding resources, with cost and latency growing in proportion or better. A system that handles 10× the load at 10× the cost and the same latency scales linearly; one that needs 100× the cost does not scale in practice.

  • Vertical scaling (scale up): bigger machines. Simple, no code changes, but has a ceiling and a single failure domain.
  • Horizontal scaling (scale out): more machines. Needs stateless services or partitioned state, load balancing and distributed coordination.

The scaling playbook (in order)

  1. Measure where time is spent (profiling, tracing). Many "scale" problems are a missing index.
  2. Make the application tier stateless: sessions in Redis or tokens, files in object storage. Then add instances behind a load balancer and autoscale.
  3. Cache reads at CDN, service and database levels.
  4. Offload work asynchronously with queues (emails, thumbnails, reports).
  5. Scale reads with read replicas and CQRS read models.
  6. Scale writes with partitioning/sharding or a horizontally scalable store.
  7. Split by function (services) when teams or workloads scale differently.
  8. Scale geographically with regional deployments and data locality.
flowchart LR
  C[Clients] --> CDN[CDN / edge cache]
  CDN --> LB[Load balancer]
  LB --> A1[App instance]
  LB --> A2[App instance]
  LB --> A3[App instance ...autoscaled]
  A1 & A2 & A3 --> R[(Redis cache)]
  A1 & A2 & A3 --> Q[[Queue]]
  Q --> W[Workers]
  A1 & A2 & A3 --> P[(Primary DB)]
  P --> RR1[(Read replica)]
  P --> RR2[(Read replica)]

Real-world example

A ticketing site sells concert tickets. Browsing is read-heavy and cacheable; seat reservation is write-heavy on a few hot rows. Horizontal app scaling and CDN caching solve browsing. The hot-seat problem is solved differently: a per-event reservation service with seats partitioned in memory (or Redis with atomic operations), a virtual waiting room to cap concurrency, and the relational DB as the durable record.

Advantages and disadvantages

Approach Advantages Disadvantages
Vertical No code change, strong consistency stays simple Hard ceiling, expensive at top end, downtime to resize
Horizontal (stateless app) Near-linear, fault tolerant, elastic Needs externalized state, load balancing
Read replicas Cheap read scaling Replication lag, stale reads
Sharding Scales writes and storage Cross-shard queries, rebalancing, operational load
Functional split (services) Independent scaling and teams Network calls, distributed data, ops overhead

Technology options

Need Options
Autoscaling compute Kubernetes HPA/KEDA, Karpenter, AWS Auto Scaling, Azure VMSS, GCP MIGs, Cloud Run, Lambda
Horizontally scalable SQL Spanner, Aurora DSQL, CockroachDB, YugabyteDB, Citus, Vitess
Horizontally scalable NoSQL DynamoDB, Cassandra/ScyllaDB, Bigtable, Cosmos DB, MongoDB sharding
Caching Redis/Valkey, Memcached, CDN

Architect-level considerations

[!ARCHITECT] Identify the scaling unit for each component and the first bottleneck. Usually it is the database primary, a hot key, or a synchronous dependency. Design partition keys early; changing them later is a migration.

[!WARNING] Hidden limits Connection limits (thousands of app pods × pool size can exhaust a database), cloud API quotas, NAT gateway port limits, and per-partition throughput limits in managed NoSQL often break before CPU does.

[!CTO] Scalability you do not need yet is cost and complexity. Design so that the next 10× is a known path, not necessarily built.

Interview questions and answers

Your API is slow under load. CPU on app servers is 30%. Where do you look?

Downstream contention: database (locks, slow queries, connection pool exhaustion), external calls, thread pool saturation, GC pauses. Use tracing to see where latency accumulates. Low CPU with high latency usually means waiting, not computing.

How do you scale a system with a single hot key, like one viral post's like counter?

Split the counter into N sub-counters updated randomly and summed on read (or periodically aggregated), cache the read, and accept slight staleness. For ordering-sensitive cases, route the key to a single-writer partition and batch updates.

When does scaling up beat scaling out?

When the workload fits on a large machine with headroom, needs strong transactional consistency, and the team lacks distributed-systems maturity. Modern instances with hundreds of cores and terabytes of RAM carry many businesses a long way.