Message Queues and Streaming
Queues vs pub/sub vs logs, competing consumers, dead-letter queues, visibility timeouts, ordering and when to choose SQS, Service Bus, RabbitMQ, Kafka or Pub/Sub.
What it is
Message brokers decouple producers from consumers so they can operate at different speeds and fail independently.
| Model | Semantics | Typical products |
|---|---|---|
| Queue (point-to-point) | Each message processed by one consumer; deleted after ack | Amazon SQS, Azure Service Bus queues, RabbitMQ queues, Google Cloud Tasks |
| Pub/sub topic | Each subscription gets a copy | SNS, Service Bus topics, Google Pub/Sub, RabbitMQ exchanges |
| Log / stream | Ordered, retained, replayable; consumers track offsets | Kafka, Kinesis, Event Hubs, Pulsar, Redpanda |
Key concepts
- Competing consumers: many workers pull from one queue for parallelism.
- Acknowledgement and visibility timeout: a message becomes visible again if not acknowledged in time; long processing needs heartbeat/extension.
- Dead-letter queue (DLQ): messages that fail N times are moved aside for inspection; alert on DLQ depth.
- Backpressure: bounded queues and consumer scaling based on lag (KEDA scales Kubernetes consumers on queue depth).
- Ordering: FIFO queues (SQS FIFO with message group IDs, Service Bus sessions) or log partitions.
- Retention and replay: logs keep data for days or forever (tiered storage); queues delete on ack.
flowchart LR P1[Producer] --> Q[[Queue]] P2[Producer] --> Q Q --> C1[Worker 1] Q --> C2[Worker 2] Q --> C3[Worker 3] Q -- after 5 failed attempts --> DLQ[[Dead-letter queue]] DLQ --> OPS[Alert + redrive tool]
Choosing
| Need | Prefer |
|---|---|
| Simple task queue, serverless, near-infinite scale | SQS, Cloud Tasks, Service Bus |
| Enterprise messaging features (sessions, transactions, scheduled messages, duplicate detection) | Azure Service Bus, RabbitMQ, ActiveMQ |
| Event streaming, replay, multiple independent consumers, stream processing | Kafka, Kinesis, Event Hubs, Pulsar |
| Global managed pub/sub with push to HTTP endpoints | Google Pub/Sub, SNS |
| Flexible routing patterns (topic, headers) | RabbitMQ exchanges |
Real-world example
An image platform puts upload events on SQS; a fleet of workers scales with queue depth to generate thumbnails. Failed images (corrupt files) go to a DLQ after three attempts. Separately, clickstream events flow through Kafka where analytics, recommendations and fraud consumers each read independently and can replay last week's data after a bug fix.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Smooths load spikes | Added latency and eventual consistency |
| Isolates failures | Another critical system to run and monitor |
| Enables retries and parallel processing | Duplicates and poison messages need handling |
Architect-level considerations
[!ARCHITECT] Size for the worst backlog: retention ≥ longest consumer outage; consumer capacity ≥ peak arrival rate or a documented drain time. Define DLQ ownership and a redrive process.
[!DEVELOPER] Make handlers idempotent, keep messages small (store large payloads in object storage and send a reference, the "claim check" pattern), and include correlation IDs for tracing.
Interview questions and answers
Kafka vs SQS?
Kafka is a partitioned, replayable log for high-throughput streams and multiple consumer groups, with ordering per partition and stream processing integration; it needs capacity planning. SQS is a fully managed queue with per-message acknowledgement, simple scaling and no replay (messages are deleted on ack). Use SQS for work queues, Kafka for event streams.
How do you handle a poison message?
Limit retries, move it to a DLQ with error context, alert, fix the bug or data, and redrive. Never let one message block a partition indefinitely; for ordered logs, park it to a retry topic.
How many partitions should a Kafka topic have?
Enough for target throughput and consumer parallelism (throughput ÷ per-partition throughput, and at least the max consumers you expect), with headroom, but not thousands per topic without reason, because partitions cost broker resources and increase rebalance time.