Message Queues and Streaming

Queues vs pub/sub vs logs, competing consumers, dead-letter queues, visibility timeouts, ordering and when to choose SQS, Service Bus, RabbitMQ, Kafka or Pub/Sub.

What it is

Message brokers decouple producers from consumers so they can operate at different speeds and fail independently.

Model Semantics Typical products
Queue (point-to-point) Each message processed by one consumer; deleted after ack Amazon SQS, Azure Service Bus queues, RabbitMQ queues, Google Cloud Tasks
Pub/sub topic Each subscription gets a copy SNS, Service Bus topics, Google Pub/Sub, RabbitMQ exchanges
Log / stream Ordered, retained, replayable; consumers track offsets Kafka, Kinesis, Event Hubs, Pulsar, Redpanda

Key concepts

  • Competing consumers: many workers pull from one queue for parallelism.
  • Acknowledgement and visibility timeout: a message becomes visible again if not acknowledged in time; long processing needs heartbeat/extension.
  • Dead-letter queue (DLQ): messages that fail N times are moved aside for inspection; alert on DLQ depth.
  • Backpressure: bounded queues and consumer scaling based on lag (KEDA scales Kubernetes consumers on queue depth).
  • Ordering: FIFO queues (SQS FIFO with message group IDs, Service Bus sessions) or log partitions.
  • Retention and replay: logs keep data for days or forever (tiered storage); queues delete on ack.
flowchart LR
  P1[Producer] --> Q[[Queue]]
  P2[Producer] --> Q
  Q --> C1[Worker 1]
  Q --> C2[Worker 2]
  Q --> C3[Worker 3]
  Q -- after 5 failed attempts --> DLQ[[Dead-letter queue]]
  DLQ --> OPS[Alert + redrive tool]

Choosing

Need Prefer
Simple task queue, serverless, near-infinite scale SQS, Cloud Tasks, Service Bus
Enterprise messaging features (sessions, transactions, scheduled messages, duplicate detection) Azure Service Bus, RabbitMQ, ActiveMQ
Event streaming, replay, multiple independent consumers, stream processing Kafka, Kinesis, Event Hubs, Pulsar
Global managed pub/sub with push to HTTP endpoints Google Pub/Sub, SNS
Flexible routing patterns (topic, headers) RabbitMQ exchanges

Real-world example

An image platform puts upload events on SQS; a fleet of workers scales with queue depth to generate thumbnails. Failed images (corrupt files) go to a DLQ after three attempts. Separately, clickstream events flow through Kafka where analytics, recommendations and fraud consumers each read independently and can replay last week's data after a bug fix.

Advantages and disadvantages

Advantages Disadvantages
Smooths load spikes Added latency and eventual consistency
Isolates failures Another critical system to run and monitor
Enables retries and parallel processing Duplicates and poison messages need handling

Architect-level considerations

[!ARCHITECT] Size for the worst backlog: retention ≥ longest consumer outage; consumer capacity ≥ peak arrival rate or a documented drain time. Define DLQ ownership and a redrive process.

[!DEVELOPER] Make handlers idempotent, keep messages small (store large payloads in object storage and send a reference, the "claim check" pattern), and include correlation IDs for tracing.

Interview questions and answers

Kafka vs SQS?

Kafka is a partitioned, replayable log for high-throughput streams and multiple consumer groups, with ordering per partition and stream processing integration; it needs capacity planning. SQS is a fully managed queue with per-message acknowledgement, simple scaling and no replay (messages are deleted on ack). Use SQS for work queues, Kafka for event streams.

How do you handle a poison message?

Limit retries, move it to a DLQ with error context, alert, fix the bug or data, and redrive. Never let one message block a partition indefinitely; for ordered logs, park it to a retry topic.

How many partitions should a Kafka topic have?

Enough for target throughput and consumer parallelism (throughput ÷ per-partition throughput, and at least the max consumers you expect), with headroom, but not thousands per topic without reason, because partitions cost broker resources and increase rebalance time.