Apache Kafka Architecture

Brokers, topics, partitions, replication, consumer groups, delivery semantics, exactly-once, schema management, Kafka Connect, Kafka Streams and operating Kafka in production.

Core concepts

  • Topic: named, append-only log split into partitions (unit of parallelism and ordering).
  • Broker: server storing partitions; replication factor (typically 3) with a leader and followers; ISR (in-sync replicas).
  • Producer: writes records with a key (decides partition) and value.
  • Consumer group: consumers share partitions; each partition is consumed by one member of a group at a time; offsets track progress.
  • KRaft: Kafka's built-in Raft-based metadata quorum; ZooKeeper mode was removed in Kafka 4.0 (2025).
flowchart LR
  P1[Producer] -->|key=orderId| T
  subgraph T[Topic orders: 3 partitions, RF=3]
    PA[Partition 0<br/>leader broker 1]
    PB[Partition 1<br/>leader broker 2]
    PC[Partition 2<br/>leader broker 3]
  end
  PA --> C1[Consumer A1]
  PB --> C2[Consumer A2]
  PC --> C2
  PA & PB & PC --> G2[Consumer group B<br/>independent offsets]

Durability settings

Setting Recommended for critical data
replication.factor 3
min.insync.replicas 2
Producer acks all
Producer enable.idempotence true (default in modern clients)
unclean.leader.election.enable false

Delivery semantics

  • At-least-once (default with manual commit after processing): idempotent consumers required.
  • Exactly-once within Kafka: transactional producer + read_committed consumers for read-process-write pipelines (Kafka Streams processing.guarantee=exactly_once_v2).
  • End-to-end exactly-once with external systems requires idempotent sinks or outbox/inbox patterns.

Java consumer (Spring for Apache Kafka)

@KafkaListener(topics = "payments.captured.v1", groupId = "ledger", concurrency = "6")
void onCaptured(ConsumerRecord<String, PaymentCaptured> rec) {
    ledger.postIfAbsent(rec.value().paymentId(), rec.value());   // idempotent by business key
}

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);           // send to <topic>.DLT
    return new DefaultErrorHandler(recoverer, new ExponentialBackOff(200, 2.0));
}

Ecosystem

  • Schema Registry (Avro/Protobuf/JSON Schema) with compatibility rules.
  • Kafka Connect with Debezium (CDC) source connectors and sink connectors (S3, Elasticsearch, JDBC).
  • Kafka Streams / Flink for stream processing.
  • Tiered storage for long retention on object storage.

Operating Kafka

  • Partition count: based on target throughput and consumer parallelism; increasing partitions changes key-to-partition mapping (affects ordering).
  • Monitor consumer lag, under-replicated partitions, request latency, disk usage.
  • Managed options: Amazon MSK, Confluent Cloud, Azure Event Hubs (Kafka API), Google Managed Service for Apache Kafka, Aiven, Redpanda.

Interview questions and answers

How does Kafka guarantee ordering?

Only within a partition. Records with the same key go to the same partition (with the default partitioner), so per-key order is preserved if there is one producer per key path and retries are idempotent.

What happens when a consumer in a group crashes?

The group coordinator rebalances: its partitions are reassigned to other members, which resume from the last committed offsets, so messages processed but not committed are reprocessed (hence idempotency). Cooperative rebalancing reduces disruption.

How do you choose between Kafka and a queue like SQS?

Kafka for high-throughput streams, replay, multiple independent consumers and stream processing. SQS for simple task queues with per-message acknowledgement and no capacity planning.