What is Kafka

A distributed, partitioned, replicated append-only log. Not a queue — the difference is that consuming does not remove anything, and every answer below follows from that one fact.

Verified 2026-08

Verified 2026-08. Kafka 4.x: ZooKeeper is fully removed, KRaft only. Naming ZooKeeper as a live dependency dates you by two major versions.

The model

topic "orders"
  partition 0 │ 0 │ 1 │ 2 │ 3 │ 4 │ ...   ← append only
  partition 1 │ 0 │ 1 │ 2 │ ...
  partition 2 │ 0 │ 1 │ 2 │ 3 │ ...
                      ▲         ▲
              group A offset   group B offset

A partition is the unit of ordering, parallelism and storage. A consumer group is a set of consumers sharing the work, and each group holds its own offsets — which is why adding a consumer group next year replays the whole history rather than stealing another group’s messages.

Ordering is per partition, never per topic

The most-asked Kafka question, and the answer is a trade rather than a feature:

# Same key -> same partition -> ordered relative to each other.
producer.send("orders", key=order_id.encode(), value=payload)

Kafka guarantees order within a partition. Global ordering across a topic means one partition, which means one consumer and no parallelism. So you pick a key that groups what must be ordered — order_id, account_id — and accept that events for different keys may interleave.

Gotcha: the partition count is effectively permanent. Adding partitions re-maps hash(key) % partitions, so a key that used to land on partition 2 starts landing on partition 5 — and its old events stay behind. Ordering breaks for exactly the keys you cared about. Size partitions up front.

Consumers, and the two ways to lose a message

consumer = AIOKafkaConsumer(
    "orders", group_id="billing",
    # commit deliberately
    enable_auto_commit=False,
    auto_offset_reset="earliest",
)
async for msg in consumer:
    await handle(msg.value)
    # after the work, not before
    await consumer.commit()

Commit after processing and a crash re-delivers — at-least-once, and your consumer must be idempotent. Commit before and a crash skips it — at-most-once, and the message is gone. enable_auto_commit=True commits on a timer whether or not your work succeeded, which is the default and quietly gives you neither guarantee. Full detail in Kafka delivery semantics — at-most-once, at-least-once, exactly-once.

A group can have at most one consumer per partition. Twelve partitions supports twelve working consumers; the thirteenth sits idle. Partition count is the ceiling on consumer parallelism, which is the other reason to size it deliberately.

Retention is time and size, not consumption

retention.ms=604800000        # 7 days
retention.bytes=-1            # unbounded, per partition
cleanup.policy=delete         # or: compact

Messages are dropped on age or size, regardless of whether anyone read them. That decouples producers from consumers — a consumer down for an hour catches up — and it means a consumer down for longer than the retention window loses data permanently.

cleanup.policy=compact is the other mode: keep the latest value per key forever, so the topic becomes a snapshot of current state rather than a history. That is what makes a topic usable as a changelog — see Kafka: Log Compaction, Rebalancing, and Advanced Topics.

KRaft replaced ZooKeeper

Kafka used to need a ZooKeeper ensemble for cluster metadata. KRaft moves that into Kafka itself using a Raft quorum of controller nodes: one system to run, faster failover, and far more partitions per cluster.

Removed in Kafka 4.0. The migration path existed in 3.x; by 4.x there is no ZooKeeper mode to fall back to.

Why it is fast

Worth being able to say, because it is not magic:

• Sequential disk I/O. Appending to a log is sequential, which is fast even on spinning disks and very fast on SSD.
• The page cache does the work. Kafka does not maintain its own cache; it writes to the OS page cache and lets the kernel flush.
• Zero-copy via sendfile, so data goes disk → socket without passing through user space.
• Batching and compression amortise per-message overhead, on both ends.

The consequence for tuning: Kafka wants RAM for page cache more than heap. A huge JVM heap on a Kafka broker is a misconfiguration.

When it is the wrong tool

• A work queue. Commands executed once by one worker with per-message retry and a DLQ are RabbitMQ’s job — Kafka vs RabbitMQ — choosing a message broker.
• Low volume. The operational cost is not repaid by a few thousand messages a day; SQS or Redis Streams are enough.
• Request/reply. Kafka has no correlation or reply-to primitive. That is gRPC or HTTP.

Related

• Kafka: Log Compaction, Rebalancing, and Advanced Topics — compaction and rebalancing
• Kafka — Schema Registry and Python Client Comparison — Avro, schemas, the Python clients
• Kafka delivery semantics — at-most-once, at-least-once, exactly-once — at-least-once, exactly-once, transactions
• Producer configuration and durability — acks, idempotence, durability
• Kafka vs RabbitMQ — choosing a message broker — queue against log