Backend / Message queues / 00_broker_comparison.md

Kafka vs RabbitMQ — choosing a message broker

Updated 5 interview angles 7 min read source
On this page9
  1. The mechanism: queue vs log
  2. Side by side
  3. When RabbitMQ fits
  4. When Kafka fits
  5. Where it breaks in production
  6. The trade not taken: Kafka for everything
  7. The rest of the field
  8. Related
  9. Interview angle

Kafka vs RabbitMQ — choosing a message broker

Article.

The most common messaging interview question. The wrong answer is “Kafka is faster.” The right answer starts from the model: RabbitMQ is a smart broker with dumb consumers (the broker routes, tracks and deletes each message); Kafka is a dumb broker with smart consumers (the broker keeps an append-only log, consumers track their own position).

The 2026 version of the answer adds one twist: each now offers the other’s model as a feature. RabbitMQ has streams (an append-only log with offset replay, since 3.9), and Kafka 4.2 made share groups (“queues for Kafka”, KIP-932) production-ready, with per-record acknowledgement. The question is which model is native to the broker, not which one it can imitate.

Deep dives: What is RabbitMQ, What is Kafka.

The mechanism: queue vs log

text 
RabbitMQ queue
  producer → exchange → queue
  consumer acks → message DELETED

Kafka log
  producer → topic partition
  consumer reads at offset → log UNCHANGED
  another group re-reads from offset 0
  • A queue consumes destructively. Once acked, the message is gone. Competing consumers on one queue split the work, and the broker tracks every message’s state: delivered, unacked, redelivered.
  • A log retains. Messages stay for a configured time or size whatever consumers do. A consumer group is a cursor (an offset per partition), so a new group can replay history, and the broker tracks almost nothing.

The difference is visible in four lines of consumer code. RabbitMQ: you tell the broker the message is done, and it stops existing:

python 
async for message in queue:
    async with message.process():   # ack on success
        await handle(message.body)  # requeue on exception

Kafka: you tell yourself where you got to, and the message stays:

python 
async for msg in consumer:
    await handle(msg.value)
    await consumer.commit()         # moves *this group's* offset only

Every other group, including one created next year, still sees the message. That is not a feature RabbitMQ lacks; it is a different data structure.

Ordering follows from the structure

A queue with competing consumers and requeue cannot keep per-entity order. A log partitioned by key can, because one partition is read by one consumer in sequence. One account’s events, where the deposit fails once:

text 
queue, nack + requeue:    ['opened', 'withdraw 80', 'closed', 'deposit 100']
log, partitioned by key:  ['opened', 'deposit 100', 'withdraw 80', 'closed']

The requeued deposit lands at the back of the queue: the account overdraws, closes, then receives a deposit. On the log the consumer retries in place and the partition waits. That waiting is also the log’s weakness, below.

Side by side

Dimension RabbitMQ Kafka
Native model message queue (AMQP 0-9-1, AMQP 1.0) partitioned log
Consumption destructive, per-message ack offsets; share groups ack per record
Replay streams only (since 3.9) any group, from any retained offset
Routing exchanges: direct, topic, fanout, headers topic + partition key only
Ordering per queue, broken by competing consumers and requeue strict per partition
Per-message features TTL, priority, delayed retry, DLX none natively; retry topics
Retention until consumed (queues); time/size (streams) time/size, or compacted
Fan-out one queue per subscriber, bound to an exchange one more consumer group, free
Consumer scaling add consumers to a queue capped by partitions; share groups lift it
Throughput tens of thousands msg/s per node, typical millions msg/s per cluster
Latency sub-millisecond possible, push low but batch-oriented, pull
Python client aio-pika, pika confluent-kafka, aiokafka

When RabbitMQ fits

When Kafka fits

  • An event backbone: services publish facts and many independent consumers react (Event-Driven Architecture).
  • Replay and audit: rebuild a read model, backfill a new service, reprocess after a bug fix.
  • High throughput: clickstreams, metrics, logs, CDC feeds.
  • Stream processing: windows, joins and aggregations over the log (Stream Processing — Windowing, Watermarks, State).

Rule of thumb: commands to a queue, events to a log. A command (“charge this card”) should be executed once by one worker and disappear. An event (“order placed”) is a fact that many consumers, present and future, may care about.

Where it breaks in production

Kafka

  • A poison record blocks its partition. The consumer retries in place to keep order, so one bad record stalls everything behind it. Classic groups have no per-message dead letter; you build retry and dead-letter topics, or use a share group where order does not matter.
  • Partitions cap parallelism. Six partitions means the seventh consumer in a classic group sits idle, and adding partitions later changes which partition a key maps to, breaking per-key order across the change.
  • A hot key makes a hot partition. One tenant’s key on one partition caps that tenant at one consumer’s speed.
  • Retention shorter than an outage. A consumer down longer than retention comes back to find its offset deleted; the data is gone for it.
  • Rebalances pause consumption. Deploying a large group stops the world briefly on every rebalance; Kafka 4’s new consumer group protocol (KIP-848) makes this incremental.

RabbitMQ

  • Unbounded queue growth. Consumers fall behind, the queue grows until the memory or disk alarm trips, and then publishers are blocked. A slow consumer becomes an outage of the producers. Bound queues (x-max-length with an overflow policy) and alert on depth.
  • Redelivery loops. A message that always fails is requeued forever on a classic queue. RabbitMQ 4.0 gave quorum queues a default delivery limit of 20, after which the message is dead-lettered or dropped, so configure a dead-letter exchange or lose it.
  • Mirrored classic queues are gone. RabbitMQ 4.0 removed classic queue mirroring; old ha-mode policies silently do nothing, and those queues are single copies. Quorum queues are the replicated choice.

The trade not taken: Kafka for everything

The tempting platform decision is one broker, Kafka, for both events and work queues. Until share groups it meant rebuilding per-message retry, delayed delivery and dead-lettering in every consumer, and fighting the partition cap for task parallelism. Share groups narrow the gap for unordered work, but priorities, TTL, routing by headers and delayed retry still live naturally in RabbitMQ.

Celery makes it concrete: Kafka works only through kombu’s experimental confluentkafka transport, which Celery’s own docs say supports a single worker. A Celery shop that standardizes on Kafka has not swapped a broker, it has lost its task queue.

The usual production shape is both: a queue for work (emails, PDFs, payment execution) and a log for events (order and user events feeding search, analytics and the warehouse), chosen per stream of data, not per company.

The rest of the field

Broker One-liner Reach for it when
SQS / SNS managed queue / pub-sub on AWS zero ops on AWS; no replay, 14-day retention
Redis Streams a log inside Redis Redis is already there; lightweight streaming
NATS JetStream lightweight messaging with persistence low latency, simple ops (NATS and JetStream)
Pulsar log with tiered, segmented storage multi-tenancy, queue and stream in one
IBM MQ enterprise standard the bank chose it (IBM MQ — Overview for Python Backends)

Interview angle 5

  • “Kafka vs RabbitMQ: when would you pick each?” - lead with queue vs log: destructive consumption vs replay, broker routing vs partition keys, commands vs events, then one concrete example of each.
    • Follow-up: but RabbitMQ has streams and Kafka has queues now? Yes: streams since 3.9, share groups production-ready in Kafka 4.2. Each imitates the other’s model; you still pick the broker whose native model matches most of your traffic.
  • “How do you add a consumer of order events a year after launch?” - Kafka: a new consumer group replays from the earliest retained offset. A RabbitMQ queue cannot: the messages were deleted on ack, so you re-emit or backfill from the database (unless they went to a stream).
  • “Can you keep per-customer order with several consumers?” - on a log, partition by customer id: one partition, one reader, in order. On a queue with competing consumers, a requeue reorders: a simulated account received its deposit after it had closed.
    • Follow-up: and the cost of that ordering? A poison record blocks its partition, and parallelism is capped at the partition count.
  • “Your team uses Celery. Where does Kafka fit?” - next to it, as the event backbone. As a Celery broker it is experimental and limited to one worker.
  • “What happens when a RabbitMQ consumer falls far behind?” - the queue grows until the memory alarm blocks publishers, so a slow consumer takes down producers. Bound the queue and alert on depth.