Kafka vs RabbitMQ — choosing a message broker

The most common messaging interview question. The wrong answer is “Kafka is faster.” The right answer starts from the model: RabbitMQ is a smart broker with dumb consumers (broker routes, tracks, and deletes messages); Kafka is a dumb broker with smart consumers (broker is an append-only log, consumers track their own position).

Deep dives: What is RabbitMQ, What is Kafka.

The core difference: queue vs log

RabbitMQ (queue):                    Kafka (log):
producer → exchange → queue          producer → topic partition
consumer ACKs → message DELETED      consumer reads at offset → log UNCHANGED
                                     another consumer group re-reads from 0

• RabbitMQ: a message is consumed destructively. Once acked, it’s gone. Competing consumers on one queue split the work.
• Kafka: messages are retained for a configured time/size regardless of consumption. Consumers are just cursors (offsets) over the log. New consumers can replay history.

Everything else follows from this.

Side by side

When RabbitMQ fits

• Task distribution / background jobs — work queues with fair dispatch, retries, DLX. This is why Celery uses it (What is Celery).
• Complex routing — route by routing-key patterns or headers without consumer-side filtering (RabbitMQ exchanges and routing).
• Per-message control — priorities, per-message TTL, delayed retries, dead-lettering (RabbitMQ Exchanges, Queues, DLX, and Quorum Queues).
• RPC over messaging — request/reply with reply_to + correlation_id.
• Low-volume, low-latency command passing between services where each message is an instruction to do work once.

When Kafka fits

• Event streaming / EDA backbone — services publish facts, many independent consumers react (Event-Driven Architecture).
• Replay & audit — rebuild a read model, backfill a new service, reprocess after a bug fix. The killer feature queues can’t offer.
• High throughput — clickstreams, metrics, logs, CDC feeds.
• Stream processing — windowing, joins, aggregations over the log (Stream Processing — Windowing, Watermarks, State).
• Event sourcing / outbox delivery — the log is the source of truth (Transactional Outbox Pattern).

Rule of thumb: commands → RabbitMQ, events → Kafka. A command (“charge this card”) should be executed once by one worker and disappear. An event (“order placed”) is a fact that many consumers, present and future, may care about.

The difference is visible in four lines of consumer code. RabbitMQ — you tell the broker the message is done, and it stops existing:

async for message in queue:
    async with message.process():   # ack on success
        await handle(message.body)  # requeue on exception

Kafka — you tell yourself where you got to, and the message stays:

async for msg in consumer:
    await handle(msg.value)
    # moves *my* offset only
    await consumer.commit()

commit() advances one consumer group’s cursor. Every other group, including one created next year, still sees the message. That is the replay property, and it is not a feature RabbitMQ is missing — it is a different data structure.

The rest of the field

Note Celery’s broker support: RabbitMQ and Redis are first-class; Kafka is not supported. If the team is Celery-based, “just use Kafka for tasks” is not a drop-in swap.

Using both is normal

A common production shape:

• RabbitMQ (or SQS) for work queues: emails, PDF generation, payment execution.
• Kafka for the event backbone: order/user/payment events feeding search indexing, analytics, notifications, and the data warehouse.

The saga/outbox machinery (Data Consistency Across Services) works over either; choose per stream of data, not per company.

Common interview confusions

• “Kafka is a message queue.” A consumer group makes it behave queue-like (each message processed by one member), but there’s no per-message ack/delete, no broker-side retry, no DLQ primitive — you build those with retry topics and consumer logic (Kafka delivery semantics — at-most-once, at-least-once, exactly-once).
• “RabbitMQ doesn’t scale.” It scales to very high volumes with clustering and quorum queues; it just doesn’t scale the same way (no partition-parallel consumption model, ordering vs parallelism trade-offs hit earlier).
• “Kafka guarantees ordering.” Only per partition. Order across a topic requires one partition (killing parallelism) or a partition key that groups what must be ordered (e.g., order_id).
• “Exactly-once means I can forget idempotency.” No — see Kafka delivery semantics — at-most-once, at-least-once, exactly-once; end-to-end you still design idempotent consumers (Celery Task Idempotency).