Backend / Message queues / Kafka / 04_delivery_semantics.md

Kafka delivery semantics

Updated 6 interview angles 7 min read source
On this page7
  1. The mechanism: semantics are crash windows
  2. Transactions, and where exactly-once stops
  3. The consumer side
  4. Where it breaks in production
  5. Choosing
  6. Related
  7. Interview angle

Kafka delivery semantics

Article.

“What delivery guarantee does Kafka give?” is a trap. Kafka gives you whichever you configure, producer and consumer contribute independently, and the honest answer names the crash window that each choice leaves open.

Almost every real system runs at-least-once with idempotent processing. Exactly-once is a real Kafka feature with a specific scope, and knowing where that scope ends is the actual question.

The mechanism: semantics are crash windows

Every guarantee is decided by what happens if the process dies at one particular instant. Name the instant and the semantics follow.

The consumer window

text 
at-most-once            at-least-once
                        
poll                    poll
commit offset           process  ← side effects happen
  X crash               X crash
process (never runs)    commit (never runs)
                        
-> message lost         -> message redelivered

That is the whole consumer story. Commit before processing loses; commit after processing duplicates. There is no third ordering, which is why at-least-once plus idempotent handlers is the standard answer rather than a compromise.

The producer window

text 
producer -> broker:  append succeeded
broker   -> producer:  ack LOST in the network
producer:  retry
broker:    appends the SAME message again

This is why at-least-once produces duplicates even when your code sends once — the duplicate is created by a retry of a write that already succeeded, and the producer has no way to know.

The idempotent producer closes exactly this window. Each producer gets a PID, each message a sequence number per partition, and the broker drops a re-send whose sequence it has already appended.

python 
producer = Producer({
    "bootstrap.servers": "broker:9092",
    "acks": "all",                # all in-sync replicas
    "enable.idempotence": True,   # Java client default since 3.0; librdkafka defaults to False
})

Its scope is narrow and worth stating precisely: broker-side retries within one producer session, per partition. Not two produce() calls from your code, not across a restart. It is not application-level deduplication.

acks Durability
0 fire and forget; loss on any hiccup
1 leader only; lost if the leader dies first
all all in-sync replicas; pair with min.insync.replicas=2

Transactions, and where exactly-once stops

Transactions make writes to several partitions and the offset commit atomic:

python 
producer = Producer({"transactional.id": "order-enricher-1"})
producer.init_transactions()

while True:
    msgs = consumer.consume(100, timeout=1.0)
    producer.begin_transaction()
    for m in msgs:
        producer.produce("enriched-orders", transform(m.value()))
    producer.send_offsets_to_transaction(          # offsets inside the txn
        consumer.position(consumer.assignment()),
        consumer.consumer_group_metadata())
    producer.commit_transaction()

Two mechanics carry it:

  • transactional.id must be stable across restarts. On restart the broker bumps the producer epoch and fences the zombie — an old instance still running cannot commit. That fencing is what makes the guarantee survive a crash rather than a clean shutdown.
  • Downstream consumers need isolation.level=read_committed, or they read aborted writes and the guarantee evaporates at the reader.

Where the guarantee stops

The scope is Kafka-to-Kafka. The unit made atomic is consume, transform, produce. The moment your consumer writes to Postgres, charges a card or sends an email, the transaction does not reach there:

text 
[topic] --txn--> [topic]        exactly-once holds
[topic] -------> [Postgres]     it does not

End to end you are back to idempotent writes, natural keys, upserts, or an idempotency key — see Idempotency keys.

“Exactly-once delivery” does not exist over arbitrary side effects. Exactly-once processing is engineered with deduplication, and saying either “yes, everywhere” or “no, it’s marketing” is wrong.

The consumer side

python 
consumer = Consumer({
    "group.id": "billing",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",   # only when no committed offset exists
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    handle(msg)                                 # side effects first
    consumer.commit(msg, asynchronous=False)    # then commit

enable.auto.commit=true commits on a timer from the polling thread, which means offsets can be committed for messages you have not finished — silent loss — or the timer can fire long after side effects, giving duplicates. You get vague semantics, which is the reason serious consumers commit manually.

auto.offset.reset applies only when the group has no committed offset, and it is a classic incident: a new group or expired offsets plus latest silently skips everything already in the topic.

Where it breaks in production

The rebalance storm

The failure that takes a consumer group down and looks like nothing is wrong.

A consumer must call poll() at least every max.poll.interval.ms — five minutes by default. If a handler is slow, the coordinator decides the consumer is dead and triggers a rebalance:

text 
consumer A: handling a batch, 6 minutes
coordinator: A is dead -> rebalance
consumer B:  takes A's partitions, reprocesses the batch
             also takes 6 minutes -> rebalance
             ...

The group spends its time rebalancing rather than consuming, lag grows without bound, and every partition’s work is reprocessed repeatedly. The symptom is lag climbing while consumers look healthy and CPU is idle.

Fixes, in order: process smaller batches (max.poll.records), move slow work off the poll loop, or raise max.poll.interval.ms deliberately — the last one is a decision about how long a stuck consumer stays undetected, not a free knob.

Committing an offset commits everything before it

Offsets are per partition, not per message. Committing offset 105 asserts everything up to 105 is done.

text 
in flight: 101 102 103 104 105
           ok  ok  FAIL ok  ok

commit(105) -> 103 is now permanently skipped

Any handler that processes out of order, or continues past a failure, must not commit past the failure. Either stop at the first failure, or route the failed message to a dead-letter topic before committing — never leave the gap silent.

One transactional.id shared by instances

Two instances with the same id fence each other continuously: each restart bumps the epoch and invalidates the other. It presents as constant ProducerFenced errors and no progress. The id must be stable and unique per logical producer.

Testing on the happy path

Duplicates appear on retries, rebalances and crashes. A test that produces a hundred messages and reads a hundred back proves nothing about your semantics. Test by killing the consumer mid-batch and asserting the sink is correct.

Choosing

Building Configure
Metrics, loss tolerable acks=1, auto-commit
Business events acks=all, idempotent producer, manual commit
Kafka to Kafka transactions plus read_committed
Kafka to a database at-least-once plus dedupe at the sink

The trade not taken

Exactly-once is expensive and it is usually the wrong thing to reach for. Transactions add coordination, latency and operational complexity, and they only cover the Kafka-to-Kafka case that many pipelines are not.

At-least-once with an idempotent sink is simpler, faster, and covers everything — including the external systems transactions cannot reach. If a write is an upsert on a natural key, duplicates are free and you have bought nothing by preventing them.

The question to ask before configuring transactions is what a duplicate would actually cost. If the answer is “nothing, the upsert handles it”, stop there.

Interview angle 6

  • “What delivery semantics does Kafka give you?” — whichever you configure. The producer contributes with acks and idempotence, the consumer with when it commits relative to processing. The useful answer names the crash window each choice leaves open.
    • Follow-up: “So which do you actually run?” — at-least-once with an idempotent handler. Commit before processing loses messages, commit after duplicates them, and there is no third ordering.
  • “How does the idempotent producer work?” — a producer id plus a per-partition sequence number, so the broker drops a re-send it has already appended. It closes the window where a write succeeded and the ack was lost.
    • Follow-up: “What does it not cover?” — anything outside one producer session: two produce() calls from your code, or a restart. It is not application-level deduplication.
  • “Is Kafka exactly-once real?” — yes, Kafka-to-Kafka, via transactions, stable transactional.id, zombie fencing on epoch bump, and read_committed downstream. It stops at the first external system; writing to Postgres or calling an API is outside the transaction.
  • “Your consumer group’s lag is climbing but the consumers look healthy.” — a handler exceeding max.poll.interval.ms, so the coordinator declares it dead and rebalances, the next consumer inherits the same slow batch, and the group rebalances instead of consuming. Smaller batches, or move slow work off the poll loop.
  • “Message 103 failed but 104 and 105 succeeded. What do you commit?” — nothing past 103. An offset commit asserts everything before it is done, so committing 105 permanently skips 103. Stop at the failure or dead-letter it first.
  • “When would you not bother with exactly-once?” — when a duplicate costs nothing, which is most of the time. An upsert on a natural key makes duplicates harmless, and at-least-once with an idempotent sink is simpler and covers the external systems transactions cannot.
Previous
Kafka — Schema Registry and Python Client Comparison