Backend / Message queues / Kafka / 05_producer_configuration.md

Producer configuration and durability

Updated 6 interview angles 4 min read source
On this page8
  1. acks decides what you can lose
  2. Idempotence stops the retry duplicating
  3. Batching is a latency-for-throughput dial
  4. The setting people forget
  5. Reading it back safely
  6. What to actually say
  7. Related
  8. Interview angle

Producer configuration and durability

Kafka’s defaults favour throughput. Every durability guarantee you want is a setting you turn on, and the interview question is usually “what did you change and why”.

acks decides what you can lose

python 
producer = AIOKafkaProducer(
    bootstrap_servers=BROKERS,
    acks="all",              # not the default
    enable_idempotence=True,
    max_in_flight_requests_per_connection=5,
    compression_type="lz4",
    linger_ms=10,
)
acks Waits for Loses
0 nothing anything, silently
1 the leader the leader’s unreplicated writes
all the in-sync replicas only a total ISR failure

acks=all alone is still not enough. It waits for the in-sync replicas, and if the ISR has shrunk to one, “all” means one:

text 
min.insync.replicas=2       # broker/topic side

That pair — acks=all on the producer and min.insync.replicas=2 on the topic with replication.factor=3 — is the durable configuration. With min.insync=1 you have the illusion of durability; with min.insync=3 on a 3-replica topic, one broker restart stops writes entirely.

Idempotence stops the retry duplicating

A producer retry after a network timeout can write the message twice — the broker got it, the ack was lost. enable_idempotence=True gives each producer a PID and each message a sequence number, so the broker drops the duplicate.

It is nearly free and it is on by default in recent clients. Say it unprompted when asked about exactly-once, because idempotent producer plus transactional consumer is what “exactly-once” actually means in Kafka — Kafka delivery semantics — at-most-once, at-least-once, exactly-once.

Gotcha: without idempotence, retries > 0 and max_in_flight_requests_per_connection > 1 together can reorder messages — a retried batch lands after one sent later. Idempotence preserves order up to 5 in-flight, which is why the two settings appear together.

Batching is a latency-for-throughput dial

python 
linger_ms=10        # wait up to 10ms to fill a batch
batch_size=32768    # or until the batch is this big

linger_ms=0 (the default) sends as soon as possible: lowest latency, worst throughput, because every message pays its own round trip. Raising it to 5-20ms typically multiplies throughput for a latency cost nobody notices, since compression also works better on a full batch.

Compression is set on the producer and stored compressed — brokers do not recompress — so it saves network and disk. lz4 or zstd; gzip costs more CPU than it is worth here.

The setting people forget

python 
# how long send() blocks when the buffer is full
max_block_ms=5000
request_timeout_ms=30000
delivery_timeout_ms=120000

When the broker is slow, the producer’s buffer fills and send() blocks. With the default max_block_ms of 60 seconds, a Kafka problem becomes an application-wide stall — request handlers waiting to produce, thread pools exhausted, health checks timing out. Bound it, and decide what your code does when producing fails.

Reading it back safely

The consumer side of durability is one setting:

python 
consumer = AIOKafkaConsumer(
    "orders", group_id="billing",
    # skip aborted transactions
    isolation_level="read_committed",
    enable_auto_commit=False,
)

read_committed matters only if producers use transactions, and then it matters absolutely: the default read_uncommitted shows messages from transactions that were later aborted.

What to actually say

“Producers run acks=all with min.insync.replicas=2 on a replication-factor-3 topic, idempotence on, and linger_ms around 10 for batching. Consumers commit manually after processing, so we are at-least-once and the handlers are idempotent on the message key.”

That sentence covers durability, throughput and delivery semantics, and it is the shape of answer the question is looking for.

Interview angle 6

  • “What does acks=all guarantee?” - that the in-sync replicas have the write. On its own that can mean one replica, so it is only durable paired with min.insync.replicas=2 on a replication-factor-3 topic. acks=1 loses the leader’s unreplicated writes on failover; acks=0 loses silently.
  • “What does the idempotent producer solve?” - a retry after a lost ack writing the message twice. Each producer gets a PID and each message a sequence number, so the broker drops duplicates. It is also what makes retries safe without reordering.
  • “How can retries reorder messages?” - with idempotence off, retries > 0 and more than one in-flight request per connection let a retried batch land after a later one. Enabling idempotence preserves order up to five in flight, which is why those settings travel together.
  • “How do you increase producer throughput?” - linger_ms of 5-20ms so batches fill, a larger batch_size, and lz4 or zstd compression — which is stored compressed, so it saves disk as well as network. The cost is a few milliseconds of latency.
  • “What happens when Kafka is slow?” - the producer buffer fills and send() blocks for max_block_ms, which defaults to 60 seconds. That turns a broker problem into an application-wide stall, so bound it and decide what failing to produce means.
  • “When does isolation_level matter?” - when producers use transactions. The default read_uncommitted shows messages from transactions that were later aborted; read_committed skips them.
Previous
Kafka delivery semantics — at-most-once, at-least-once, exactly-once