Consumer idempotency

Article.

Every broker delivers at least once, so every task will eventually run twice. If the task charges a card, that is a customer with a complaint.

This is the consumer side of the problem. The API side — a client retrying a POST, with a key it supplies — is Idempotency keys. Here there is no client to supply a key, and the duplicate is created by infrastructure doing its job.

The mechanism: the ack window

A worker takes a message, does the work, and acknowledges it. Those are separate events, and the gap between them is where duplicates come from.

ack early (acks_late=0)     ack late (acks_late=1)

receive                     receive
ack        <- broker done   work    <- side effects
work                        X crash
X crash                     ack (never sent)

-> work never happened      -> broker redelivers
-> silent loss              -> work happens twice

Same shape as every other delivery guarantee: ack before working loses, ack after working duplicates. Celery defaults to acks_late=False; setting it True is the right choice for anything that matters, and it buys duplicates deliberately.

The redelivery that overlaps

The detail that changes the design, and the one most people miss.

Brokers do not always wait for a worker to die. SQS and Celery’s Redis transport redeliver when a visibility timeout expires: SQS’s own default is 30 seconds, but Celery’s SQS transport sets 1,800 s and its Redis transport 3,600 s. RabbitMQ has no visibility timeout: unacknowledged deliveries return when the channel closes, including when consumer_timeout (30 minutes by default) closes it. The timeline below assumes a queue tuned down to 30 s.

t0    worker A receives, starts a 90-second task
t30   visibility timeout expires
t30   broker redelivers to worker B
t30+  A and B are now running the SAME task simultaneously

That is not “the task ran twice, one after the other”. It is two concurrent executions racing. A dedupe check that only asks “has this run before” does not help, because for both workers the honest answer is no.

Concurrency needs a lock, not a flag. Which is why the strategies below are not interchangeable.

Three strategies

1. A natural unique key — the best one

If the operation already has an identifier, let the database enforce uniqueness:

@app.task(bind=True, acks_late=True)
def record_payment(self, payment_id: str, amount: Decimal, user_id: int):
    try:
        with transaction.atomic():
            Payment.objects.create(id=payment_id, amount=amount, user_id=user_id)
            charge_card(amount, user_id, idempotency_key=payment_id)
    except IntegrityError:
        return          # already recorded; this is a duplicate delivery

The unique constraint is the check, and it is atomic, so it holds under concurrent redelivery. No extra store, no TTL to tune, no race.

Prefer this whenever an id exists, and reach for the others only when nothing natural is available.

2. Dedupe on message identity

def handle(message_id: str) -> None:
    if not r.set(f"done:{message_id}", "1", nx=True, ex=86400):
        return                      # someone already claimed it
    do_work()

This is the naive version, and it has a bug.

If the worker dies after the SET and before do_work(), the task is marked done and never runs. You have converted at-least-once into at-most-once, and the loss is silent.

Claim, then confirm:

key = f"job:{message_id}"

if r.set(key, "running", nx=True, ex=300):       # claim, short TTL
    do_work()
    r.set(key, "done", ex=86400)                 # confirm, long TTL
elif r.get(key) == b"done":
    return                                       # genuinely finished
else:
    raise Retry()      # someone is running it, or died: let the TTL expire

The short TTL on the claim is the recovery mechanism: a dead worker’s claim expires and the work becomes available again. Set it above the task’s p99 duration, or you will re-run work that was merely slow.

3. Make the operation naturally idempotent

The strongest option is to need none of this:

# Not idempotent - runs twice, adds twice.
user.credits += 10

# Idempotent - the second run is a no-op.
Credit.objects.update_or_create(
    grant_id=grant_id, defaults={"user": user, "amount": 10},
)

Upserts, absolute assignments (status = "paid" rather than incrementing a counter), and DELETE WHERE are all naturally safe. Reshaping the operation beats guarding it, because there is no key, no TTL and no race to get wrong.

Where it breaks in production

The partially-completed task

Idempotency at the task boundary says nothing about the middle:

@app.task(acks_late=True)
def fulfil(order_id):
    charge_card(order_id)         # succeeds
    reserve_stock(order_id)       # succeeds
    send_email(order_id)          # crashes
    # redelivered: charges again, reserves again, emails

The retry re-runs everything. Each step needs its own idempotency, or the task needs to record progress so a retry resumes rather than restarts. This is the same problem as a saga, and the same answers apply — see Sagas and compensation.

The poison message

Under Celery’s defaults a task that raises is acknowledged even with acks_late=True (task_acks_on_failure_or_timeout is true), so a plain failure does not loop (What is Celery).

The poison loop appears when that setting is turned off, when the task uses Reject(requeue=True), or when it kills its own worker (out of memory, a segfault in a C extension) and task_reject_on_worker_lost=True puts it back every time. Then it consumes a worker slot on every cycle and blocks nothing visibly.

Cap retries and route the failure somewhere a human looks:

@app.task(bind=True, acks_late=True, max_retries=5, retry_backoff=True,
          autoretry_for=(TransientError,))       # retry_backoff needs autoretry_for
def handle(self, payload):
    try:
        process(payload)
    except PermanentError:
        dead_letter.put(payload)     # do not retry what cannot succeed

Distinguish permanent from transient failures. Retrying a validation error five times with backoff is five guaranteed failures and a delay.

A visibility timeout below the task duration

If the timeout is 30 seconds and the task takes 90, the message is redelivered every 30 seconds while the first execution is still running. Three workers, then four, all doing the same work, and the queue appears to grow while throughput falls.

Set the timeout above the task’s p99, or extend it periodically from a long-running task if the broker supports it.

The dedupe store as a single point of failure

Strategy 2 puts Redis in the path of every task. If it is down, either every task fails or every task runs unguarded — and that decision should be deliberate, not whatever the exception handler happens to do.

Strategy 1 has no such dependency, which is the main argument for it.

The trade not taken

The instinct is to make delivery exactly-once. That is not available: every broker is at-least-once, and the ones claiming otherwise are doing deduplication somewhere and calling it delivery.

The available move is to make execution idempotent, and the cheapest version of that is not a framework or a Redis key — it is choosing operations that are safe to repeat. An upsert keyed on something the message already carries needs no guard, no store and no TTL.

Reach for a dedupe cache only when the operation genuinely cannot be reshaped — sending an email, calling a third-party API with no idempotency support. Those are real, and they are rarer than the amount of dedupe code in most codebases suggests.

Related

• Idempotency keys - the API-side pattern
• arq and dramatiq: the Celery alternatives - the same problem in other queues
• Kafka delivery semantics - the same window in Kafka