Celery Task Idempotency

At-least-once delivery from any broker means tasks can run more than once. If your task is “send a $100 charge” and it runs twice, that’s a real customer with a real complaint. Idempotency is the only sane defense.

Why tasks run twice

Even without bugs:

• Worker crashes between executing the task and ack-ing the broker → broker redelivers.
• task.delay() retried by the caller after timeout, but the original succeeded.
• acks_late=True means ack only after success — also means crashes between work and ack = redelivery.
• Visibility timeout expired on SQS while the task was running → message redelivered to another worker.
• Manual retries via self.retry() that succeed after the original also did.

Treat at-least-once as a fact of life. Make tasks idempotent.

Three idempotency strategies

1. Use a natural unique identifier

If the operation already has a unique ID, persist by that ID.

@app.task(bind=True, autoretry_for=(IntegrityError,), retry_backoff=True)
def record_payment(self, payment_id, amount, user_id):
    try:
        with transaction.atomic():
            Payment.objects.create(id=payment_id, amount=amount, user_id=user_id)
            charge_card(amount, user_id, idempotency_key=payment_id)
    except IntegrityError:
        # Already inserted — task ran before. Done.
        return

The DB’s unique constraint on payment_id is the idempotency check. Second run gets IntegrityError; we treat it as “already done.”

This is the cleanest pattern when you have an externally-supplied ID.

2. Dedupe by message ID in a cache

When there’s no natural ID, dedupe by message-level identity.

import redis
r = redis.Redis()

@app.task(bind=True, acks_late=True)
def send_email(self, email_id, to, body):
    # Atomic: set if not exists; 24h TTL
    if not r.set(f"task:done:{email_id}", "1", nx=True, ex=86400):
        # Another run already started/finished
        return
    try:
        smtp.send(to, body)
    except Exception:
        # let retry try again
        r.delete(f"task:done:{email_id}")
        raise

Caveats:

• The TTL must outlive the worst-case retry window.
• “Set the flag before doing work” means a crash mid-work leaves the flag set and the work undone. Use sparingly; only when the work is itself idempotent or where loss is OK.

A safer variant: set the flag after success, but check before. Re-runs are caught only after the first success; less protection against concurrent retries.

3. Application-level idempotency keys

Same pattern as HTTP idempotency keys but at the queue layer. Generate a key client-side; store (key → result) in the DB.

class IdempotencyKey(models.Model):
    key = models.CharField(primary_key=True, max_length=64)
    result = models.JSONField(null=True)
    # in_progress, completed, failed
    status = models.CharField(max_length=20)
    created_at = models.DateTimeField(auto_now_add=True)

@app.task
def process_with_idempotency_key(key, payload):
    obj, created = IdempotencyKey.objects.get_or_create(key=key, defaults={"status": "in_progress"})
    if not created and obj.status == "completed":
        return obj.result
    if not created and obj.status == "in_progress":
        # Another worker is on it (or crashed mid-work).
        # Either wait/retry, or accept brief duplicate.
        raise Retry(countdown=5)

    try:
        result = do_work(payload)
        obj.result = result
        obj.status = "completed"
        obj.save()
        return result
    except Exception:
        obj.status = "failed"
        obj.save()
        raise

Production patterns like Stripe’s idempotency keys work this way.

Idempotency in distributed transactions

External side effects must support their own idempotency keys, or you can’t make the task idempotent end-to-end. Stripe, Twilio, modern payment APIs all accept an Idempotency-Key header.

@app.task
def charge_payment(payment_id, amount, user_id):
    stripe.Charge.create(
        amount=amount,
        customer=user_id,
        idempotency_key=payment_id,   # Stripe ignores duplicate requests with same key
    )

For an API that doesn’t, you must check “has this already happened?” before doing it. Often impossible without a side channel — design payment flows around APIs that support idempotency keys.

Idempotency + outbox + at-least-once = exactly-once-effect

The proper pattern for “do this once and only once across services”:

# Producer side (web layer)
@app.post("/orders")
async def create_order(order):
    with transaction.atomic():
        order_row = Order.objects.create(...)
        Outbox.objects.create(
            event_type="OrderPlaced",
            payload=json.dumps(order_row.to_dict()),
            idempotency_key=order_row.id,
        )

# Outbox relay (Celery beat / cron / separate worker)
@app.task
def publish_outbox():
    for row in Outbox.objects.filter(published_at__isnull=True)[:100]:
        kafka_publish(row.event_type, row.payload, key=row.idempotency_key)
        row.published_at = timezone.now()
        row.save()

# Consumer side (subscriber Celery worker)
@app.task
def handle_order_placed(payload, idempotency_key):
    if ProcessedEvent.objects.filter(event_id=idempotency_key).exists():
        return
    with transaction.atomic():
        ProcessedEvent.objects.create(event_id=idempotency_key)
        do_downstream_work(json.loads(payload))

Three layers of safety: producer writes outbox atomically with business data; relay publishes at-least-once with stable key; consumer deduplicates by that key. Net effect: every order is processed once and only once downstream.

Common mistakes

• “My task is idempotent because the function returns the same value.” Idempotency is about side effects, not return values. set_user_email is idempotent if calling it twice with the same email results in one email value. Sending an email twice is NOT idempotent — the second email is a side effect.
• Relying on acks_late = False. Default Celery acks as soon as the worker receives the task. A crash during execution loses the task. Setting acks_late = True is correct for important work but means redelivery on crash — handle accordingly.
• Setting the dedup flag before the work. A crash mid-work leaves the flag set, work undone. Set the flag after commit, or use a “claim then complete” two-phase state.
• Long-running tasks + short visibility timeout. SQS / RabbitMQ redelivers. Either lengthen the timeout or break the work into smaller idempotent units.
• Using PID + timestamp as idempotency key. Pid recycles, timestamps repeat under load. Use UUIDs or domain-meaningful IDs.