What is Celery

Article.

A distributed task queue: your web process hands work to a broker, workers pull it off and run it, and the request returns without waiting. The reason it exists is that a request should not spend thirty seconds generating a PDF.

A task occasionally runs twice, and another sometimes never runs. List every cause. That question needs the delivery mechanics below, measured on Celery 5.6.3 (kombu 5.6.2) with a Redis 8.10 broker in Docker. Retries are How to Make Retry in Celery and idempotency is Consumer idempotency.

The three moving parts

producer ──▶ broker ──▶ worker ──▶ result backend
(your app)   (Rabbit/    (pulls,    (optional:
              Redis)      runs)      stores return values)

The broker is required; the result backend is not — and that distinction is the first thing to get right. If nothing reads the return value, configuring a backend just writes results nobody collects.

app = Celery("tasks", broker="amqp://...", backend=None)

@app.task
def send_invoice(order_id: int) -> None:
    ...

send_invoice.delay(order_id)      # returns immediately

Broker choice is a durability decision

RabbitMQ is the recommended broker and Redis is the pragmatic one. The honest framing: if losing a queued task on a Redis restart is acceptable, Redis saves you a component; if it is not, that is what RabbitMQ is for (quorum queues since Celery 5.5). Kafka is listed only as an experimental transport, without events or remote control, which is worth knowing before someone proposes it.

Everything that matters is the argument, not the task

The single most common Celery bug:

# Wrong: the ORM object is pickled, and by the time the worker
# runs it the row has changed — or the object will not serialise.
send_invoice.delay(order)

# Right: pass the id, re-fetch inside the task.
send_invoice.delay(order.id)

Arguments cross a process boundary and are serialised, so pass primitives and re-read state inside the task. The default serialiser is JSON, which enforces this by simply failing on anything else — and that is a feature. pickle as a serialiser accepts arbitrary objects and turns your broker into a remote code execution path if anyone can write to it.

Gotcha: the other half of the same bug is dispatching inside a database transaction. send_invoice.delay(order.id) fires immediately, and if the transaction then rolls back — or has simply not committed yet — the worker looks up an order that does not exist. Dispatch after commit, with transaction.on_commit(...) in Django or an outbox row otherwise.

Routing: queues, not priorities

app.conf.task_routes = {
    "tasks.send_invoice": {"queue": "email"},
    "tasks.render_video": {"queue": "heavy"},
}

celery -A tasks worker -Q email -c 8      # many, cheap, IO-bound
celery -A tasks worker -Q heavy -c 2      # few, expensive, CPU-bound

Separate queues with separate worker pools is how you stop a slow task starving a fast one. It is more reliable than broker priorities, which behave differently per broker and do nothing once a worker has already prefetched the work.

Prefetch is the setting people miss. A worker grabs prefetch_multiplier × concurrency messages up front, so a worker with long tasks can hold a queue of work it will not get to for an hour while other workers idle:

app.conf.worker_prefetch_multiplier = 1   # for long tasks
app.conf.task_acks_late = True            # ack after success, not on receipt

Measured with -c 2 and the default multiplier of 4, and 20 slow tasks queued: 2 active and 6 reserved, prefetched and unacknowledged in that worker, unavailable to any other worker that starts up.

When a task is acknowledged

With the default, a task is acknowledged when the worker receives it, so a crash mid-task loses it. task_acks_late = True acknowledges after the task finishes, which is the durability setting people reach for, but “finishes” includes failing:

A failure is acknowledged because task_acks_on_failure_or_timeout defaults to true; otherwise a task that always fails would loop forever.

A child killed by the OOM killer is acknowledged too, unless task_reject_on_worker_lost=True, which brings the redelivery loop back for tasks that kill their worker. To send a failed task to a dead-letter queue instead, raise Reject(exc, requeue=False) with a RabbitMQ dead-letter exchange (Dead-Letter Queue (DLQ) for Celery).

The Redis visibility timeout

Redis has no acknowledgements of its own, so the Redis transport keeps unacknowledged messages aside and restores them after visibility_timeout (3,600 s by default). With the timeout lowered to 10 s and a 30-second task:

two workers running, task runs 30 s                 ran 1 time
second worker started 15 s after the task began     ran 2 times

kombu restores expired messages when a worker starts, so the duplicate appears around deploys, restarts and autoscaling, which is why it looks random. Any task, countdown or ETA longer than the timeout is exposed. Set broker_transport_options = {"visibility_timeout": ...} above the longest task and the longest countdown, and keep long ETAs out of Celery (a scheduler or database row is the better place).

Every reason a task runs twice, or never

Runs twice: the visibility timeout passed and a worker started; the connection dropped with acks_late; a retry after a partial side effect; a Beat double-fire from two Beat processes; the client retried delay().

Never runs: acknowledged early and the worker crashed; the child was killed and acknowledged by default; dispatched inside a transaction that rolled back; expires passed before a worker got it; routed to a queue no worker consumes.

Both columns argue for the same design: tasks must be idempotent (Consumer idempotency), and dispatch must happen after the database commit.

Pools: pick one deliberately

prefork forks processes, so -c 8 is eight processes and eight sets of database connections — the same multiplication as Deployment. For a task that spends its life waiting on HTTP, gevent with a concurrency of hundreds costs a fraction of the memory.

Runtime guards

app.conf.update(
    task_soft_time_limit=60,          # raises SoftTimeLimitExceeded in the task
    task_time_limit=90,               # kills and replaces the child
    worker_max_tasks_per_child=1000,  # recycle against leaks
    worker_max_memory_per_child=500_000,   # in KB: recycle above ~500 MB
)

The soft limit lets a task clean up; the hard limit protects the worker. Recycling children contains slow leaks in native code (Python memory in production). On Kubernetes, terminationGracePeriodSeconds must exceed the longest task, or every deploy kills tasks mid-run (and, with acks_late, redelivers them).

Never wait for a task inside a task

child.delay().get() inside a task:
RuntimeError: Never call result.get() within a task!

Celery refuses by default: a task blocking on another can deadlock when every worker slot is held by a parent waiting for a child that has nowhere to run. Use a chain or a chord (Celery Canvas: Chord, Group, Chain — and Their Pitfalls).

Where it breaks in production

Duplicates around every deploy

A 2-hour report task on a Redis broker with the default 1-hour visibility timeout: every deploy after the first hour starts workers that restore the message, and the report is generated and emailed twice. Raise the timeout, or split the task.

Idle workers beside a backlog

One worker holds 32 prefetched messages while new workers have nothing. worker_prefetch_multiplier = 1 (and -O fair) for long tasks; Celery 5.6 adds worker_disable_prefetch.

Monitoring queue depth instead of age

Depth says how much is waiting, not how long. Alert on the age of the oldest message (or the time from task_sent to task_started events), export it to Prometheus, and trace across the broker with OpenTelemetry’s Celery instrumentation. celery inspect active and reserved show what each worker holds right now.

When Celery is the wrong answer

• A single background call after a request — FastAPI BackgroundTasks or a thread, with no broker to run: BackgroundTasks vs Celery — Decision Matrix.
• A workflow that must survive days, restarts and deploys — that is Temporal, which is durable execution rather than a task queue: Temporal vs Celery vs Other Orchestrators.
• An event other services should react to — that is a broker topic, not a task: Kafka vs RabbitMQ — choosing a message broker.

Related

• How to Make Retry in Celery — backoff, jitter, max retries
• Consumer idempotency — what at-least-once demands
• Dead-Letter Queue (DLQ) for Celery — where poison tasks go
• Celery Beat: HA, Scheduling, and Lock Contention — the scheduler is a singleton
• Celery Canvas: Chord, Group, Chain — and Their Pitfalls — chains, groups and chords