Backend / Message queues / Celery / 01_what_is_celery.md

What is Celery

Updated 7 interview angles 5 min read source
On this page8
  1. The three moving parts
  2. Broker choice is a durability decision
  3. Everything that matters is the argument, not the task
  4. Routing: queues, not priorities
  5. Pools: pick one deliberately
  6. When Celery is the wrong answer
  7. Related
  8. Interview angle

What is Celery

A distributed task queue: your web process hands work to a broker, workers pull it off and run it, and the request returns without waiting. The reason it exists is that a request should not spend thirty seconds generating a PDF.

Retries are How to Make Retry in Celery, idempotency is Celery Task Idempotency, and the failure cases are Celery — Common Interview Questions and Answers.

The three moving parts

text 
producer ──▶ broker ──▶ worker ──▶ result backend
(your app)   (Rabbit/    (pulls,    (optional:
              Redis)      runs)      stores return values)

The broker is required; the result backend is not — and that distinction is the first thing to get right. If nothing reads the return value, configuring a backend just writes results nobody collects.

python 
app = Celery("tasks", broker="amqp://...", backend=None)

@app.task
def send_invoice(order_id: int) -> None:
    ...

send_invoice.delay(order_id)      # returns immediately

Broker choice is a durability decision

RabbitMQ Redis SQS
Durability strong, persistent weaker, memory-first managed, durable
Ops burden a broker to run already there none
Priority queues yes emulated no
Celery support first-class first-class good, some gaps

RabbitMQ is the recommended broker and Redis is the pragmatic one. The honest framing: if losing a queued task on a Redis restart is acceptable, Redis saves you a component; if it is not, that is what RabbitMQ is for. Kafka is not a supported Celery broker, which is worth knowing before someone proposes it.

Everything that matters is the argument, not the task

The single most common Celery bug:

python 
# Wrong: the ORM object is pickled, and by the time the worker
# runs it the row has changed — or the object will not serialise.
send_invoice.delay(order)

# Right: pass the id, re-fetch inside the task.
send_invoice.delay(order.id)

Arguments cross a process boundary and are serialised, so pass primitives and re-read state inside the task. The default serialiser is JSON, which enforces this by simply failing on anything else — and that is a feature. pickle as a serialiser accepts arbitrary objects and turns your broker into a remote code execution path if anyone can write to it.

Gotcha: the other half of the same bug is dispatching inside a database transaction. send_invoice.delay(order.id) fires immediately, and if the transaction then rolls back — or has simply not committed yet — the worker looks up an order that does not exist. Dispatch after commit, with transaction.on_commit(...) in Django or an outbox row otherwise.

Routing: queues, not priorities

python 
app.conf.task_routes = {
    "tasks.send_invoice": {"queue": "email"},
    "tasks.render_video": {"queue": "heavy"},
}
bash 
celery -A tasks worker -Q email -c 8      # many, cheap, IO-bound
celery -A tasks worker -Q heavy -c 2      # few, expensive, CPU-bound

Separate queues with separate worker pools is how you stop a slow task starving a fast one. It is more reliable than broker priorities, which behave differently per broker and do nothing once a worker has already prefetched the work.

Prefetch is the setting people miss. A worker grabs prefetch_multiplier × concurrency messages up front, so a worker with long tasks can hold a queue of work it will not get to for an hour while other workers idle:

python 
app.conf.worker_prefetch_multiplier = 1   # for long tasks
app.conf.task_acks_late = True            # ack after success, not on receipt

acks_late is the durability pair: with the default, a task is acknowledged when the worker receives it, so a worker crash loses it. With acks_late it is acknowledged on completion, so a crash redelivers — at-least-once, which means Celery Task Idempotency is not optional.

Pools: pick one deliberately

Pool For
prefork (default) CPU-bound, isolation between tasks
gevent / eventlet many concurrent network calls
threads I/O with libraries that release the GIL
solo debugging only

prefork forks processes, so -c 8 is eight processes and eight sets of database connections — the same multiplication as Deployment. For a task that spends its life waiting on HTTP, gevent with a concurrency of hundreds costs a fraction of the memory.

When Celery is the wrong answer

Interview angle 7

  • “What is Celery and when do you use it?” - a distributed task queue: work is handed to a broker and run by separate worker processes, so the request returns immediately. Use it for anything slow, retryable or scheduled that a user should not wait for.
  • “Broker and result backend — what’s the difference?” - the broker carries the task and is required; the result backend stores return values and is optional. Configuring a backend nobody reads just writes results that are never collected.
  • “What do you pass to a task?” - primitives, usually an id, and re-fetch inside the task. Arguments are serialised across a process boundary, so an ORM object is either unserialisable or stale by the time it runs. JSON as the default serialiser enforces this; pickle accepts anything and makes the broker a code-execution path.
  • “Why might a worker fail to find a row that definitely exists?” - the task was dispatched inside a transaction that had not committed. Dispatch on commit — transaction.on_commit in Django, or an outbox row.
  • “How do you stop a slow task blocking fast ones?” - separate queues with separate worker pools, not broker priorities. And set worker_prefetch_multiplier = 1 for long tasks, or one worker hoards messages it will not reach for an hour while others idle.
  • “What does task_acks_late do?” - acknowledges after the task completes rather than when it is received, so a worker crash redelivers instead of losing the task. It buys durability and makes delivery at-least-once, so consumers must be idempotent.
  • “Which pool would you choose?” - prefork for CPU-bound work and isolation, gevent when tasks are mostly waiting on network calls, since hundreds of greenlets cost far less memory than hundreds of processes.