Backend / Message queues / 01_arq_and_dramatiq.md

arq and dramatiq: the Celery alternatives

Updated 6 interview angles 4 min read source
On this page7
  1. Why anyone leaves Celery
  2. arq: async-native, Redis only
  3. dramatiq: Celery’s shape, less of it
  4. Choosing between the three
  5. What actually matters, whichever you pick
  6. Related
  7. Interview angle

arq and dramatiq: the Celery alternatives

Job descriptions increasingly write “Celery, or arq/dramatiq”, which is a signal that the interviewer knows Celery is not the only answer and may ask why you would pick something else.

Why anyone leaves Celery

Celery is the default and it works, but it carries weight:

  • A large surface. Chords, chains, canvas, routing, rate limits and four broker backends, most of which a given team never uses.
  • Async is bolted on. Celery tasks are synchronous functions. In an async FastAPI codebase, every task that touches the database needs either a separate sync engine or an asyncio.run wrapper, so you maintain two database stacks.
  • Configuration depth. The defaults that bite - acks_late, prefetch, visibility timeout - are learned by outage.

The alternatives are smaller and pick a lane.

arq: async-native, Redis only

arq is written for asyncio. A task is a coroutine, and it shares your existing async session and client.

python 
async def send_invoice(ctx, invoice_id: str):
    async with Session() as s:      # the same engine
        inv = await s.get(Invoice, invoice_id)
        await mailer.send(inv)

class WorkerSettings:
    functions = [send_invoice]
    redis_settings = RedisSettings(host="cache")
    max_jobs = 20

ctx is a dict the worker builds at startup - the place to put a connection pool or an HTTP client so it is created once per worker, not once per job.

Enqueueing returns a job handle you can poll:

python 
pool = await create_pool(RedisSettings(host="cache"))
job = await pool.enqueue_job(
    "send_invoice", invoice_id, _defer_by=60
)

What you give up: Redis is the only broker, there is no result backend beyond Redis, and the ecosystem is small. What you gain: one event loop, one database engine, and a codebase you can read in an afternoon.

dramatiq: Celery’s shape, less of it

dramatiq is synchronous like Celery but deliberately smaller, and its defaults are the ones Celery makes you find.

python 
import dramatiq

@dramatiq.actor(max_retries=3, time_limit=30_000)
def resize_image(path: str):
    ...

Retries are on by default with exponential backoff. Time limits are enforced. It runs on RabbitMQ or Redis, and its middleware system is how you add behaviour rather than reading a settings table.

The trade is the same as arq’s: fewer features, fewer footguns, a smaller community.

Choosing between the three

Pick it when
Celery You need Beat, canvas, or the ecosystem
arq The codebase is async and Redis is present
dramatiq Sync code, but Celery feels heavy

The honest framing for an interview: this is rarely a technical decision that matters much. All three enqueue a message and run a function elsewhere. What matters is what you do about the properties every queue shares.

What actually matters, whichever you pick

An interviewer asking about task queues is usually probing these, and they are identical across the three.

Idempotency. Every one of these is at-least-once. A worker that dies after doing the work but before acknowledging will redo it. Design the task so a second run is harmless - key on a natural id, upsert rather than insert, check state before acting.

python 
async def charge(ctx, order_id: str, key: str):
    if await seen(key):        # replay guard
        return
    await gateway.charge(order_id, idem_key=key)
    await mark_seen(key)

Never pass objects, pass ids. The payload is serialised. Sending an ORM instance means the worker acts on a snapshot that may be stale by the time it runs, and it bloats the message.

Visibility timeout versus task duration. If the broker redelivers after 30 seconds and the task takes 60, it runs forever in a loop of duplicates.

A separate queue for slow work. One queue means a batch of slow jobs starves the fast ones. Route by expected duration, not by feature.

Poison messages. A task that fails deterministically and retries forever will saturate the workers. Cap retries and route the failure to a dead-letter queue you actually monitor.

Interview angle 6

  • “Why would you use arq instead of Celery?” - the codebase is async. Celery tasks are sync functions, so an async service ends up maintaining a second database engine just for workers. arq tasks are coroutines sharing the same session and pools. The cost is Redis-only and a small ecosystem.
  • “What does dramatiq give you?” - Celery’s model with a smaller surface and better defaults: retries with backoff and enforced time limits are on out of the box, and behaviour is added through middleware rather than a large settings table.
  • “When do you stay on Celery?” - when you need Beat for scheduling, canvas workflows like chords and chains, or the ecosystem’s integrations. Familiarity across a team is a legitimate reason too.
  • “What breaks in any of them?” - at-least-once delivery. A worker dying after the work but before the ack redoes it, so tasks must be idempotent: guard on a key, upsert rather than insert, check state first.
  • “Why pass an id rather than the object?” - the payload is serialised, so an object is a snapshot that may be stale when the task runs, and it inflates the message. Pass the id and re-read inside the task.
  • “A task retries forever. What is wrong?” - either a deterministic failure with no retry cap, so it never drains and starves the workers, or a task running longer than the visibility timeout, so the broker redelivers it while the first run is still going. Cap retries into a dead-letter queue, and keep task duration under the timeout.
Previous
Kafka vs RabbitMQ — choosing a message broker