Backend / Caching / Redis / 08_job_queues_bullmq.md

Redis job queues and BullMQ

Updated 4 interview angles 5 min read source
On this page8
  1. The job lifecycle
  2. Stalled jobs are the mechanism to understand
  3. Retries, backoff and the DLQ you have to build
  4. Backpressure
  5. The Redis settings that lose your jobs
  6. Where it breaks in production
  7. Related
  8. Interview angle

Redis job queues and BullMQ

BullMQ is the Redis-backed job queue behind most Node workloads, n8n’s queue mode included. Celery’s concepts transfer — see Consumer idempotency — but the failure modes are different enough that naming them specifically is the signal.

The job lifecycle

text 
add() ─> waiting ──> active ──> completed
             ▲          │
          delayed       ├──> failed  (attempts exhausted)
             ▲          │
             └──────────┘  retry with backoff
                        │
                        └──> stalled ──> waiting  (lock not renewed)

Jobs live in Redis lists and sorted sets, one key group per queue. Adding is a Lua script so state transitions are atomic — which is why BullMQ is safe under concurrent workers and a naive LPUSH/BRPOP pair is not.

Stalled jobs are the mechanism to understand

A worker holds a lock on an active job and renews it on an interval. If the process is killed, blocked on CPU, or paused long enough that the lock expires, the job is stalled: another worker picks it up and runs it again.

Two consequences that get asked about:

  • Delivery is at-least-once. A long synchronous step that blocks the event loop past lockDuration gets the job re-run while the first copy is still going. Handlers must be idempotent; the queue will not do it for you.
  • maxStalledCount bounds the loop. After that many stalls, the job is failed rather than retried forever. Without it a job that reliably kills its worker cycles indefinitely.

The fix for spurious stalls is usually not a longer lock. It is finding the blocking call — a 4MB JSON parse, a sync crypto call — and moving it off the loop, the same diagnosis as run_in_executor and asyncio.to_thread.

Retries, backoff and the DLQ you have to build

javascript 
await queue.add("process", payload, {
  attempts: 5,
  backoff: { type: "exponential", delay: 2000 },  // 2s, 4s, 8s…
  removeOnComplete: { age: 3600, count: 1000 },   // bound Redis memory
  removeOnFail: { age: 86400 },                   // keep failures a day
});

BullMQ has no dead-letter queue primitive. Exhausted jobs land in the failed set and stay there — which is a dead-letter store, not a workflow. Build the missing half: a listener that moves permanently-failed jobs onto a dedicated queue with the original payload and the error, so failures are inspectable and replayable without hand-editing Redis.

javascript 
new Worker("process", handler, { concurrency: 10 })
  .on("failed", async (job, err) => {
    if (job.attemptsMade >= job.opts.attempts) {
      await dlq.add("dead", { original: job.data, error: err.message,
                              queue: job.queueName, jobId: job.id });
    }
  });

Distinguish retryable from terminal in the handler. A 502 deserves five attempts; a 400 from a malformed payload deserves none, and retrying it five times with exponential backoff wastes 30 seconds to reach the same answer.

Backpressure

Redis will happily accept a million jobs, and then you have a million jobs.

  • concurrency per worker caps in-flight work per process. Total parallelism is workers × concurrency, and that product is what your database and downstream APIs actually see — it is the number to size against, not the worker count.
  • Rate limiting (limiter: { max, duration }) throttles a queue against a third-party API’s limits, which is a different constraint from concurrency.
  • Bound the producer. If nothing checks queue depth before adding, a burst becomes unbounded memory. Check getWaitingCount() and shed or defer.
  • Alert on depth and age. Depth alone is ambiguous — 10,000 jobs draining fast is fine. The oldest waiting job’s age is the number that means “we are behind”.

The Redis settings that lose your jobs

Two configuration facts matter more than anything above, because both silently destroy work:

  • maxmemory-policy must be noeviction. Any allkeys-* policy lets Redis evict queue keys under memory pressure. Jobs vanish with no error anywhere — the queue simply gets shorter. This is the single most damaging misconfiguration for a Redis-backed queue, and shared or default-tuned Redis instances often ship with an eviction policy set for caching.
  • Persistence decides restart behaviour. RDB snapshots lose everything since the last save; AOF with appendfsync everysec loses about a second. If the queue is the system of record for in-flight work, you need AOF — see Persistence, eviction and memory.

The related sizing point: completed jobs are retained unless you say otherwise, so removeOnComplete is not an optimisation, it is what stops Redis memory growing without bound until eviction or OOM.

Where it breaks in production

  • One Redis for cache and queue. The cache’s eviction policy evicts the queue. Separate instances, or at minimum separate databases with the policy set for the queue’s needs.
  • Idempotency assumed from attempts: 1. Stalls re-run jobs regardless of the retry setting.
  • Unbounded removeOnComplete. Memory grows until the eviction policy saves you badly, or Redis OOMs.
  • Concurrency tuned per worker, ignoring the product. Eight workers at concurrency 10 is 80 concurrent database connections’ worth of demand.
  • Delayed jobs used as a scheduler. Millions of delayed jobs make the sorted set the bottleneck; use a real scheduler for recurring work.

Interview angle 4

  • “What is a stalled job?” - a worker held a lock on an active job and failed to renew it, so another worker takes it. That makes delivery at-least-once regardless of your retry setting, and it usually means something blocked the event loop rather than that the lock was too short.
  • “How do you build a DLQ in BullMQ?” - you build it. Exhausted jobs land in the failed set, which is storage, not a workflow. Add a failed listener that republishes to a dedicated queue with the payload and error so failures are inspectable and replayable.
  • “Jobs are disappearing and nothing is logged.” - check maxmemory-policy. An allkeys-lru policy set for caching evicts queue keys under pressure, silently. It must be noeviction, and the queue should not share an instance with a cache.
  • “How do you size concurrency?” - by the product, workers × concurrency, against whatever is scarcest downstream — usually database connections. Then alert on the oldest waiting job’s age rather than raw queue depth, because depth alone does not tell you whether you are falling behind.
Previous
Redis from Python