n8n at scale: topology, connections and retention

n8n and low-code automation covers what n8n is and where the canvas/code boundary belongs. This is the operational half: the topology that survives load, the connection arithmetic that decides whether it stays up, and the three failure modes that account for most self-hosted incidents.

The scenario to have an answer for: queue mode, Postgres and Redis, a handful of workers, payloads in the megabytes, and it works fine until it doesn’t.

The topology is three roles, not two

Most setups separate main from workers and stop there. There is a third role, and it is usually the one missing when webhooks start timing out.

n8n                 # main: UI, cron triggers, queue orchestration
n8n webhook         # webhook processor: accepts and enqueues, runs nothing
n8n worker          # worker: pulls from Redis, executes

Why the webhook role matters. If main accepts webhooks, then main’s event loop is competing with UI requests and schedule evaluation. A burst of inbound calls makes the editor unresponsive and can make main miss cron windows. Split webhook processors out and put them behind the load balancer; they do almost no work per request — validate, write, enqueue — so they scale cheaply and absorb bursts.

Main does not scale horizontally on the community edition. Multi-main is a licensed feature, so on OSS a single main is a single point of failure for schedules and for the UI. Say this plainly in an interview; it is a real architectural constraint, not a detail.

Connection arithmetic is the thing that breaks first

Every n8n process opens its own Postgres pool. The total is not obvious, and exceeding max_connections produces the error people arrive with:

timeout exceeded when trying to connect

That message is the pool acquisition timing out, not the network. Two distinct causes, and they need opposite fixes:

• The pool is too small for the process. Requests queue for a free connection and time out waiting. Raise DB_POSTGRESDB_POOL_SIZE.
• Postgres is out of connections. The pool cannot open more because the server is at max_connections. Raising the pool size makes this worse.

Diagnose before touching either:

-- Where the connections actually are. If most are 'idle in transaction',
-- the problem is a leak, not capacity.
SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state ORDER BY count DESC;

SHOW max_connections;

The connection budget

The budget to compute out loud:

(main × pool) + (webhooks × pool) + (workers × pool) + admin headroom
   1 × 10    +      2 × 10       +      8 × 10       +      10        = 120

Eight workers at a pool of 10 is 80 connections from the workers alone. A managed Postgres on a small plan may cap well below that — check the plan’s limit before scaling workers, because adding workers to fix slowness is the move that causes the outage.

PgBouncer in transaction mode is the standard answer: hundreds of client connections multiplexed onto a small server pool. The caveat is the usual one — transaction mode breaks prepared statements, session state and advisory locks.

Test it before relying on it, because an ORM that pins prepared statements fails in ways that look intermittent. Modes and their costs are in Connection pooling.

Execution data is a disk-growth bomb

Every execution stores its full payload — inputs and outputs, per node. With 1–5MB payloads and high volume, the execution tables grow by gigabytes a day, and the table churn also generates dead tuples faster than a default autovacuum will reclaim them.

Decide three things explicitly:

EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=168          # hours of history to keep
EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none # the big one
EXECUTIONS_DATA_SAVE_ON_ERROR=all

Not saving successful executions is the highest-leverage setting, because successes are the overwhelming majority and nobody reads their payloads. Keep errors — those are what you debug and reprocess from.

Two failure modes that follow:

• Deletes do not return disk. Pruning marks tuples dead; autovacuum has to reclaim them, and on a high-churn table with default settings it falls behind. Tune autovacuum on the execution tables specifically — a lower autovacuum_vacuum_scale_factor so it triggers on a percentage that makes sense for millions of rows. Detail in VACUUM, autovacuum, table bloat, and transaction wraparound.
• Bulk pruning is itself an incident. A delete of ten million rows takes a long lock-free but WAL-heavy path and competes with live writes. Delete in batches with a sleep, or partition.

Partitioning by time is the durable fix. With execution tables partitioned by day or week, retention becomes DROP TABLE — instant, no vacuum, no bloat — instead of a delete that has to be cleaned up afterwards. It is more work to set up and it is the answer when pruning is already failing to keep up.

Backpressure and concurrency caps

Redis will absorb an unbounded backlog; that is the danger. The queue mechanics are in Redis job queues and BullMQ.

• N8N_CONCURRENCY_PRODUCTION_LIMIT caps concurrent executions per worker. Uncapped, one worker will happily start hundreds and exhaust memory on megabyte payloads.
• Total load is workers × concurrency, and that product is what hits Postgres and any downstream API. Size it against the scarcest resource.
• Memory is per execution, not per worker. A 5MB payload held through a 20-node workflow with intermediate results is not 5MB. This is why NODE_OPTIONS=--max-old-space-size and container limits need to agree.
• Token-heavy inference changes the shape. LLM steps are slow and mostly waiting, so a worker can hold many concurrently — but each holds its payload in memory the whole time. Concurrency is bounded by memory, not CPU.

Capacity planning for bursty load

The arithmetic to present:

peak executions/min ÷ 60          = executions/sec
× median execution seconds        = concurrent executions needed
÷ concurrency per worker          = workers

Then check that number against connections, memory per execution × concurrency, and Redis memory for the queued backlog at peak. Bursts are absorbed by the queue, so size workers for sustainable throughput and the queue for burst depth — and alert on the oldest waiting job’s age, which is the signal that sustainable throughput is no longer sustainable.

The platform constraints (Railway and similar PaaS)

• Ephemeral disk. Anything not in Postgres or object storage is gone on redeploy. Never SQLite; never local files between nodes.
• Managed Postgres connection caps are often the real ceiling, and lower than you would provision yourself. This drives the PgBouncer decision.
• No sidecars on some plans, so PgBouncer becomes its own service.
• Metrics are coarse. Platform CPU/memory graphs will not tell you queue age or pool saturation; export those yourself.

Related

• n8n and low-code automation - what belongs in the canvas at all.
• Redis job queues and BullMQ
  • stalled jobs, DLQ, eviction policy.
• VACUUM, autovacuum, table bloat, and transaction wraparound
  • bloat, autovacuum tuning, wraparound.
• Connection pooling
  • PgBouncer modes and their costs.
• Sharding and Partitioning
  • partitioning execution tables by time.
• Incident response, runbooks and postmortems
  • triage order and the postmortem.
• Load and Performance Testing
  • building the repeatable benchmark.