Scaling containers

Scaling out is the easy half. The hard half is that horizontal scaling assumes a stateless container, and most applications are not one until you make them so.

What has to move before you can scale

Any state living in the container is state one replica has and the others do not:

That last pair catch people. Five replicas each running the same cron means the nightly job runs five times — you need a leader election, an external scheduler, or an advisory lock. And writing logs to a file inside a container means losing them on restart; write to stdout and let the platform collect.

Sticky sessions are the tempting shortcut and a trap: they make scale-in lossy, defeat even load distribution, and turn a replica restart into a set of logged-out users.

Limits are not optional

services:
  api:
    image: myapp
    deploy:
      replicas: 4
      resources:
        limits: { cpus: "1.0", memory: 512M }
        reservations: { cpus: "0.25", memory: 256M }

Without a memory limit, one leaking container can consume the host and the kernel OOM-killer picks a victim that may be something else entirely. With one, only that container dies.

Gotcha: a runtime that does not read cgroup limits sizes its thread and connection pools from the host’s CPU count. A 1-CPU container on a 64-core host then creates 64 workers and thrashes. Modern Python, Java and Node read the cgroup, but any pool size you configure by hand must use the limit, not os.cpu_count().

Graceful shutdown, or you drop requests on every deploy

The runtime sends SIGTERM, waits a grace period, then SIGKILL. A process that ignores SIGTERM is killed mid-request.

# Exec form — the process is PID 1 and receives signals.
CMD ["python", "-m", "src.main"]

# Shell form — sh is PID 1 and does not forward them.
CMD python -m src.main

That distinction is the most common cause of “my container takes 10 seconds to stop”: it is not stopping, it is being killed after the grace period.

The application side:

1. On SIGTERM, stop accepting new connections.
2. Fail the readiness check so the load balancer stops sending traffic.
3. Finish in-flight requests, then exit.

Order matters — a process that exits immediately on SIGTERM still drops whatever was in flight, and one that keeps accepting until it exits drops whatever arrives after the balancer notices.

PID 1 and zombies

If your process spawns children, PID 1 must reap them or they accumulate as zombies. docker run --init inserts a minimal init that does this; so does tini. Only needed when you actually fork.

Health checks are how the platform knows

healthcheck:
  test: ["CMD", "curl", "-fsS", "localhost:8000/up"]
  interval: 10s
  timeout: 2s
  retries: 3
  start_period: 30s

start_period is the one people omit: without it a slow-starting app is marked unhealthy and restarted, forever.

Keep liveness and readiness distinct in concept even when the platform gives you one hook. Liveness asks “should I be restarted”; readiness asks “should I get traffic”. Putting a database check in liveness means a database blip restarts every replica — turning a degraded system into a dead one.

Where Compose stops

docker compose up --scale api=4 runs four containers on one machine, which is fine for local testing and for a single-host deployment behind a proxy.

It gives you no rolling deploy, no rescheduling when a host dies, no bin packing across machines. The moment you need any of those, the answer is an orchestrator — Kubernetes, ECS, Nomad — and saying so is the expected answer rather than defending Compose in production.

Scaling up first is also legitimate: a bigger machine is often cheaper than the operational cost of a cluster, right up until availability requires more than one host.

Related

• Docker network modes — how replicas are reached
• Kubernetes — where this goes next
• Architecture & design — making a service stateless