Python memory in production

Article.

The mechanisms are covered one by one in the notes before this: where objects live (Stack and heap in Python), when they are freed (Python memory model — allocation, refcounting, GC), and how the allocator hands out and keeps memory (The allocator: heap, arenas, pools, blocks). This note puts them together around the question a senior is actually asked:

A worker’s memory is 2 GB and growing. Walk me through it. Every number below was measured on Python 3.14.8 in a python:3.14-slim Linux container, because what the operating system reports depends on Linux and glibc behaviour that Windows and macOS do not share.

One object, end to end

x = {"id": 1}
  │  pymalloc block > 16 KiB pool > 1 MiB arena
  │  (over 512 bytes: glibc malloc instead)
  ▼
refcount > 0 ....... alive
refcount hits 0 .... freed at once, block to its pool
in a cycle ......... freed when the collector runs
  ▼
pool empty ......... reusable by any size class
arena empty ........ returned to the OS
malloc'd memory .... glibc decides; often kept

Two consequences drive everything below. Freed is not returned: memory goes back to the process’s allocators long before, if ever, it goes back to the OS. And Python can only report what Python allocated: native libraries allocate behind its back.

Which number are you looking at?

The gap between them is information:

100k x 1 kB bytes objects:   tracemalloc 99 MiB
then 200 MiB from C malloc:  tracemalloc 99 MiB, RSS 325 MiB

tracemalloc did not see the native 200 MiB at all. When RSS is far above what tracemalloc reports, the memory is in a C extension (NumPy, a database driver, an image library), in the allocator’s free lists, or in shared pages counted per process. For prefork servers the right per-worker number is USS or PSS, not RSS, which counts shared pages in every worker.

Four reasons the number is high

A leak: references kept by something

class Report:
    @functools.lru_cache(maxsize=None)     # the cache key includes self
    def total(self):
        return sum(self.rows)

Two thousand Report objects, each discarded after use: all 2,000 were still alive. lru_cache on a method keys on self, so the cache holds every instance forever. Two tracemalloc snapshots, diffed, point at the line that allocated what was kept:

tracemalloc.start(10)
before = tracemalloc.take_snapshot()
...                                       # run the suspect workload
after = tracemalloc.take_snapshot()
for stat in after.compare_to(before, "lineno")[:5]:
    print(stat)
# +23109 KiB  mem.py:66  +490000 blocks    <- self.rows = list(range(n))

The diff shows what was allocated; to find who holds it, take one object and walk gc.get_referrers() or use objgraph.show_backrefs(). The usual holders: module-level dicts and caches, lru_cache on methods, registries of callbacks, stored exceptions (their traceback keeps every frame’s locals), and closures (Memory profiling — tracemalloc, memray, and what holds memory).

A plateau: freed but not returned

300k x 600-byte objects (> 512 B, so glibc malloc):
  RSS 122 -> 203 MiB
  98% freed, 1 in 50 kept:   201 MiB
  malloc_trim(0):             41 MiB

Python freed the objects; glibc kept the memory in its heap, ready for the next allocation, because a few survivors pinned the top of it. malloc_trim(0) (via ctypes) asked glibc to return what it could. Small objects behave the same through pymalloc arenas, where one survivor keeps a 1 MiB arena (The allocator: heap, arenas, pools, blocks).

This is not a leak: the memory is reused by the next request. The process size is set by its peak, so the fix is lowering the peak (stream instead of loading everything), not chasing the plateau.

Allocator overhead: glibc arenas per thread

glibc gives threads their own malloc arenas to avoid lock contention, up to 8 per CPU core on 64-bit. Each keeps its own free memory:

16 threads, each churning 2 MB of 4 kB objects:
  default:             RSS 187 -> 209 MiB
  MALLOC_ARENA_MAX=2:  RSS 187 -> 187 MiB

Small here; in thread-heavy services on many-core machines it is often hundreds of megabytes, and MALLOC_ARENA_MAX=2 in the environment, or a different allocator (jemalloc via LD_PRELOAD), is the standard fix.

Copy-on-write lost after fork

gunicorn, uWSGI and Celery’s prefork pool load the application once and fork workers that share the parent’s pages until someone writes to them. In CPython, reading an object writes to it: the refcount changes. Child private memory after fork, with 300,000 small dicts inherited:

Reading copied 46 MiB per worker; with 16 workers that is 736 MiB of memory that was meant to be shared. gc.freeze() before forking moves everything to a permanent generation the collector never touches, so collections stop writing to inherited objects.

It cannot stop refcount writes on objects the worker actually uses; immortal objects (3.12+) only cover built-in singletons and interned strings (Python memory model — allocation, refcounting, GC).

Containers: the limit Python cannot see

docker run --memory=128m:   cgroup memory.max 128 MiB
allocated 50 MiB ... allocated 100 MiB ...
exit code: 137

No MemoryError, no traceback, no log line from Python: the kernel’s OOM killer sent SIGKILL when the cgroup hit its limit, and 137 is 128 + 9. Python does not read the cgroup limit, and os.cpu_count() reported all 12 host CPUs inside the container too. Consequences:

• Size workers from the limit: workers × (baseline + per-request peak) must fit, with headroom for the allocator plateau.
• Recycle workers where memory creeps: gunicorn --max-requests with --max-requests-jitter, Celery --max-tasks-per-child, max_tasks_per_child in ProcessPoolExecutor (3.11). This contains slow leaks and native fragmentation without finding them.
• Alert before the kill: export the worker’s RSS or the cgroup’s memory.current and alert at 80%; a 137 in the logs means you were too late.

Making objects smaller

100k instances, two attributes:
  regular class   128 B each
  __slots__        88 B each

__slots__ removed the per-instance dictionary. CPython has stored instance attributes inline since 3.11, so that dictionary was already smaller than it used to be, and the gain is about 30%, not the 50-60% older articles quote.

Bigger wins usually come from not materialising data at all: generators and iterator() instead of lists, array/NumPy instead of lists of numbers (a Python int is 28 bytes plus an 8-byte pointer; an int64 array slot is 8), and streaming responses instead of building them in memory.

The playbook

1. Read the shape over hours: steady growth, plateau, or growth after each fork.
2. Compare RSS with tracemalloc. Close means Python objects; far apart means native memory or allocator overhead.
3. For growth, diff snapshots with tracemalloc over a repeated workload, then find the holder with gc.get_referrers or objgraph.
4. For native memory, use memray (memray run --native), which follows allocations into C extensions, or py-spy dump for a live process.
5. For a plateau, lower the peak: stream, paginate, chunk. Try malloc_trim or MALLOC_ARENA_MAX to confirm it is the allocator.
6. For prefork, measure USS per worker; gc.freeze() before fork.
7. Contain what you cannot fix: worker recycling and a memory alert.

The trade not taken

Recycling workers every N requests hides almost any memory problem, leak or fragmentation, at the cost of a cold start per recycle and never learning the cause; it is the right first response and the wrong final one.

Replacing the allocator (jemalloc, mimalloc, which the free-threaded build already uses internally) often cuts fragmentation and arena overhead without code changes, but it is another moving part in the image. Rewriting hot data structures in NumPy or Rust gives the largest reductions and costs the most. Measure which kind of growth you have before choosing.

Related

• Python memory model — allocation, refcounting, GC - refcounting and the cycle collector
• The allocator: heap, arenas, pools, blocks - arenas, pools, blocks
• Stack and heap in Python - where objects live
• Memory profiling — tracemalloc, memray, and what holds memory - the tools
• When to use processes - process pools