ProcessPoolExecutor: what processes add

A process pool runs Python in separate interpreters, so pure-Python CPU work runs in parallel without the GIL. Everything else follows from “separate processes”: work and results cross a pipe as pickles, workers are started by a start method that differs per platform, and a dying worker breaks the whole pool.

The common concurrent.futures API (submit, map, as_completed, cancellation, timeouts, calling a pool from asyncio) is in concurrent.futures. This note is the process-specific part.

Start methods

• spawn re-imports the main module, so creating a pool at module level recurses; every script that uses a pool needs if __name__ == "__main__":.
• fork copies the parent’s threads’ locks in whatever state they were, which deadlocks children of multi-threaded programs; that is why 3.14 moved POSIX to forkserver. Code that relied on fork inheriting globals (a model loaded before the pool started) now pays to load it again per worker.
• Choose explicitly when it matters: ProcessPoolExecutor(mp_context=multiprocessing.get_context("spawn")).

Only module-level callables cross the pipe

The function is pickled by reference (module plus name), so it must be importable in the worker:

def square(x: int) -> int:            # module level: works
    return x * x

def main():
    def cube(x): return x ** 3        # local function
    with ProcessPoolExecutor() as pool:
        list(pool.map(square, range(10)))     # fine
        list(pool.map(cube, range(10)))       # PicklingError: Can't pickle local object

Lambdas, nested functions, bound methods of unpicklable objects, open files, sockets and database connections all fail the same way. Arguments and results are pickled by value: passing a 500 MB DataFrame to each task copies 500 MB per task.

chunksize is the first lever

map sends items to workers in chunks; the default chunk is one item. For small tasks the per-item pickling and pipe round trip dominates:

chunksize=1       4.90 s for 20,000 tiny items
chunksize=100     0.05 s for 20,000 tiny items
chunksize=1000    0.01 s for 20,000 tiny items

Measured on Python 3.14.5, Windows, four workers. The work was a squaring function, so this is pure overhead; a 500x difference from one argument. Pick a chunk large enough that each costs milliseconds, small enough that the last chunks still spread across workers. submit has no chunking: batch the work yourself.

Worker lifecycle

def init_worker(model_path: str) -> None:
    global MODEL
    MODEL = load_model(model_path)            # once per worker, not per task

with ProcessPoolExecutor(
    max_workers=os.process_cpu_count(),      # 3.13+: respects CPU affinity
    initializer=init_worker, initargs=("model.bin",),
    max_tasks_per_child=500,                  # 3.11+: recycle leaky workers
) as pool:
    results = list(pool.map(score, batches, chunksize=50))

• initializer loads expensive state once per worker; with spawn and forkserver, globals from the parent are not inherited.
• max_tasks_per_child replaces a worker after N tasks, which contains memory leaks in C extensions (it forces the spawn method on POSIX).
• Size by cores, not by I/O wait: about os.process_cpu_count() workers for CPU work. More processes than cores add memory and switching, not speed. For I/O-bound work use threads or asyncio, not processes.
• Windows caps max_workers at 61.

When a worker dies

If a worker is killed (out of memory, a segfault in a C extension, os._exit), the pool cannot tell which task did it. Every pending future fails with BrokenProcessPool, and the pool is unusable:

try:
    for result in pool.map(transform, items, chunksize=100):
        save(result)
except BrokenProcessPool:
    log.error("a worker died; pool unusable")   # recreate the pool, find the input

Timeouts do not stop a task

Timeouts do not help here or elsewhere: future.result(timeout=...) stops waiting, but the task keeps running in its worker, and leaving the with block waits for it. Python 3.14 added pool.terminate_workers() and pool.kill_workers() to stop workers outright.

Sharing data without copying

• multiprocessing.shared_memory.SharedMemory shares a block of bytes, for example the buffer behind a NumPy array; workers attach by name and view it without copying. Someone must unlink() it, or it outlives the program.
• Memory-mapped files (numpy.memmap, Parquet read lazily) let each worker read the slice it needs.
• Pass references, not data: a file path or an object key is a few bytes to pickle; the worker loads what it needs.

Threads, processes or interpreters

Python 3.14 added InterpreterPoolExecutor (PEP 734), which runs each worker in a subinterpreter of the same process, each with its own GIL: parallel like processes, cheaper to start, with the same pickling rules. On the free-threaded build, plain threads run Python in parallel without pickling at all.

The comparison, measured on the same CPU work, is in concurrent.futures; free threading in Free-threaded Python.

Related

• concurrent.futures - the shared executor API, cancellation, asyncio
• When to use processes - when processes are the right tool
• run_in_executor and asyncio.to_thread - offloading from a coroutine
• The GIL, and free-threaded Python - why threads do not parallelise pure-Python CPU work