Synchronization primitives

Article.

Python ships the same names three times, in threading, asyncio and multiprocessing: Lock, Semaphore, Event, Condition, Barrier, Queue. They solve the same problem for different schedulers, and they do not mix.

What does a lock actually guarantee, and what did the GIL guarantee without one? That question decides when you need a primitive at all, and it changed with the free-threaded build. Outputs below are from CPython 3.14.5, with and without the GIL.

What the GIL did and did not make safe

counter = 0

def bump(n):
    global counter
    for _ in range(n):
        counter += 1                  # read, add, write

# 4 threads x 500,000 each

On the GIL build the unlocked count came out right, because threads switch only at certain points and rarely between this read and this write. It was never guaranteed. The free-threaded build lost two thirds of the updates.

Single operations on built-in objects stay safe on both builds: a list.append, a dict assignment, a deque.popleft are done under the GIL or, on 3.14t, under a per-object lock. Any sequence of operations (counter += 1, check-then-set on a dict, read-modify-write of an attribute) needs a lock, and always did (The GIL, and free-threaded Python).

Threading primitives

lock = threading.Lock()
with lock:                     # always as a context manager: released on error
    balance -= amount

Semantics that get asked about

• A Lock has no owner. Any thread may release it; that is legal and is how a lock can be used as a one-shot signal. An RLock has an owner: releasing it from another thread raises RuntimeError: cannot release un-acquired lock.
• No fairness. threading.Lock makes no promise about which waiter gets it next, so a thread can starve. asyncio.Lock wakes waiters in FIFO order.
• acquire(timeout=5) returns False instead of waiting forever. Used with a log line, it turns a silent deadlock into a reported one.
• Condition.wait_for(predicate) wraps the re-check loop: waits, wakes, re-tests the predicate, waits again. A bare wait() after an if is the bug, because another waiter may have consumed the state first.
• Cost. An uncontended acquire and release took about 116 ns from Python, call overhead included. Contention, not the lock, is what is expensive.

asyncio primitives

asyncio has Lock, Event, Condition, Semaphore, BoundedSemaphore, Barrier (3.11) and Queue. They suspend the coroutine instead of blocking the thread, and they are not thread-safe: they are for coroutines on one loop.

Coroutines only switch at an await, so code between awaits is atomic. An await in the middle of a read-modify-write is exactly as broken as a thread switch:

async def withdraw(amount):
    global balance
    current = balance
    await audit_log(amount)        # another withdraw runs here
    balance = current - amount     # overwrites its update

Five concurrent withdrawals of 10 from 100 left the balance at 90, not 50. async with lock: around the read, the await and the write fixes it.

Bound to a loop

Since 3.10 an asyncio primitive binds to the running loop the first time it has to make a waiter. Creating one at import time is fine, but reusing it across two asyncio.run() calls (common in tests) fails once there is contention: RuntimeError: <asyncio.locks.Lock ...> is bound to a different event loop. Create primitives inside the loop that uses them, or per test.

Never mix the two families

A threading.Lock held across an await blocks the whole event loop while another coroutine waits for it; an asyncio.Lock used from two threads is not protected at all. Bridge threads and loops with run_coroutine_threadsafe, call_soon_threadsafe or a queue (asyncio and threads).

Queues, and shutting them down

queue.Queue and asyncio.Queue are the preferred coordination tool: a bounded queue (maxsize=) gives backpressure as well as hand-off. Since 3.13 both have shutdown(), which wakes every blocked get() with QueueShutDown instead of the old pattern of one sentinel per consumer (Comparing asyncio.Queue and threading.Queue in Python).

async def worker(q):
    while True:
        try:
            item = await q.get()
        except asyncio.QueueShutDown:
            return                     # no sentinel needed
        await handle(item)
        q.task_done()

Across processes

multiprocessing.Lock, Semaphore, Event, Condition, Barrier and Queue mirror the threading API but are built on OS primitives (named semaphores, pipes) so they work across processes.

Each call on a Manager proxy is an IPC round trip, so a shared counter through a manager is orders of magnitude slower than a local one (When to use processes, ProcessPoolExecutor: what processes add).

Across machines, an in-process lock means nothing; distributed locks need leases and fencing tokens (Redlock and the Kleppmann Critique).

Where it breaks in production

Deadlock by lock order

Thread A takes lock 1 then lock 2; thread B takes 2 then 1. Both wait forever, CPU at zero, no error. Always acquire in one global order. To see it, py-spy dump --pid prints every thread’s stack, and faulthandler.dump_traceback_later(60) dumps them from inside after a timeout.

A lock held at fork

If another thread holds a lock when the process forks, the child inherits it locked, and nothing in the child will ever release it: a hang that appears only under load. 3.12+ warns when forking a process with threads, 3.14 defaults to forkserver on Linux, and os.register_at_fork(after_in_child=...) lets a library reset its locks in the child.

Races that 3.14t surfaces

Code that relied on the GIL making races rare starts losing updates on the free-threaded build, as the counter above did. The fix is the lock that was always missing; the way to find them is the test suite run under 3.14t with many threads.

Holding a lock across slow work

A lock held during network I/O serialises every caller behind the slowest request. Keep critical sections to the shared-state update; do the I/O outside.

The trade not taken

Locks protect shared mutable state; the alternative is not to share it. Message passing (a queue per worker, an actor that owns its state), immutable data, or a process per task removes the need for most locks, at the cost of copying data and designing around ownership. In Python services the usual answer is both: queues and pools for coordination, and a few small, well-named locks for the state that must be shared.

Related

• The GIL, and free-threaded Python - what the GIL did and did not protect
• Comparing asyncio.Queue and threading.Queue in Python - queues and backpressure
• asyncio and threads - choosing a model
• TaskGroup and structured concurrency - semaphores for bounded fan-out
• Redlock and the Kleppmann Critique - distributed locks