Concurrency and parallelism

Two words used interchangeably in conversation and meaning different things in a design discussion. The distinction is worth being precise about, because almost every “we need concurrency” argument is really about which of the two the problem needs — and they have different costs.

Concurrency is dealing with many things at once. Parallelism is doing many things at once.

One core is enough for one of them

CONCURRENCY, one core   PARALLELISM, two cores

A ██░░░░██░░██      A ████████
B ░░██░░░░██░░      B ████████
    ↑ interleaved        ↑ genuinely simultaneous
      progress

Concurrency is a structuring property: the program is written as tasks that can make progress independently, and whether any two of them ever run at the same instant is a separate question. Parallelism is an execution property: two things are physically running at once, which requires two cores.

So concurrency without parallelism is normal and useful — one core interleaving a thousand waiting sockets. Parallelism without concurrency is also possible — a vectorised numeric kernel with no task structure at all.

Which one your problem needs

The question is what the program is spending time on.

That is the whole heuristic. Interleaving does not create compute: if the bottleneck is the CPU actually executing, giving it more tasks to switch between makes it slower, not faster.

The same work four ways

One pure-Python task takes 0.25 s here. Four of them, and ten one-second waits, timed on CPython 3.14.5 with and without the GIL (4 workers on a 12-thread machine):

Ten waits overlap perfectly on one thread: concurrency is all they need. Four CPU tasks under gather take four times one task, because the event loop interleaves nothing while a coroutine computes. Threads do the same on the GIL build and scale on the free-threaded one; processes scale on both but pay start-up and pickling, which is why they reached 0.62 s and not 0.25 s.

Where the I/O parallelism comes from

A single asyncio thread can saturate a network link even though only one piece of Python runs at a time. While it waits, the kernel, the network card, the disk’s command queues and the remote servers all work in parallel. The event loop’s job is to keep enough requests in flight that they never idle.

Which model implements which — asyncio, threads, processes, free-threaded builds — is the decision in CPU-bound versus I/O-bound, and the GIL’s part in it is The GIL, and free-threaded Python.

Cooperative and preemptive

Concurrency comes in two flavours, and the difference decides how a mistake behaves.

async def handler():
    data = await fetch(url)      # yields; other tasks run
    return heavy_parse(data)     # does NOT yield; everything waits

Under asyncio, a task keeps the thread until it hits an await. Nothing can take it away. That is cooperative scheduling, and it is why one slow synchronous call inside a coroutine stalls every other task in the process — the classic async production incident.

Threads are preemptive: the interpreter switches between them whether they cooperate or not, so a thread running a long computation still lets the others run. You pay for that with the possibility of a switch between any two bytecodes, which is where race conditions come from.

On the GIL build “any time” is narrower than it sounds: a thread hands over the GIL at a blocking call, or after another thread has waited the 5 ms switch interval, at the next safe point (a call or a backward jump). That is why races are rare there and frequent on the free-threaded build, where threads really run at once (The GIL, and free-threaded Python).

Neither is safer in general — they move the danger. Cooperative scheduling makes the switch points visible in the source, which is a real advantage when reasoning about shared state; it also makes one careless blocking call fatal rather than merely slow. See asyncio and threads.

Two laws for sizing it

Amdahl’s law bounds parallel speed-up by the part that stays serial: speedup = 1 / ((1 - p) + p / n) for a parallel fraction p on n cores. With 90% parallel work, 8 cores give 4.7x; with 95%, no number of cores gets past 20x. In a Python process pool the serial part is real: pickling inputs and results, merging them, starting workers.

Little’s law sizes concurrency: requests in flight = arrival rate x latency. 2,000 requests a second at 50 ms each means 100 concurrent requests, so a semaphore, thread pool or database pool smaller than that queues, and one much larger only adds contention. The same arithmetic sizes how many outbound calls a crawler or fan-out service keeps open.

The production signature

Adding concurrency to a CPU-bound path raises p99 latency without raising throughput: one core sits at 100% per process while the queue in front of it grows. When a service is in that state, more tasks or threads make it worse; more processes, a free-threaded build or less work per request make it better.

Related

• CPU-bound versus I/O-bound - which model for which workload
• The GIL, and free-threaded Python - why Python threads and CPU work disagree
• asyncio and threads - the two in detail