Backend / Web frameworks / FastAPI / 04_async_and_performance.md

Async and performance

Updated 6 interview angles 5 min read source
On this page9
  1. Measure before you tune
  2. Concurrency inside one request
  3. The database is usually the ceiling
  4. Caching, in the order of cheapness
  5. Serialisation is a real cost at scale
  6. Streaming, when the response is large
  7. What actually moves the number
  8. Related
  9. Interview angle

Async and performance

The event-loop model is FastAPI and the event loop and the process model is Deployment. This is what to do when the service is slow — in the order that actually finds the problem.

Measure before you tune

Almost every “FastAPI is slow” report is one of four things, and they are distinguishable:

Symptom Usually
p99 rises on every endpoint at once something blocking the loop
One endpoint slow, low CPU waiting on a downstream or the database
One endpoint slow, high CPU serialisation, or real computation
Slow only under load pool exhaustion, or too few workers

The first row is the FastAPI-specific one and it is the first thing to rule out, because the fix is unrelated to everything else:

python 
# logs any callback that hogs the loop
asyncio.run(main(), debug=True)

Concurrency inside one request

The single biggest win when a handler calls several things:

python 
# Serial: 300ms. Nothing here needs to be.
prices = await prices_api.get(sku)
stock = await stock_api.get(sku)
reviews = await reviews_api.get(sku)

# Concurrent: ~100ms, and one failure cancels the rest.
async with asyncio.TaskGroup() as tg:
    p = tg.create_task(prices_api.get(sku))
    s = tg.create_task(stock_api.get(sku))
    r = tg.create_task(reviews_api.get(sku))

TaskGroup over gather because a failure cancels the siblings rather than leaving them running, and the errors arrive together as an ExceptionGroup. Use gather(..., return_exceptions=True) only when partial success is the product decision — as in Follow-Up Questions: Slow/Failing APIs, Auth, Retry, Tests.

The database is usually the ceiling

python 
engine = create_async_engine(
    url,
    pool_size=10,          # per worker process
    max_overflow=5,
    pool_pre_ping=True,    # cheap liveness check
    pool_recycle=1800,     # under a proxy that closes idle connections
)

pool_size is per worker, so four uvicorn workers at pool_size=10 is fifty connections including overflow. Postgres max_connections is typically 100, so two replicas of that service exhaust it — this is the most common production capacity surprise, and it is arithmetic rather than tuning.

Under load, symptoms of pool exhaustion look like slowness: requests queue waiting for a connection while the database itself is idle. pool_timeout turns that into an error you can see instead of latency you cannot explain.

And the N+1 problem does not care that you are async — see Loading Strategies and N+1.

Caching, in the order of cheapness

python 
# per process, config only
@lru_cache(maxsize=1)
def get_settings() -> Settings: ...
python 
# Shared, and the only kind that works across workers.
async def get_user(uid: int) -> User:
    if (hit := await redis.get(f"u:{uid}")):
        return User.model_validate_json(hit)
    user = await repo.get(uid)
    await redis.setex(f"u:{uid}", 300, user.model_dump_json())
    return user

lru_cache is per process, so with four workers you have four caches and no invalidation story. It is right for settings and computed constants and wrong for anything a user changes. Redis is the shared answer, with the stampede caveat in Cache Stampede and Mitigation Patterns.

The cheapest cache is the one at the edge: Cache-Control on a response that a CDN can hold means the request never reaches Python at all.

Serialisation is a real cost at scale

Returning ten thousand rows through a response_model validates ten thousand objects. Options, in order:

  1. Paginate. Almost always the right answer, and the one that also fixes the database side.
  2. ORJSONResponse — a faster JSON encoder, a one-line change:
python 
app = FastAPI(default_response_class=ORJSONResponse)
  1. Skip validation on a hot read by returning a Response directly with pre-serialised bytes. You lose the contract guarantee, so do it deliberately and only where profiling says it matters.

Streaming, when the response is large

python 
@app.get("/export")
async def export():
    async def rows():
        async for row in repo.stream():
            yield row.to_csv_line()
    return StreamingResponse(rows(), media_type="text/csv")

Time-to-first-byte drops and memory stays flat regardless of size. Two things break it: BaseHTTPMiddleware buffers the body (Middleware), and a proxy may buffer too — nginx needs proxy_buffering off for the client to see it stream.

What actually moves the number

In the order I would try them:

  1. Stop blocking the loop. Nothing else matters until this is true.
  2. Parallelise independent awaits in the handler.
  3. Fix the queries — N+1, missing index, then pool size.
  4. Cache, at the edge first, then Redis.
  5. Paginate, which fixes serialisation and the database together.
  6. Add workers or replicas, last, because it multiplies connection pools and memory rather than fixing anything.

Interview angle 6

  • “The API is slow. How do you find out why?” - split it first: p99 rising on every endpoint at once means something is blocking the event loop; one slow endpoint with low CPU means waiting on a dependency; with high CPU means serialisation or real work; slow only under load means pool exhaustion or too few workers.
  • “A handler makes three independent API calls. What do you do?” - run them in a TaskGroup so total latency is the slowest rather than the sum, and a failure cancels the siblings. gather(return_exceptions=True) only when partial success is a deliberate product decision.
  • “Why does adding workers sometimes make things worse?” - pool_size is per process, so four workers at ten connections is forty plus overflow. Two replicas exhaust a default Postgres max_connections, and the symptom is queued requests while the database sits idle.
  • “When is lru_cache the wrong cache?” - whenever the data changes or the value must be shared. It is per process, so four workers hold four copies with no invalidation. It is right for settings and computed constants only.
  • “How do you return a very large result?” - paginate. If you genuinely cannot, stream it with StreamingResponse so memory stays flat — but check that no BaseHTTPMiddleware and no proxy buffering is undoing it.
  • “What would you try last?” - more workers or replicas. It multiplies connection pools and memory without fixing the cause, so it belongs after unblocking the loop, parallelising, fixing queries, caching and paginating.
Previous
Dependencies and injection