Concurrency and backpressure
An agent run is slow, expensive, and rate-limited by someone else’s API. Those three facts mean you cannot simply scale workers until the queue drains — the constraint is upstream, and pushing harder makes it worse.
The bottleneck is the provider, not your workers
100 workers ──▶ provider TPM limit ──▶ 429s
▲ │
└────────── more retries ◀───────────┘Adding workers past the provider’s tokens-per-minute allowance converts throughput into retry traffic. The queue still grows, latency rises, and the error rate climbs — a classic congestion collapse with a modern cause.
The fix is a concurrency limit that reflects the real constraint, not the worker count:
sem = asyncio.Semaphore(MAX_CONCURRENT_RUNS)
async def run_agent(job: Job) -> Result:
async with sem:
return await agent.run(job)Size it from the provider’s limits and your average tokens per run, not from CPU cores. An agent worker is almost entirely waiting.
Gotcha: rate limits are on tokens as well as requests. A limiter that counts only requests will pass 100 small calls and then trip on one large one. Budget the tokens you are about to send, not the call.
Backpressure means refusing work
The queue is not infinite storage; it is a buffer with a policy. When depth exceeds what you can clear in a reasonable time, the honest responses are:
| Response | Fits |
|---|---|
| Reject, 429 | interactive callers |
| Shed low priority | mixed workloads |
| Scale out | provider headroom exists |
| Queue with an ETA | batch work |
What you must not do is accept everything and let latency grow unbounded. A queue with a two-hour wait behaves worse than a rejection, because the client has already given up and will retry — adding the work again.
Measure queue depth and age, and alert on age. Depth alone is ambiguous; a depth of 5,000 that clears in a minute is fine, and a depth of 50 that has not moved in ten minutes is an incident.
Fairness: the noisy-tenant problem
One customer submitting 10,000 documents will starve everyone else out of a single FIFO queue. This is the most common multi-tenant AI failure and it is invisible until it happens.
FIFO: [A A A A A A A A A B C]
B and C wait behind A
Per-tenant: A:[..] B:[..] C:[..]
consumers round-robinOptions, cheapest first:
- Per-tenant concurrency cap — one tenant may hold at most N slots. Simple, and usually enough.
- Queue per tenant, round-robin the consumers — real fairness, more moving parts.
- Weighted fair queueing — when tiers genuinely differ.
A per-tenant cap plus a global limit gets you most of the benefit for a day’s work, and “we cap per tenant so one customer can’t starve the rest” is a strong sentence in a design interview.
Cost is a rate limit too
Concurrency limits protect the provider’s API. Nothing there protects your budget — an agent loop with a bug can spend a month’s allowance in an hour.
Enforce a per-tenant and global token budget in the same place you enforce concurrency, and make exceeding it a refusal rather than an alert. An alert tells you about the money after it is gone.
The related failure: retries multiply cost. An agent that retries three times and runs ten steps costs thirty LLM calls for one logical unit of work. Cap the total spend for a run, not just its step count.
Timeouts have to compose
Every layer needs a deadline, and they have to nest:
run deadline 10 min
step deadline 2 min
tool timeout 30 s
HTTP timeout 10 sIf the inner timeout exceeds the outer one, the outer never fires cleanly and you get a run killed mid-step with no record of where it was. Deriving each layer’s deadline from the remaining budget — rather than hardcoding all four — is what makes this hold under load.
What to watch
| Signal | Says |
|---|---|
| Queue age p95 | are we keeping up |
| Concurrency use | is the limit binding |
| 429 rate | are we over the provider’s line |
| Cost per tenant | who is expensive |
| Step-limit rate | runs failing expensively |
Queue age rather than depth, and cost per tenant rather than total, are the two that catch problems while they are still cheap.
Related
Interview angle 5
- “How do you scale agent workers?” - you mostly don’t. The constraint is the provider’s tokens-per-minute, so past that point extra workers convert throughput into 429s and retry traffic. Size a concurrency limit from the provider’s limits and tokens per run, not from CPU cores.
- “What is backpressure here?” - refusing work rather than queueing it indefinitely. A two-hour queue is worse than a 429, because the caller has already given up and retried, adding the work again. Alert on queue age, not depth — depth alone doesn’t say whether you’re keeping up.
- “How do you stop one customer starving the others?” - a per-tenant concurrency cap under a global limit, or a queue per tenant with round-robin consumers if you need real fairness. A single FIFO queue plus one customer bulk-uploading is the classic multi-tenant AI outage.
- “How do you cap cost?” - a token budget enforced in the same place as the concurrency limit, as a refusal rather than an alert. And cap total spend per run, not just steps, because retries multiply: three retries over ten steps is thirty calls for one unit of work.
- “How do timeouts interact?” - they nest, and each layer’s deadline should derive from the remaining budget. An inner timeout longer than the outer means the outer never fires cleanly and you lose the record of where the run was.