Timeouts, retries and backoff

Article.

A retry is a bet: that the failure was transient, and that trying again costs less than giving up. Most of the time the bet is good, which is why retries are everywhere and mostly invisible.

The question worth being able to answer is when the bet is wrong — because the case where it is wrong is exactly the case where you are already having an outage, and a retry policy tuned for the good case will finish the job.

The mechanism: a retry is load

Every resilience discussion treats retries as a client-side concern. From the server’s side they are indistinguishable from traffic.

service degrades  ->  calls time out
                        ↓
                      clients retry
                        ↓
                      load increases
                        ↓
                      service degrades further
                        ↑___________________|

That loop is the whole story. Under normal failures — one dead node, a dropped packet, a restarting pod — the dependency has spare capacity, the retry lands on a healthy path, and the bet pays. Under saturation the dependency has no spare capacity, and the retry is the thing consuming it.

So the same policy is correct and catastrophic depending on a condition the client cannot see. That is why the interesting mechanisms below — budgets, breakers, deadline propagation — are all ways of inferring that condition.

Metastable failure

The loop has a property that makes it worse than it first looks: it can sustain itself after its cause is gone.

t0   a deploy makes the service 3x slower
t1   timeouts, retries begin
t2   retry load = 4x normal, service saturates
t3   the deploy is rolled back
t4   still down

At t4 the original trigger is gone and the system stays broken, because the retry traffic alone is now enough to keep it saturated. This is a metastable failure — a stable bad state the system will not leave on its own.

The practical consequence: you cannot recover by fixing the trigger. You have to break the loop — shed load, drain queues, or turn retries off — before the service can come back. Teams that have not seen this spend the outage re-deploying and wondering why nothing helps.

The arithmetic of amplification

Retries multiply through a call chain, and the numbers are worse than intuition suggests because they compound rather than add.

A --3x--> B --3x--> C --3x--> D

one user action reaches D as 27 requests

The service that dies is the one furthest from the user, and it dies from load generated by clients that each believe they are being modest. Nobody’s policy looks unreasonable in code review; the product of them is a denial of service you built yourself.

Timeouts come first

A retry policy without a timeout is not a resilience strategy. Without a timeout the call hangs, the worker is held, the pool drains, and the failure spreads upstream — one slow dependency taking down a service that is not broken.

import httpx

client = httpx.AsyncClient(
    timeout=httpx.Timeout(connect=2.0, read=10.0, write=5.0, pool=1.0),
)

pool is the one people omit and then cannot explain. When the pool is exhausted, requests queue inside your own process, so the caller waits far longer than the read timeout suggests and the latency has no server-side trace.

Budget downward, do not set per-call

If your endpoint has a 3-second SLA and calls three services, they cannot each have a 3-second timeout. Allocate once and pass the remainder down:

class Deadline:
    def __init__(self, budget: float) -> None:
        self.expires_at = loop.time() + budget

    def remaining(self) -> float:
        return max(0.0, self.expires_at - loop.time())


async def handler(request) -> Response:
    dl = Deadline(3.0)
    profile = await fetch_profile(request.user_id, dl)   # each call
    prefs = await fetch_prefs(request.user_id, dl)       # sees what
    return render(profile, prefs)                        # is left


async def fetch_profile(user_id: str, dl: Deadline) -> dict:
    left = dl.remaining()
    if left < 0.05:
        raise DeadlineExceeded          # do not start what cannot finish
    resp = await client.get(f"/users/{user_id}", timeout=left)
    return resp.json()

The check before the call is the point. A retry inside an already-expired budget is pure waste — you are computing a result nobody is waiting for, using capacity the next request needs. Deadline propagation is the cheapest defence against amplification, because an expired deadline stops retries at every layer at once without any layer coordinating.

gRPC propagates deadlines natively; over HTTP, send the remaining budget in a header and have the callee honour it.

Retry only what can succeed

RETRYABLE = (
    httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout,
    httpx.RemoteProtocolError,
)

def should_retry(exc: Exception | None, response: httpx.Response | None) -> bool:
    if isinstance(exc, RETRYABLE):
        return True
    if response is None:
        return False
    return response.status_code in {429, 502, 503, 504}

The nuance worth stating unprompted: 500 is not automatically retryable. A deterministic server bug returns 500 every time, so retrying converts one failure into four and adds load during an incident. Worse, if the handler partially applied before failing, the retry duplicates that part.

Honour Retry-After

A 429 is the dependency telling you its capacity, which is more information than your backoff formula has. Ignoring it and applying your own schedule is how a rate limit becomes an outage.

def retry_after_seconds(resp: httpx.Response) -> float | None:
    value = resp.headers.get("Retry-After")
    if value is None:
        return None
    if value.isdigit():                       # delta-seconds form
        return float(value)
    when = parsedate_to_datetime(value)       # HTTP-date form
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

Both forms are legal and servers use both, so parsing only the integer form silently discards the header from the servers that send a date.

The idempotency precondition

Retrying a non-idempotent operation is a correctness bug, not a resilience feature. A timeout is precisely the case where you do not know whether the call applied, so a blind retry on POST /charges can charge twice.

await client.post(
    "/charges",
    json=payload,
    headers={"Idempotency-Key": str(order.idempotency_key)},
)

The key must be stable across attempts and derived from the business operation — generating it inside the retry loop defeats the entire mechanism. See Idempotency keys.

Safe without a key: GET, PUT with the same body, DELETE. Not safe: a POST that creates something, anything that increments, anything that sends a message.

Backoff, jitter, and a budget

from tenacity import (
    retry, retry_if_exception_type, stop_after_attempt,
    wait_exponential_jitter, before_sleep_log,
)

@retry(
    retry=retry_if_exception_type(RETRYABLE),
    wait=wait_exponential_jitter(initial=0.5, max=10, jitter=2),
    stop=stop_after_attempt(3),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
async def fetch(url: str) -> httpx.Response:
    resp = await client.get(url)
    resp.raise_for_status()
    return resp

Jitter is the part people leave out and the part that matters. Without it, every client that failed at the same instant retries at the same instant:

no jitter          full jitter

t+1s  ████████     t+0..1s  ██  █ ██ █
t+2s  ████████     t+0..2s  █ ██  █  █
t+4s  ████████     t+0..4s  █  █ █  ██

The left column is a thundering herd — synchronised waves that re-break the service at each recovery attempt, so it never gets a quiet moment to catch up. Full jitter (random.uniform(0, delay)) spreads hardest and is the right default; the exponential part alone does nothing about synchronisation.

Cap the delay so a retry cannot outlive the caller’s budget, and cap attempts at three. More attempts against a genuinely dead service is just more load.

The retry budget

Backoff limits how fast one client retries. It does nothing about the total, which is what the dependency experiences. A budget does:

class RetryBudget:
    """Allow retries only while they stay a small fraction of traffic."""

    def __init__(self, ratio: float = 0.1, window: int = 1000) -> None:
        self.ratio = ratio
        self.calls = deque(maxlen=window)     # True = retry, False = initial

    def record(self, was_retry: bool) -> None:
        self.calls.append(was_retry)

    def allow_retry(self) -> bool:
        if len(self.calls) < 50:              # too little data to judge
            return True
        return sum(self.calls) / len(self.calls) < self.ratio

Under normal conditions a few percent of calls fail, the ratio stays low, and every retry is permitted. Under a broad outage nearly everything fails, the ratio hits the cap, and retries stop automatically — exactly when they are harmful and exactly when a per-call policy would be trying hardest.

That inversion is the argument for budgets over attempt counts: a budget is aware of the condition an individual call cannot see.

Where it breaks in production

Retries at every layer

Each team adds a sensible three-attempt policy to their own client. Nobody added an unreasonable one, and the bottom service sees 27x.

Fix it by deciding once, for the whole call chain, where retries live — usually the outermost client that owns the SLA, or the innermost adapter that knows the dependency. Write it down, because the failure mode is invisible in any single repository.

The library that also retries

The subtle version of the same bug: botocore retries, httpx transports can, gRPC has a service config, the database driver reconnects. A @retry decorator on top of a client that already retries three times gives nine, and the code shows three.

Read the defaults of every client library in the path. This is the most common source of amplification that nobody wrote.

A timeout longer than the caller’s patience

A 30-second timeout on a service whose caller gives up at 5 seconds means 25 seconds of work on every failed request that nobody will read, consuming capacity while a queue builds behind it. Timeouts are only meaningful relative to the budget above them.

Retrying the 500 that half-applied

The handler wrote a row, then failed on the second write and returned 500. The retry writes the row again. The 500 was honest; the retry was the bug, and it shows up as duplicate records with no error in any log.

The recovery that will not take

If the loop above has set in, restarting the service does not help — it comes up into a full queue and dies again. Recovery means breaking the feedback loop first: shed load at the edge, drain the queue, or disable retries globally, and only then restore traffic. Having a switch that turns retries off is worth more during an incident than any tuning of the backoff curve.

The trade not taken

Two alternatives worth naming, because “add retries” is not the only answer.

Fail fast and let the caller decide. A retry hides a failure inside a latency increase, which is the wrong trade for anything a human is waiting on. An immediate error the UI can act on beats twelve seconds of silence followed by the same error. Retry when the caller is a machine with time to spare; surface it when the caller is a person.

Make it asynchronous instead. If the work does not have to happen inside the request, a queue gives you retries with none of the amplification: the retry load is bounded by consumer concurrency rather than by user traffic, backoff can be minutes instead of milliseconds, and a dead-letter queue is a better place to land than a 500. Most “we need a robust retry policy” conversations are really about work that should not have been synchronous.

And the opposite trade, for tail latency rather than failure: hedged requests — send a second request once the first exceeds p95, take whichever answers, cancel the other. It spends a few percent extra load to cut the tail, which is the right trade for a read-only, idempotent call and a catastrophic one for anything that writes.

Related

• Circuit breakers and bulkheads - stopping retries at a dead dependency
• Fallbacks, partial failure and graceful degradation - what to return when you stop trying
• Idempotency keys - the precondition for retrying a write
• PUT vs PATCH: which writes are safe to retry - which writes are safe to retry