Resilience
How a service survives its dependencies failing. Four files, each covering a different failure duration.
| # | File | Covers |
|---|---|---|
| 01 | Timeouts, retries and backoff | timeouts, which errors to retry, jitter, retry amplification |
| 02 | Circuit breakers and bulkheads | breakers, resource isolation, load shedding |
| 03 | Fallbacks, partial failure and graceful degradation | the fallback ladder, stale-while-revalidate, partial failure, error budgets |
| 04 | Resilient orchestration across services | sagas, compensation, durable state, idempotency |
The layered answer
load shedding -> are we over capacity at all?
bulkhead -> is this dependency's resource allocation free?
circuit breaker -> is this dependency known-bad right now?
timeout + retry -> attempt, bounded, with jitter
fallback -> what do we return if it still failed?Each layer handles a different failure duration: retries for a blip, breakers for a sustained outage, bulkheads for slowness, shedding for overload.
Three things worth saying unprompted
Retry amplification. Three retries at each of three layers is nine requests per user action. Under partial degradation, retry load is often what turns a slow service into a dead one. Retry at one layer, add a budget, use a breaker.
Idempotency is the precondition. Retrying a non-idempotent operation is a correctness bug. A timeout is exactly the case where you don’t know whether it applied, so a blind retry can double-charge.
Order operations so failure is cheap. Reversible and likely-to-fail first, irreversible last. Every step you reorder is a compensating transaction you don’t have to write.