SLO, SLI, SLA, and Error Budgets

Article.

The Google SRE framework for turning “is the service healthy?” into a number that decides engineering work. The interview question under it: how do you turn reliability into a budget, and alert on that budget without paging on every blip or sleeping through a slow leak? The answer is burn-rate alerting over two windows, and the reason for the two windows is worth being able to show.

The four terms

The SLO is tighter than the SLA, so an internal incident spends budget before it breaches a contract. Promise 99.9% externally, run to 99.95% internally.

A good SLI

It is a ratio of user-visible outcomes, measured where the user experiences them:

• Availability: successful requests / all requests.
• Latency: requests faster than 300 ms / all requests.
• Correctness: orders processed correctly / all orders.
• Freshness: writes visible to readers within a minute / all writes.

CPU, memory, queue depth and pod counts are not SLIs. They are causes, useful on a dashboard, but no user experiences 85% CPU.

The nines

At 99.999% the whole month’s budget is shorter than acknowledging a page. Start from what the service actually achieved over the last 90 days, and tighten only where users would notice the difference.

The mechanism: budget, burn rate, and two windows

The error budget is a quantity: at 99.9% over 28 days, 0.1% of requests, or about 40 minutes of total failure. Burn rate is how fast you are spending it, relative to spending it evenly:

burn_rate = observed_error_rate / (1 - SLO)

burn 1    -> the budget lasts exactly the window (28 days)
burn 14.4 -> 2% of a 30-day budget gone in one hour

An alert on burn rate pages on budget consumption, not on any particular error rate. Each rule uses two windows that must both exceed the threshold:

• the long window (1 hour) says the problem is significant: enough budget spent to matter;
• the short window (5 minutes) says it is still happening, so the alert stops soon after recovery.

The standard rule set from the SRE Workbook:

Three incidents, minute by minute

A simulation against a 99.9% SLO, comparing a static threshold (“error rate above 1% for 5 minutes”), a 1-hour burn rule alone, and the two-window rule:

A. 3-minute blip at 20% errors (starts 01:00): 1.5% of the 28-day budget spent
  static >1% for 5 min     never fires
  burn 1 h only            never fires
  burn 1 h + 5 min         never fires
B. 10-minute outage at 100% (01:00-01:10): 24.8% of the 28-day budget spent
  static >1% for 5 min     fires at 01:04, last firing at 01:09
  burn 1 h only            fires at 01:00, last firing at 02:08
  burn 1 h + 5 min         fires at 01:00, last firing at 01:13
C. 0.5% errors for 3 days: 53.6% of the 28-day budget spent
  static >1% for 5 min     never fires
  burn 1 h only            never fires
  burn 1 h + 5 min         never fires
  burn 3 d + 6 h (ticket)  fires at 14:24, last firing at 2d 23:59

Read it as three arguments:

• A is a rolled-back deploy that cost 1.5% of the budget. Nothing pages, which is correct: nobody should be woken for it.
• B is why the short window exists. The 1-hour rule alone keeps firing until 02:08, almost an hour after recovery, because the outage stays inside its window. With the 5-minute window it clears at 01:13.
• C is why static thresholds fail. A 0.5% error rate never crosses “1%”, yet it spends more than half the month’s budget in three days. Only the slow-burn ticket sees it, 14 hours in.

As a Prometheus rule

# Prometheus: the fast-burn page, 99.9% SLO
- alert: CheckoutErrorBudgetFastBurn
  expr: |
    (sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
      / sum(rate(http_requests_total{job="checkout"}[1h]))) > 14.4 * 0.001
    and
    (sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{job="checkout"}[5m]))) > 14.4 * 0.001
  labels: { severity: page }

Sloth or Pyrra generate the full rule set from a short SLO definition, which beats hand-writing four rules per SLO per service.

Latency SLOs

Latency is continuous, so it becomes a ratio of good events:

good_events  = requests completing in < 300 ms
total_events = all requests
SLO: 99% of requests in < 300 ms over 28 days

Count it from histogram buckets (le="0.3" over the total), not from a stored p99: percentiles cannot be averaged across instances or time, and a “p99 of p99s” is not the p99 of anything. For a multi-step user flow, put the SLO on the step the user waits for.

Where it breaks in production

• The SLI is measured in the wrong place. The app reports 99.99% success while the load balancer returns 502s for requests that never reached it, and clients time out on others. Measure at the edge (load balancer, gateway) or from the client, not only inside the process.
• Low traffic makes burn rates noisy. At 100 requests an hour one failure is a 1% error rate, a burn of 10 against 99.9%. Require a minimum event count in the rule, add synthetic traffic, or use longer windows for small services.
• Static thresholds kept alongside. Two alerts for the same problem page twice, and the static one still sleeps through scenario C. Migrate fully.
• No consequence for a spent budget. If an exhausted budget does not slow feature work, the SLO is a dashboard, not a policy. Agree the response (feature freeze, reliability work first) with product before the first breach.
• Too many SLOs. Two to four per service. Twenty means none of them decides anything.

The dashboard the SLO implies

An alert says something broke; the dashboard has to answer what to look at, in what order. Build three:

The service-health board fits one screen: SLO burn rate first, then the RED metrics (rate, errors, duration), then the two or three saturation signals most likely to be the cause. Draw the SLO threshold on every graph, show p50 next to p99, and annotate deploys. Link the runbook from the dashboard and the dashboard from the alert (Incident response, runbooks and postmortems).

The trade not taken: paging on causes

The traditional pager watches causes: CPU above 80%, queue depth above 1,000, a pod restarting. Each is easy to set up and each pages without a user being affected, so the team learns to ignore pages, which is how the one that matters gets missed. Causes also miss failures nobody thought to threshold, such as a bad config that returns wrong answers at normal CPU.

Paging on symptoms (budget burn on user-visible SLIs) pages when users are affected and only then. The causes stay on the saturation dashboard, where they explain an incident instead of being one.

Related

• Prometheus - histograms, rate(), recording rules
• The Three Pillars of Observability - metrics, logs and traces together
• Incident response, runbooks and postmortems - what happens after the page
• Cost as a signal - cost treated as a signal, like latency