Incident response, runbooks and postmortems
Observability tells you something is wrong. This is the operational layer on top: who decides, in what order they look, what they are allowed to do about it, and what happens afterwards so it does not recur.
Interviewers probe here because it separates people who have carried a pager from people who have read about it. The tell is whether you reach for mitigation before diagnosis.
Stop the bleeding, then diagnose
The single most useful principle, and the one people invert under pressure: restore service first, find the cause second. Rolling back a deploy you are only 70% sure is the culprit is correct — the rollback is cheap and reversible, and the investigation is much calmer once users are unaffected.
That means every system needs a small set of pre-authorised mitigations that anyone on call may apply without a meeting:
| Mitigation | Costs | Buys |
|---|---|---|
| Roll back the deploy | the release | usually everything |
| Scale workers out | money | throughput |
| Reduce concurrency | throughput | stability |
| Shed or defer load | some requests | the rest of them |
| Disable a feature flag | that feature | its blast radius |
| Shorten retention | history | disk |
Notice that half of these make the system worse in a chosen dimension. That is the point — degrading deliberately beats degrading arbitrarily.
A triage runbook is an ordered list, not prose
A runbook someone reads at 3am is a checklist with commands in it. Order the checks by cheapest and most likely first, and put the actual command inline — nobody composes SQL under pressure.
-- 1. Connections: is the pool exhausted, and is it a leak?
SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity GROUP BY state ORDER BY count DESC;
-- 2. Disk: what is actually big?
SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS size,
n_dead_tup, last_autovacuum
FROM pg_stat_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;
-- 3. Slow queries: where the time goes (cumulative since the last reset;
-- "what changed?" needs two snapshots and a delta)
SELECT calls, round(mean_exec_time::numeric, 1) AS avg_ms, query
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;
-- 4. Blocking: is something holding a lock?
SELECT pid, wait_event_type, wait_event, now() - query_start AS dur, query
FROM pg_stat_activity WHERE wait_event_type = 'Lock';Then the layers above the database — queue depth and oldest-job age, memory per worker, error rate by endpoint. The structure that works is: symptom → the three checks that discriminate between causes → the mitigation for each.
Write runbooks from real incidents. A runbook invented in advance covers what you imagined; one written during a postmortem covers what happened.
SLIs and SLOs give the alerts a reason to exist
Alert on symptoms users feel, not on causes. CPU at 90% is not an incident if latency is fine; it is a capacity conversation.
For a queue-backed automation system the four that matter:
- Latency — end-to-end, trigger to completion, at p95 and p99.
- Error rate — failed executions over total, excluding deliberate rejects.
- Queue delay — the age of the oldest waiting job. This is the leading indicator; it moves before latency does and it is what tells you throughput has stopped being sustainable.
- Saturation — connection-pool usage, disk headroom, memory per worker.
Set thresholds from the SLO’s error budget rather than from a round number, and use burn rate so a fast burn pages and a slow burn opens a ticket. Mechanics in SLO, SLI, SLA, and Error Budgets.
Alert on absence too. A workflow that stops running fires no errors. A heartbeat — “executions in the last hour is zero” — catches the silent failure class that error-rate alerting structurally cannot.
Severity, so the response matches the problem
| Sev | Means | Response |
|---|---|---|
| 1 | users blocked, data at risk | page, all hands |
| 2 | degraded, workaround exists | page in hours |
| 3 | contained, no user impact | ticket |
Two roles are worth naming even on a small team: someone coordinating (who decides, who communicates, who keeps the timeline) and someone investigating. One person doing both does neither well, and the timeline is what the postmortem depends on.
The postmortem
Blameless is not politeness — it is the only way to get accurate information. A person who expects blame reports a version of events that protects them, and then you fix the wrong thing.
A template that produces action rather than narrative:
## Summary
One paragraph: what broke, who was affected, for how long.
## Impact
Users affected, requests failed, data lost or delayed. Numbers.
## Timeline
UTC timestamps. Detection, escalation, mitigation, resolution.
Note the gaps: detection lag and mitigation lag are separate problems.
## Root cause
The chain, not one line. "X because Y because Z."
## What went well / what was luck
Separate them. Luck is not a control.
## Action items
| Action | Owner | Due | Ticket |
Each one: preventive, detective, or mitigating. No "be more careful".The two questions with the most value in them:
- “How long until we knew?” Detection lag is usually the biggest lever and the one nobody lists as an action item.
- “What made this hard to diagnose?” The answer is a missing metric, log field or dashboard, and fixing it shortens every future incident.
Track actions to completion or do not write them. A postmortem whose actions are never done is a document that teaches the team the process is theatre.
Change control that does not slow you down
Most incidents follow a change. The lightweight controls that pay for themselves:
- Deploy small, deploy often. A small change has a small suspect list.
- Make rollback boring and rehearsed, so it is nobody’s brave decision.
- Config behind flags, not deploys, so mitigation is seconds.
- Migrations decoupled from code — expand, migrate, contract, per Zero-downtime migrations.
- Freeze windows only where they earn it. A blanket freeze pushes risk into a bigger batch afterwards.
Related
Interview angle 4
- “Walk me through responding to a production incident.” - stabilise before diagnosing. Apply a pre-authorised mitigation — roll back, scale, shed load, flip a flag — then investigate with users unaffected. Name the two roles: someone coordinating and communicating, someone investigating.
- “What goes in a runbook?” - an ordered checklist with the actual commands, cheapest and most likely check first, structured as symptom, the checks that discriminate, and the mitigation for each. Written from real incidents, not imagined ones.
- “What do you alert on?” - symptoms users feel, off an SLO with burn-rate thresholds. For a queue system: end-to-end latency, error rate, oldest-job age as the leading indicator, and saturation. Plus a heartbeat, because a workflow that silently stops produces no errors to alert on.
- “What makes a postmortem useful?” - blameless so the facts are accurate, a timeline that separates detection lag from mitigation lag, and action items with owners and dates that are tracked to done. The highest-value question is usually “how long until we knew”, because detection is the biggest lever and nobody lists it.