Writing Runbooks for Queue Incidents

Every alert defined in SLOs & Alerting for Job Queues, part of Observability & Monitoring for Job Queues, should link to a runbook. This is the template, and the reasoning behind each section.

Problem Statement

A page fires at 03:12: "urgent queue burning budget at 14x". The responder has never worked on that queue. They spend eleven minutes finding the right dashboard, six more determining that the backlog is caused by a slow downstream API rather than by dead workers, and then hesitate — is scaling the fleet safe when the dependency is already struggling? The incident lasts forty minutes, of which perhaps eight were the actual fix. You want the first five minutes to be mechanical, and the decision about mitigation to have been made in advance by someone with context.

Prerequisites

  • Alerts carrying runbook and dashboard annotation links, per burn-rate alerts for queue backlogs.
  • A named owner per queue who can approve the mitigation ladder.
  • Read access for responders to broker metrics, worker logs, and the scaling controls the runbook tells them to use.

Step 1 — Open With Orientation, Not Theory

The first screen must answer: what broke, who is affected, and where do I look. No background, no architecture diagrams.

# Runbook: urgent queue latency

**Objective:** 99% of urgent jobs start within 30s (28-day window).
**Impact when violated:** password resets and 2FA codes are delayed;
users retry at ~45s and file tickets at ~2min.
**Dashboard:** https://grafana.internal/d/queues/urgent
**Owner:** #payments-platform (escalate to on-call lead after 30 min)

## First 5 minutes
1. Is anything consuming? `kubectl get pods -l queue=urgent` — expect 4 Running.
2. Is the queue moving? Depth AND oldest-message age panels; falling age = draining.
3. Is the dependency healthy? Check the payments-api error-rate panel.
4. Post in #incident-queues with the answers to 1–3. Do not skip this.

Step four is not bureaucracy. Writing the three answers down forces the responder to actually check them, and it gives anyone joining a five-second summary instead of a scroll-back.

Step 2 — Diagnose by Signal Shape

Most queue incidents fall into four shapes, and the combination of depth, age, and throughput distinguishes them without any deep investigation.

Depth Oldest age Throughput Diagnosis First move
Rising Rising Normal Arrival rate exceeds capacity Scale consumers
Rising Rising Falling Consumers are slow or blocked Check the dependency
Flat > 0 Rising Zero Consumers dead, stuck, or disconnected Restart / check broker connectivity
Falling Falling High Recovering from a burst Wait; verify it clears
Diagnosing a queue incident from three signals Four quadrant cards. Rising depth with normal throughput indicates insufficient capacity, so scale consumers. Rising depth with falling throughput indicates slow consumers, so investigate the downstream dependency. Flat depth with zero throughput indicates dead or stuck consumers, so check connectivity and restart. Falling depth with high throughput indicates recovery, so wait and verify. Three signals, four diagnoses Depth ↑ · age ↑ · throughput normal Arrivals simply exceed capacity — nothing is broken. scale consumers Depth ↑ · age ↑ · throughput ↓ Consumers are slow: a dependency is degraded. do NOT scale first Depth flat > 0 · age ↑ · throughput 0 Nothing is consuming: crash loop, lost lease, broker split. check connectivity, restart Depth ↓ · age ↓ · throughput high Recovering from a burst — the system is working. wait, verify it clears

The top-right quadrant is the one worth spelling out in every runbook, because the instinctive response is wrong: adding consumers to a queue whose dependency is already saturated increases pressure on the thing that is failing and lengthens the incident.

Mitigation ladder ordered by blast radius Four rungs from least to most disruptive. Scaling consumers is safe when the dependency is healthy. Raising retry backoff relieves a struggling dependency at the cost of slower recovery. Shedding bulk producers delays reports. Pausing the producer entirely requires incident-lead approval and creates a drain obligation. Stop at the first rung that restores the objective 1 · scale consumers safe only if the dependency is healthy 2 · raise retry backoff slower recovery for transient faults 3 · shed bulk producers reports late; notify the owning team 4 · pause the producer creates a drain obligation — lead approval

Step 3 — Order Mitigations by Blast Radius

List mitigations from least to most disruptive, with the cost of each stated plainly, so the responder can stop at the first one that works.

## Mitigation ladder (stop at the first that restores the objective)

1. **Scale consumers** — `kubectl scale deploy/worker-urgent --replicas=12`
   Cost: more load on payments-api. Safe ONLY if the dependency is healthy.
   Verify: throughput rises within 60s and oldest age starts falling.

2. **Raise retry backoff** — set `RETRY_BASE_DELAY=30` and rolling-restart.
   Cost: slower recovery for genuinely transient failures.
   Use when: retry rate is a large fraction of throughput.

3. **Shed bulk producers** — disable the nightly rollup schedule.
   Cost: reports are late; no data loss. Requires: #data-platform notified.

4. **Pause the producer** — feature-flag `payments.async_receipts=off`.
   Cost: receipts queue in the application DB instead; drain within 2h or the
   table grows unbounded. Requires: incident lead approval.

Each rung carries a cost and a verification step. That is what lets someone unfamiliar with the system act confidently — and, equally, know when to stop and escalate rather than reaching for the biggest lever.

Step 4 — Include the Commands, Verbatim

A runbook that says "check the broker" instead of giving the command wastes minutes and invites mistakes.

# Consumers present and connected?
kubectl get pods -l queue=urgent -o wide
redis-cli CLIENT LIST | grep -c 'cmd=blpop'          # Redis: blocked consumers
rabbitmqctl list_queues name consumers messages_ready # RabbitMQ: consumer count

# Is the head of the queue actually moving?
redis-cli LLEN urgent; sleep 10; redis-cli LLEN urgent

# What are workers doing right now?
kubectl exec deploy/worker-urgent -- celery -A app inspect active | head -40

# Dependency latency from inside the worker network namespace
kubectl exec deploy/worker-urgent -- \
  curl -s -o /dev/null -w '%{time_total}\n' https://payments.internal/health
Where incident time actually goes Two stacked timelines for the same incident. Without a runbook, eleven minutes are spent finding the dashboard, six determining the cause, six hesitating over the mitigation, and eight applying the fix. With a runbook, orientation takes two minutes, diagnosis three, and the same eight-minute fix dominates the total. The fix was always eight minutes no runbook — 31 min finding the dashboard diagnosis hesitation the fix with a runbook — 13 min orient diag the fix 0 min 35 min A runbook does not make the fix faster. It removes everything that was not the fix.

Step 5 — Close With Recovery and Follow-Up

## After the objective is met again
- [ ] Confirm depth and oldest age are back to baseline for 15 minutes.
- [ ] Revert any temporary scaling (or record why it should stay).
- [ ] Re-enable anything shed in steps 3–4, oldest work first.
- [ ] Check the DLQ: did anything dead-letter during the incident? Replay it.
- [ ] Record budget spent and the attributed cause in the SLO log.
- [ ] Open a follow-up if any step above was unclear or wrong.

The DLQ check is the step most often skipped and the most expensive to skip: an incident that dead-lettered 4,000 jobs is not over when latency recovers, because that work has not run. The replay procedures in replaying dead-letter messages in RabbitMQ and the SQS redrive path belong linked directly from here.

Step 6 — Keep Runbooks From Rotting

A runbook is code that runs on humans, and like code it decays silently. Three mechanisms keep it honest, and all three are cheap.

Exercise it on a schedule. Once a quarter, run a game day: break a staging queue in one of the four shapes from step 2 and have someone who did not write the runbook respond using only the document. Every hesitation is a defect to file. This finds far more rot than reading the document does, because reading skips the steps that no longer work.

Version it with the code. Keep the runbook in the same repository as the worker it describes, so a pull request that renames a deployment touches the runbook in the same diff. A wiki page in another system is a page nobody updates.

# CI check: every alert rule must reference a runbook that exists in this repo
for rb in $(yq '.groups[].rules[].annotations.runbook // ""' alerts/*.yml | grep -o '[^/]*$'); do
  test -f "runbooks/${rb}.md" || { echo "missing runbook: $rb"; exit 1; }
done

Update it during the incident, not after. The responder is the only person who will ever know exactly which step was wrong, and that knowledge has a half-life of about a day. A one-line "edit this runbook" link at the top of the page, pointing straight at the file in the repository, captures more corrections than any retrospective action item — because it costs thirty seconds while the frustration is still fresh, rather than a scheduled hour a week later that competes with everything else.

Verification

  1. Someone unfamiliar can follow it. Have an engineer from another team execute the first five minutes against a staging incident, unaided, and time them. Ten minutes means the runbook is not specific enough.
  2. Every command runs as written. Copy-paste each into a staging shell; stale flags and renamed deployments are the most common rot.
  3. Every alert links to a runbook. Query the alerting rules for any rule missing a runbook annotation and treat each as a defect.
  4. Ladder steps have verifications. Every mitigation must state how to tell it worked; a step without one gets applied repeatedly under stress.

Gotchas & Edge Cases

Runbooks that explain instead of instruct. Architecture belongs in documentation, not on the page an on-call engineer opens at 3am. Keep background below the actions, or in a linked page.

Commands that need permissions the responder lacks. Test the runbook as the on-call role, not as an administrator. A kubectl scale that returns Forbidden at 03:15 converts a five-minute fix into an escalation.

No stated cost for the destructive options. "Pause the producer" without "receipts accumulate in the application DB and must drain within two hours" invites a mitigation that creates the next incident.

Rot after a rename. Deployment names, dashboard URLs, and queue names all drift. Link to the runbook from the alert and from the code that defines the queue, so a rename touches both.

One runbook for every queue. A generic document is ignored because nothing in it is specific. Keep a shared ladder template, but give each queue its own page with its own commands, owners, and costs.

Related