Burn-Rate Alerts for Queue Backlogs

This is the alerting implementation for the objectives defined in SLOs & Alerting for Job Queues, part of Observability & Monitoring for Job Queues — the recording rules, window pairs, and severity tiers that make a page mean something.

Problem Statement

Your queue-depth alert fires four times a week. Three of those are a nightly batch that always drains in ten minutes, and one is real. The rotation has learned to acknowledge and move on, so when a broker partition genuinely stalled the consumers for forty minutes, the page was silenced within seconds of arriving. You need alerts that stay quiet during healthy bursts and fire quickly when users are actually waiting — which means alerting on error-budget consumption rather than on a raw number.

Prerequisites

  • A latency objective with a threshold and a window, per defining SLOs for job latency.
  • job_queue_time_seconds as a histogram with a bucket boundary exactly on the threshold.
  • A counter of enqueued jobs, so the denominator includes work that has not started.
  • Prometheus 2.40+ (or a compatible backend) with recording rules and a rule evaluation interval of 15–30 seconds.

Step 1 — Precompute the SLI

Burn-rate alerts evaluate several windows at once. Doing that against raw histogram buckets on every evaluation is expensive enough to cause rule-evaluation lag, which delays the very alerts you are tuning.

groups:
  - name: queue-slo-recording
    interval: 30s
    rules:
      - record: job:queue_time_good:rate5m
        expr: sum by (queue) (rate(job_queue_time_seconds_bucket{le="30"}[5m]))
      - record: job:queue_time_total:rate5m
        expr: sum by (queue) (rate(jobs_enqueued_total[5m]))
      # Failure ratio — the quantity a burn rate is measured against.
      - record: job:queue_time_bad:ratio5m
        expr: |
          1 - (
            job:queue_time_good:rate5m
            / clamp_min(job:queue_time_total:rate5m, 0.001)
          )
      # Repeat for each window the alerts need.
      - record: job:queue_time_bad:ratio1h
        expr: |
          1 - (
            sum by (queue) (rate(job_queue_time_seconds_bucket{le="30"}[1h]))
            / clamp_min(sum by (queue) (rate(jobs_enqueued_total[1h])), 0.001)
          )

clamp_min prevents a zero-traffic window from producing NaN, which compares false and would silently disable the alert exactly when a queue has stopped receiving work.

Step 2 — Understand What a Burn Rate Is

Burn rate is how fast the error budget is being consumed relative to the rate that would exactly exhaust it over the window. A 99% objective over 28 days allows a 1% failure ratio; a sustained 14.4% failure ratio is a burn rate of 14.4, and it consumes the entire budget in under two days.

Budget lifetime at different burn rates Three lines showing cumulative error-budget consumption. At a burn rate of one, the budget is exhausted exactly at the end of the twenty-eight-day window. At six times, it is gone in under five days. At fourteen point four times, it is gone in under two days, which is why that rate warrants an immediate page. How fast is the budget going? 100% 0 1x — lasts the full 28 days 6x — gone in 4.7 days (ticket) 14.4x — gone in 1.9 days (page) day 0 day 28 Each severity tier is just a slope: pick the slope at which you want to be woken.

The severity tiers below are the widely-used defaults; they trade how much budget is consumed before firing against how noisy the alert is.

Severity Burn rate Long window Short window Budget spent at fire
Page 14.4× 1h 5m 2%
Page 6× 6h 30m 5%
Ticket 3× 24h 2h 10%
Ticket 1× 72h 6h 10%

Step 3 — Write the Multi-Window Rules

The short window is what stops an alert from firing on a degradation that has already recovered; the long window is what stops it firing on a momentary spike.

groups:
  - name: queue-slo-alerts
    rules:
      - alert: QueueLatencyBudgetBurnFast
        expr: |
          job:queue_time_bad:ratio1h{queue="urgent"} > (14.4 * 0.01)
          and
          job:queue_time_bad:ratio5m{queue="urgent"} > (14.4 * 0.01)
        for: 2m
        labels:
          severity: page
          queue: urgent
        annotations:
          summary: "urgent queue burning budget at 14x — 2% of the 28d budget gone"
          description: "{{ $value | humanizePercentage }} of jobs are starting later than 30s."
          runbook: "https://runbooks.internal/queues/urgent-latency"
          dashboard: "https://grafana.internal/d/queues/urgent"

      - alert: QueueLatencyBudgetBurnSlow
        expr: |
          job:queue_time_bad:ratio6h{queue="urgent"} > (6 * 0.01)
          and
          job:queue_time_bad:ratio30m{queue="urgent"} > (6 * 0.01)
        for: 15m
        labels:
          severity: page
          queue: urgent

The for: clause is short on the fast alert because the short window already provides the debounce; making it longer just delays the page without reducing noise.

Step 4 — Cover the Cases Burn Rate Misses

Burn rate is a ratio, so it goes blind whenever the denominator goes to zero. Two failures produce exactly that: a consumer that has stopped entirely with no new arrivals, and a producer that has stopped publishing. Cover both with absolute signals.

      # A queue whose head has not moved: no completions, work still waiting.
      - alert: QueueStalled
        expr: |
          max by (queue) (queue_oldest_message_age_seconds) > 600
          and max by (queue) (queue_depth) > 0
          and rate(jobs_completed_total[10m]) == 0
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.queue }} has work but no completions for 10 minutes"

      # Producers stopped: enqueue rate collapsed against its own weekly baseline.
      - alert: QueueEnqueueRateCollapsed
        expr: |
          rate(jobs_enqueued_total[15m])
            < 0.2 * rate(jobs_enqueued_total[15m] offset 1w)
        for: 20m
        labels: { severity: ticket }
Three failure shapes, three alerts Three panels. A degradation with continuing traffic is caught by the multi-window burn-rate alert. A stalled consumer, where depth is non-zero and completions are zero, is caught by the stalled-queue alert because burn rate would go blind. A collapsed producer, where nothing is enqueued at all, is caught by comparing enqueue rate against the same window a week earlier. A ratio cannot see a queue that stopped Degraded, still flowing arrivals normal, waits long burn-rate alert Consumer stalled depth > 0, completions = 0 flat — no signal in the ratio oldest-age alert Producer stopped nothing enqueued at all week-over-week alert Ship all three. Each is blind to the failures the others catch. Queue depth alone appears in none of them — it is dashboard context, not an alert.

Step 5 — Route by Owner, Not by Catch-All

# alertmanager.yml
route:
  group_by: [queue, alertname]
  routes:
    - matchers: [severity="page", queue=~"urgent|checkout"]
      receiver: payments-oncall
      group_wait: 30s
      repeat_interval: 4h
    - matchers: [severity="ticket"]
      receiver: platform-tickets
      repeat_interval: 24h
inhibit_rules:
  # A stalled queue implies budget burn — do not send both pages.
  - source_matchers: [alertname="QueueStalled"]
    target_matchers: [alertname=~"QueueLatencyBudgetBurn.*"]
    equal: [queue]

The inhibition rule matters as much as the alerts: during a real stall, the burn-rate alerts will also fire, and three pages for one incident is how a good alerting setup gets muted.

Step 6 — Tune the Windows to Your Traffic Shape

The 14.4× / 6× / 3× tiers assume traffic that is roughly continuous. Job queues frequently are not: many have a diurnal shape with an order-of-magnitude difference between peak and trough, and some are almost entirely bursty. Two adjustments make the alerts behave under those shapes.

First, raise the short window when overnight traffic is thin. With 20 jobs an hour at 04:00, a five-minute window contains one or two observations, so a single slow job pushes the ratio to 50% and pages. Scaling the short window with the inverse of the traffic rate is over-engineering; picking a window that holds at least fifty observations at the quietest hour is not.

# How many observations does the short window actually contain at the trough?
min_over_time(
  (sum by (queue) (rate(job_queue_time_seconds_count[5m])) * 300)[7d:5m]
)
# → 1.8 for the "reports" queue: a 5m window is far too small there.
#   Use 30m for its fast pair, accepting a slower page on a queue nobody watches at 4am.

Second, add a minimum-volume guard to the alert itself so a handful of observations can never page:

      - alert: QueueLatencyBudgetBurnFast
        expr: |
          job:queue_time_bad:ratio1h{queue="urgent"} > (14.4 * 0.01)
          and job:queue_time_bad:ratio5m{queue="urgent"} > (14.4 * 0.01)
          and job:queue_time_total:rate5m{queue="urgent"} > 0.5   # ≥ 150 jobs / 5m

The guard is a deliberate trade: below the volume floor, the burn-rate alert stops protecting the queue, and the stalled-queue and oldest-age alerts from step 4 become the only coverage. That is the correct division of labour — ratios are the right instrument for busy queues, absolute ages for quiet ones — but it is worth writing down explicitly, because a queue that is quiet at night and busy by day switches between the two regimes twice a day.

Why both windows must exceed the threshold Two scenarios. In the first, a two-minute spike pushes the five-minute window above the burn threshold while the one-hour window stays below it, so the alert does not fire. In the second, a sustained degradation pushes both windows above the threshold and the alert fires within minutes. Short window catches it; long window confirms it Brief spike — suppressed 5m window: over 1h window: under no page — correct Sustained degradation — fires 5m window: over 1h window: over page after ~5 minutes The long window alone would page ten minutes late; the short window alone would page on every blip.

Verification

  1. Rules evaluate cheaply. prometheus_rule_group_last_duration_seconds for the SLO group must stay well under the evaluation interval.
  2. A synthetic burn fires the page. Pause consumers on a staging queue and confirm the fast alert fires within roughly 5–7 minutes, not immediately and not after an hour.
  3. A brief spike does not fire. Inject a two-minute delay burst; the short window should exceed the threshold while the long window does not, so nothing pages.
  4. The stalled case is caught. Stop consumers with no new arrivals; burn-rate alerts stay silent and QueueStalled fires.
  5. Inhibition works. During the stall test, confirm only one notification is delivered.

Gotchas & Edge Cases

Windows shorter than the scrape interval. A 5-minute window with a 60-second scrape has five samples; one missed scrape moves the ratio dramatically. Keep the short window at least five times the scrape interval.

Recording rules with mismatched labels. If the good-count and total-count rules aggregate by different label sets, the division silently produces an empty vector and the alert can never fire. Always aggregate both by the same by (queue).

Burn thresholds copied without adjusting the objective. 14.4 * 0.01 encodes a 99% target. For a 99.9% objective the multiplier applies to 0.001, and copying the rule without changing the constant makes it a hundred times less sensitive.

Alerts that fire during planned batch windows. A nightly bulk queue legitimately exceeds its threshold. Either give that queue its own objective or add a time-based silence — but never widen the interactive queue's threshold to accommodate it.

Recording rules that lag behind the alert interval. If the rule group evaluates every 30 seconds but the alert group evaluates every 15, half the alert evaluations read a stale SLI. Keep the alert interval equal to or slower than the recording interval it depends on.

No runbook link. An alert without one is a research project at 3am; see writing runbooks for queue incidents for the template that makes the first five minutes mechanical.

Related