SLOs & Alerting for Job Queues

Dashboards tell you what happened; service level objectives tell you whether it mattered. This guide turns the raw signals collected in Observability & Monitoring for Job Queues into objectives, error budgets, and alerts that fire when users are affected rather than when a number looks unusual.

Async systems make this harder than request/response services in one specific way: there is no status code and no client waiting on a socket, so "success" has to be defined by you. A job that completes correctly four hours late has failed the user even though every metric says it succeeded. Queue-time — the interval between enqueue and the start of execution — is therefore the primary SLI for most job systems, and the one teams most often forget to measure.

Choosing the Right Indicator

Four candidate indicators exist for a job system, and they answer different questions. Most teams need two: a latency objective on queue time and a correctness objective on eventual completion.

Measuring queue time requires the producer to stamp the message. Without an enqueued_at field the consumer cannot know how long the message waited, and you are left inferring it from queue depth — a much weaker proxy that breaks whenever throughput changes.

# producer
task.apply_async(args, headers={"enqueued_at": time.time()})

# consumer — record the wait as a histogram, per queue and task
from prometheus_client import Histogram
QUEUE_TIME = Histogram(
    "job_queue_time_seconds", "Enqueue to execution start",
    ["queue", "task"],
    # Buckets must straddle your objective: 30s here, with resolution around it.
    buckets=(0.1, 0.5, 1, 2, 5, 10, 30, 60, 300, 900, 3600),
)

@task_prerun.connect
def on_prerun(sender=None, task_id=None, task=None, **kw):
    enq = (task.request.headers or {}).get("enqueued_at")
    if enq:
        QUEUE_TIME.labels(queue=task.request.delivery_info.get("routing_key", "?"),
                          task=sender.name).observe(time.time() - enq)

Writing an Objective That Means Something

An SLO has three parts: an indicator, a target, and a window. "99% of send_receipt jobs start within 30 seconds, measured over 28 days" is an objective. "Queue latency should be low" is a wish.

Set the target from what the product actually needs, not from what the system currently does. A password reset needs seconds; a nightly report needs hours. Writing one objective for every queue guarantees it is simultaneously too strict for batch work and too loose for interactive work — which is how a team ends up with an SLO nobody trusts.

# slo.yaml — objectives live in version control next to the code they describe
objectives:
  - name: interactive-job-timeliness
    queue: urgent
    indicator: queue_time_seconds
    target: 0.99            # 99% of jobs
    threshold: 30           # start within 30 seconds
    window: 28d
    budget_minutes: 403     # 1% of 28 days — the error budget

  - name: bulk-job-completion
    queue: bulk
    indicator: completion_ratio
    target: 0.999           # 99.9% of enqueued jobs complete
    window: 28d

The error budget is what makes the number operational: 99% over 28 days permits about 403 minutes of violation. Spending half of it in one incident is a signal to slow down feature work on that queue — and, equally, a quiet month with no spend is a signal the objective may be too loose.

Alerting on Burn Rate, Not on Thresholds

A threshold alert on queue depth fires whenever traffic is high, which trains people to ignore it. Burn-rate alerting fires on how fast the error budget is being consumed, which correlates with user impact by construction.

A burn rate of 1 means the budget will be exactly exhausted at the end of the window; a burn rate of 14.4 means it will be gone in two days. Pairing a fast window with a slow one filters out brief spikes while still catching real degradation quickly.

# Fraction of jobs that missed the 30s objective, over two windows
- alert: UrgentQueueBudgetBurnFast
  expr: |
    (
      1 - (
        sum(rate(job_queue_time_seconds_bucket{queue="urgent",le="30"}[1h]))
        / sum(rate(job_queue_time_seconds_count{queue="urgent"}[1h]))
      )
    ) > (14.4 * 0.01)
    and
    (
      1 - (
        sum(rate(job_queue_time_seconds_bucket{queue="urgent",le="30"}[5m]))
        / sum(rate(job_queue_time_seconds_count{queue="urgent"}[5m]))
      )
    ) > (14.4 * 0.01)
  for: 2m
  labels: { severity: page }
  annotations:
    summary: "Urgent queue burning error budget 14x — 2% consumed"
    runbook: "https://runbooks.internal/queues/urgent-latency"

The and between the long and short window is what suppresses a one-minute blip: the condition only holds when the degradation is both severe and still happening. Burn-rate alerts for queue backlogs covers the recording rules that make these queries cheap enough to evaluate every fifteen seconds.

The Four Signals Worth Paging For

Queue depth is deliberately absent. Depth without duration is meaningless — 50,000 messages draining in 40 seconds is healthy, and 200 messages that have not moved in an hour is an outage. Alert on age and burn rate, and keep depth on the dashboard as context.

Reporting the Budget So It Changes Decisions

An error budget only does its job if someone looks at it between incidents. The mechanism that works is a weekly report per queue showing budget remaining, the largest single consumer of budget in the period, and the trend — followed by an agreed policy for what happens at each level.

# Budget consumed so far in the 28-day window, as a fraction of the 1% allowance
1 - (
  sum_over_time(job:queue_time_sli:ratio5m{queue="urgent"}[28d])
  / count_over_time(job:queue_time_sli:ratio5m{queue="urgent"}[28d])
) / 0.01

The policy is more valuable than the number. A budget with no agreed consequence is a dashboard nobody opens; a budget that visibly reorders the next sprint is a tool the whole team starts checking. Attribute spend to a cause each time — a dependency outage, a deploy that shipped a slow query, a tenant burst that outran the autoscaler — because after two quarters that attribution list, not the raw percentage, is what tells you where the reliability work actually belongs.

Making a Page Actionable

An alert that fires without a runbook link is an alert that turns into a twenty-minute archaeology session at 3am. Every paging rule should carry: what the objective is, which dashboard shows the current state, the three most likely causes in order of probability, and the safe mitigations available before root cause is known.

For queues, the standard mitigation ladder is worth writing down once and reusing: scale the consumer fleet, shed or defer low-priority producers, raise the retry backoff to relieve a struggling dependency, and — last — pause the producer entirely. Each step has a cost, and the runbook's job is to make that cost explicit so an on-call engineer can choose without needing the original author. Writing runbooks for queue incidents provides a template and worked examples.

Instrumenting the Denominator Correctly

Every burn-rate query divides "jobs that met the objective" by "jobs that should have". Getting the denominator wrong is the most common way an SLO quietly stops describing reality, and it fails in both directions.

Counting only jobs that completed excludes the ones still stuck in the queue — precisely the population that is violating the objective. During a backlog, a completion-based denominator shrinks exactly when the numerator does, and the ratio stays flat while the system burns. The denominator must therefore include enqueued-but-not-yet-started work.

# Recording rules — evaluate once, reuse in every burn-rate alert.
groups:
  - name: queue-slo
    interval: 15s
    rules:
      # Jobs that started within the 30s objective, per second.
      - record: job:queue_time_good:rate5m
        expr: sum by (queue) (rate(job_queue_time_seconds_bucket{le="30"}[5m]))

      # Everything that was enqueued, whether or not it has started yet.
      - record: job:enqueued:rate5m
        expr: sum by (queue) (rate(jobs_enqueued_total[5m]))

      # The SLI itself: good / total, clamped so an idle queue reads as healthy.
      - record: job:queue_time_sli:ratio5m
        expr: |
          job:queue_time_good:rate5m
          / clamp_min(job:enqueued:rate5m, 0.001)

The clamp_min matters at three in the morning: an idle queue divides zero by zero, which evaluates to NaN, and a NaN comparison is false — so a completely stalled consumer with no new arrivals silently disables its own alert. Clamping makes an empty queue read as healthy, which is correct, while the oldest-message-age alert covers the stalled-consumer case that depth and ratio both miss.

The second denominator trap is retries. A job that fails twice and succeeds on the third attempt should count once, not three times, or a retry storm inflates both sides of the ratio and masks the latency it caused. Count attempts in a separate series and keep the SLI keyed on unique job identity.

Connecting Objectives to Capacity

An objective is a promise about latency, and queue latency is governed by utilisation in a way that is worth making explicit before setting a target. As a fleet approaches saturation, queue time does not degrade linearly — it degrades hyperbolically, and the knee is much earlier than intuition suggests.

For a queue with arrival rate λ and service rate μ per worker across c workers, utilisation is ρ = λ / (c μ). Average queue time rises roughly with ρ / (1 − ρ). At 50% utilisation, jobs wait about one service time. At 80% they wait four. At 90% they wait nine, and at 95% nineteen — from the same code, the same brokers, and the same job durations.

def expected_wait(arrival_rate: float, service_seconds: float, workers: int) -> float:
    """Rough M/M/c queue wait. Good enough to size a fleet against an SLO."""
    rho = arrival_rate * service_seconds / workers
    if rho >= 1:
        return float("inf")        # arrivals exceed capacity: the queue grows forever
    return (rho / (1 - rho)) * service_seconds

expected_wait(40, 0.5, 25)     # 1.5s   — 80% utilised
expected_wait(40, 0.5, 22)     # 4.4s   — 91% utilised
expected_wait(40, 0.5, 21)     # 9.5s   — 95% utilised, same workload

Two operational conclusions follow. First, an objective of "start within 30 seconds" implies a utilisation ceiling, and that ceiling belongs in the autoscaler's target rather than in a quarterly capacity review. Second, the last few percent of utilisation are extraordinarily expensive in latency terms and nearly free in cost terms — which is why the honest answer to "can we run this fleet hotter?" is usually no. The scaling policies in horizontal worker scaling are where the ceiling gets enforced.

Failure Modes & Recovery

Alert fatigue from depth thresholds. Symptom: the queue-depth alert fires several times a week and is routinely acknowledged without action. Cause: alerting on a symptom that varies with normal traffic. Fix: replace with burn-rate and age-based alerts; keep depth as a dashboard panel.

Objectives with no measured queue time. Symptom: an SLO dashboard that only shows execution time. Cause: the producer never stamped enqueued_at. Fix: add the header at enqueue; until then, treat any latency claim as unmeasured.

Burn-rate alerts that never fire. Symptom: a quarter with zero alerts and several user complaints. Cause: a target set from current behaviour rather than user need, so almost nothing violates it. Fix: derive the threshold from the product requirement and re-check it after each incident.

Multi-window queries too expensive to evaluate. Symptom: Prometheus rule evaluation lag, missed alerts. Cause: nested rate() over long windows across high-cardinality labels. Fix: precompute with recording rules and drop per-tenant labels from the SLI series.

Objectives measured on a queue nobody owns. Symptom: an alert fires and routes to a rotation that has never seen the queue before. Cause: objectives defined centrally by a platform team for queues owned by product teams. Fix: require a named owner in the objective definition, and route alerts by that field rather than by a catch-all receiver.

Budget consumed by a known-bad tenant. Symptom: one customer's malformed payloads eat the whole budget. Cause: the SLI counts jobs the system cannot possibly succeed at. Fix: exclude permanently-failing classes from the SLI and route them to a dead-letter path, which the DLQ alert covers separately.

Performance Tuning

• Histogram buckets must straddle the objective. A 30-second SLO needs a le="30" bucket; without it the query interpolates and the alert is fiction.
• Keep SLI cardinality low. Queue and task labels only. Per-tenant SLIs belong in a separate, coarser series.
• Use recording rules for every burn-rate numerator and denominator; the alert then evaluates a single division.
• Window length: 28 days for a product-facing objective, 7 days for a fast-moving internal one. Shorter windows make the budget noisy.
• Exercise the alerts, not just the dashboards. A quarterly game day that breaks a staging queue is the only reliable way to discover that a rule stopped firing three months ago.
• Review objectives quarterly against actual incidents and complaints — an SLO never revised is an SLO nobody uses.
• Add a minimum-volume guard to every ratio-based alert. On a quiet queue two slow jobs are 50% of the window, and a page at 04:00 for two jobs is how a rotation learns to mute the rule.
• Alert on the DLQ separately from the latency SLO. They fail independently and demand different responses.

FAQ

What should the SLI for a job queue actually be? Queue time — enqueue to execution start — for anything a user or downstream process is waiting on, plus a completion ratio for correctness. Execution time is a useful engineering metric but a poor SLI, because a backlog leaves it completely unchanged while users wait.

Why burn-rate alerts instead of a simple threshold? Because a threshold on depth or latency fires on normal bursts and stays silent during slow degradation. Burn rate measures consumption of the error budget, which is proportional to user impact, and the multi-window form distinguishes "brief spike" from "still broken".

Should queue depth ever page? Only as a proxy when you have no queue-time instrumentation, and then paired with a duration condition — depth above N sustained for M minutes. As soon as enqueued_at exists, retire the depth page and alert on age and burn rate instead.

Do scheduled and delayed jobs belong in the same objective? No — measure them separately, because their queue time is intentional. A job scheduled for 02:00 has an enqueue-to-start interval of hours by design, and mixing it into the same histogram destroys the SLI. For scheduled work the meaningful indicator is schedule adherence: the difference between the intended run time and the actual start. Instrument that as its own histogram keyed on the schedule's identity, with an objective measured in seconds of drift; the mechanics are covered in scheduled and delayed jobs.

How many objectives should a job system have? One per user-visible class of work, typically two to four in total: interactive, standard, and bulk, plus a completion objective. Per-task objectives multiply maintenance without changing decisions, because the response to a violation is nearly always the same handful of mitigations.

Related

• Observability & Monitoring for Job Queues — the metrics, logs, and traces these objectives are computed from.
• Defining SLOs for job latency — choosing targets and windows, with worked examples.
• Burn-rate alerts for queue backlogs — the recording rules and alert definitions in full.
• Writing runbooks for queue incidents — the mitigation ladder and a reusable template.
• Prometheus Metrics for Workers — instrumenting the underlying counters and histograms.