Defining SLOs for Job Latency

This walkthrough builds one concrete latency objective end to end, applying the model from SLOs & Alerting for Job Queues under Observability & Monitoring for Job Queues.

Problem Statement

Your team is asked for "an SLA on background jobs". The dashboards show execution-time percentiles, which look excellent — p99 of 1.8 seconds — while support tickets say exports arrive an hour late. Nobody can say what the objective should be, because nobody measures the interval that actually hurts: the time a job spends waiting before a worker picks it up. You need an objective that is measurable today, derived from what the product needs rather than from what the system currently does, and validated against history so it is neither unachievable nor meaningless.

Prerequisites

  • Producers stamping enqueued_at in the message envelope; without it queue time cannot be measured at all.
  • A metrics backend with histogram support and at least four weeks of retention, so the objective can be validated against real traffic.
  • A named owner per queue — an objective with no owner is never revised.
  • Agreement on which queues are user-facing; interactive and batch work need different objectives.

Step 1 — Pick the Indicator

Use queue time (enqueue → execution start) for anything a person or downstream system waits on. It isolates the part of the delay that capacity and scheduling control, and it moves immediately when a backlog forms.

from prometheus_client import Histogram

QUEUE_TIME = Histogram(
    "job_queue_time_seconds", "Seconds from enqueue to execution start",
    ["queue"],
    buckets=(0.5, 1, 2, 5, 10, 20, 30, 45, 60, 120, 300, 900),
)

def on_task_start(queue: str, enqueued_at: float) -> None:
    QUEUE_TIME.labels(queue=queue).observe(time.time() - enqueued_at)

Add end-to-end latency (enqueue → finish) only if the product genuinely cares about completion time rather than responsiveness — a report download does, a webhook delivery does not.

Step 2 — Derive the Target from Product Need

Ask what a user notices, not what the system does today. Three questions settle it: what is the user doing while they wait, what do they do if it is late, and what is the cost of that behaviour?

Work class User behaviour on delay Implied target
2FA code, password reset Retries within 30s, files a ticket at 2min 99.9% start < 10s
Checkout receipt Notices at ~1 minute 99% start < 30s
CSV export Tolerates minutes if progress is visible 99% start < 5min
Nightly aggregation Only notices if the morning report is missing 99% complete by 06:00
Latency targets by class of work Four horizontal bars on a logarithmic time axis. Two-factor codes target ten seconds, checkout receipts thirty seconds, CSV exports five minutes, and nightly aggregation a deadline of six in the morning. The chart notes that one objective across all four classes is simultaneously too strict and too loose. One objective cannot cover every queue 2FA code 99.9% start within 10s Checkout receipt 99% start within 30s CSV export 99% start within 5min Nightly aggregation deadline-based: complete by 06:00 1s 1min 10min hours Batch work with a wall-clock deadline is an objective too — just not a percentile one.

Batch work with a hard deadline deserves a deadline objective rather than a percentile: "the aggregation completes before 06:00 on 99% of days" is measurable, alertable, and matches how anyone actually talks about it.

Step 3 — Size the Histogram Buckets Around the Threshold

Burn-rate queries count observations in the bucket at or below the threshold, so a bucket boundary must sit exactly on it. Without one, Prometheus interpolates and the SLI becomes an estimate.

# Objective is 30s → 30 MUST be a boundary. Add resolution either side so the
# dashboard can show how close to the edge the distribution sits.
buckets=(0.5, 1, 2, 5, 10, 20, 30, 45, 60, 120, 300, 900)
#                              ^^ the objective

Keep the bucket count modest — every bucket is a time series per label combination, and a twelve-bucket histogram across eight queues is already ninety-six series before any other labels. Bucket sizing in more depth is covered in choosing histogram buckets for job duration.

Step 4 — Validate Against History Before Committing

Never publish an objective without first checking what it would have done over the last month. Two failure modes are equally bad: a target so loose nothing ever violates it, and one so tight the budget is exhausted in week one.

# What compliance would have been over the last 28 days
sum(increase(job_queue_time_seconds_bucket{queue="urgent",le="30"}[28d]))
  / sum(increase(job_queue_time_seconds_count{queue="urgent"}[28d]))
# → 0.9948  … a 99% target leaves healthy headroom; 99.9% would already be blown

Aim for a target that historical data meets with a little room — roughly 20–50% of the budget left unspent in a normal month. That leaves capacity to absorb one genuine incident without the objective becoming theatre.

Validating a target against a month of history Two lines over a twenty-eight day window. The 99 percent target consumes about half its error budget with one visible incident step, ending the window with headroom. The 99.9 percent target consumes the entire budget within the first week, making the objective unusable. Budget burn under two candidate targets 100% 0 budget used 99% — ends at ~68% used 99.9% — exhausted on day 9 broker incident, day 8 day 0 day 28 Pick the target that survives a normal month with 20–50% of budget unspent.

Step 5 — Write It Down Where It Is Reviewable

# slo/urgent-queue.yaml
name: urgent-job-timeliness
owner: platform-team
queue: urgent
indicator:
  type: queue_time_seconds
  threshold: 30
target: 0.99
window: 28d
rationale: >
  Password resets and 2FA codes. Support data shows users retry at ~45s and
  file tickets at ~2min, so 30s keeps almost all users inside one attempt.
exclusions:
  - permanently-failing payloads routed to the dead-letter queue
  - scheduled jobs, which have an intentional delay (tracked as schedule drift)
review: quarterly

Recording the rationale is what makes the objective revisable a year later by someone who was not in the room. The exclusions block is equally important: without it, the first argument during an incident is about whether the numbers even count.

Step 6 — Publish the Budget

# Remaining error budget as a percentage, for a dashboard single-stat panel
100 * (1 - (
  (1 - (sum(increase(job_queue_time_seconds_bucket{queue="urgent",le="30"}[28d]))
        / sum(increase(job_queue_time_seconds_count{queue="urgent"}[28d]))))
  / 0.01
))

Put that number on the team dashboard next to throughput, and review it weekly. An objective that is only consulted during incidents provides no early warning, which was most of the reason for defining it.

Step 7 — Decide What a Violation Costs

An objective without an agreed consequence changes nothing. Before publishing, write down what happens at each level of budget consumption and who decides. The point is not bureaucracy — it is removing the argument from the middle of an incident, when nobody has the context to have it well.

policy:
  budget_above_50pct:
    posture: normal
    action: ship features; reliability work competes on merit
  budget_25_to_50pct:
    posture: watch
    action: one reliability item enters the next sprint; owner reviews weekly
  budget_below_25pct:
    posture: constrained
    action: capacity and dependency work take priority over new features
  budget_exhausted:
    posture: freeze
    action: >
      No new feature work on this queue until the window rolls forward.
      Requires the queue owner and the product lead to lift early, in writing.

Two details make this survive contact with reality. First, the freeze is scoped to the queue, not the team — a team that owns six queues should not stop shipping because one of them had a bad month. Second, an early lift is allowed but must be recorded, because a policy that can be silently ignored is not a policy, while one that can be explicitly overridden with a written reason produces a useful record of the trade-offs made over a year.

Attribute every significant burn to a cause at the time it happens: a dependency outage, a slow query shipped in a release, a tenant burst the autoscaler could not follow. That attribution list is what turns the budget from a compliance number into a prioritisation tool — after two quarters it will point directly at the two or three changes that would buy back the most reliability, which is rarely what anyone guessed at the start.

A bucket boundary on the objective itself A row of histogram buckets from half a second to fifteen minutes. The boundary at thirty seconds is highlighted because it matches the objective exactly, which lets the compliance query count observations rather than interpolate them. Neighbouring buckets at twenty and forty-five seconds give resolution around the threshold. The objective must be a bucket boundary 0.5s 2s 10s 20s 30s objective 45s 2m 15m With le="30" present, compliance is a count. Without it, it is an interpolation. The neighbouring boundaries show how close to the edge the distribution is sitting. Every bucket costs one series per label combination — keep the count modest.

Verification

  1. The SLI is measured, not inferred. job_queue_time_seconds_count must be non-zero for every queue with an objective, and its rate must track enqueue rate.
  2. The threshold is a bucket boundary. job_queue_time_seconds_bucket{le="30"} must exist; if the query returns nothing, the histogram is misconfigured.
  3. Historical compliance is inside the target with headroom. Re-run the 28-day query; if it returns exactly the target, the target was fitted to the data rather than to user need.
  4. Exclusions actually apply. Send a deliberately poisonous payload and confirm it lands in the DLQ without moving the SLI.

Gotchas & Edge Cases

Clock skew between producer and consumer. Queue time is computed from two clocks. Skew of a few seconds produces negative or absurd observations; a NTP-synced fleet keeps this below noise, and clamping at zero prevents nonsense values from poisoning the histogram.

Scheduled jobs pollute the SLI. A job with a deliberate ETA has an enqueue-to-start interval measured in hours. Exclude them explicitly, or record their drift from the intended time instead.

Retries double-count. An attempt that fails and re-enqueues generates a second queue-time observation. Decide deliberately: measuring first-attempt latency answers "how responsive are we", measuring every attempt answers "how long until work actually starts". Do not accidentally mix the two.

A per-tenant label explodes cardinality. Twelve buckets multiplied by thousands of tenants is millions of series. Keep the SLI at queue granularity and drill into tenants through logs or exemplars.

The objective outlives its rationale. Traffic patterns and product expectations both drift. A quarterly review with the original rationale in front of you is the cheapest way to notice that a target set for a feature that no longer exists is now shaping capacity decisions.

Related