Choosing Histogram Buckets for Job Duration

Bucket boundaries decide what your latency data can answer, and the defaults were designed for HTTP requests rather than background jobs. This guide extends Prometheus Metrics for Workers, part of Observability & Monitoring for Job Queues.

Problem Statement

Your worker histogram uses the client library's defaults, which top out at ten seconds. Job durations range from 40 milliseconds to twelve minutes, so 60% of observations land in the +Inf bucket and every quantile query returns a number that is mathematically meaningless. When someone asks "did the p99 regress after the deploy?", nobody can answer, and the SLO built on that metric is fiction.

Prerequisites

  • A histogram already exporting job duration; this is about its boundaries, not its existence.
  • Actual duration distribution — p50, p90, p99, and max — from logs or a temporary summary.
  • The SLO thresholds those durations will be measured against, per defining SLOs for job latency.
  • A cardinality budget: buckets multiply with every label value.

Step 1 — Understand What a Bucket Actually Stores

A Prometheus histogram is a set of cumulative counters: le="1" counts every observation ≤ 1s, le="5" counts every observation ≤ 5s, and so on. Quantiles are interpolated linearly within the bucket that contains them, so precision depends entirely on how narrow that bucket is.

Bucket width determines quantile accuracy With boundaries at one and ten seconds, a p99 falling in that bucket can be anywhere between one and ten seconds, and linear interpolation reports a single value with no basis. Adding boundaries at two, three, five and seven seconds narrows the containing bucket so the reported p99 is close to the true value. Quantiles are interpolated inside the containing bucket coarse: le = 1, 10 ≤1s 1s – 10s · true p99 could be anywhere in here true p99 = 6.4s reported p99 = 5.5s — off by 14%, and the error is invisible refined: le = 1, 2, 3, 5, 7, 10 ≤1s 1–2 2–3 3–5 5–7 7–10 reported p99 = 6.3s — close enough to act on Precision costs one time series per boundary, per label combination.

Two consequences follow. Anything above the highest boundary lands in +Inf, where no interpolation is possible and histogram_quantile simply returns the last finite boundary — silently under-reporting. And any threshold you want to alert on must be a boundary, or the count of "jobs under 30 seconds" is an estimate rather than a fact.

Step 2 — Put a Boundary on Every Threshold You Care About

Start from the questions rather than from a mathematical layout. Every SLO threshold, every timeout, and every "is this job slow?" number becomes a boundary.

from prometheus_client import Histogram

JOB_DURATION = Histogram(
    "job_duration_seconds", "Execution time per job",
    ["queue", "task"],
    buckets=(
        0.05, 0.1, 0.25, 0.5,     # fast path resolution
        1, 2, 5,
        10,                       # ← soft time limit
        30,                       # ← SLO threshold
        60, 120,
        300,                      # ← hard time limit
        600, 900,                 # long tail before +Inf
    ),
)

Three of those boundaries exist for operational reasons, not statistical ones: the soft limit, the SLO threshold, and the hard limit. Being able to query "how many jobs are approaching the timeout" without interpolation is worth a series each.

Step 3 — Cover the Whole Range with an Exponential Layout

Between the meaningful boundaries, an exponential progression gives roughly constant relative precision, which is what you want when durations span orders of magnitude.

def exponential_buckets(start: float, factor: float, count: int) -> tuple:
    return tuple(round(start * factor ** i, 4) for i in range(count))

exponential_buckets(0.05, 2, 12)
# (0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8, 25.6, 51.2, 102.4)

def merged(*groups) -> tuple:
    """Merge an exponential spine with the boundaries you must have."""
    return tuple(sorted(set(v for g in groups for v in g)))

BUCKETS = merged(exponential_buckets(0.05, 2, 12), (10, 30, 300))

A factor of 2 gives about 14 buckets across three orders of magnitude, which is a reasonable default. Use 1.5 when you need tighter quantiles in a narrow range, and 3 when you mostly care about order of magnitude and want to save series.

Step 4 — Budget the Cardinality

Every bucket is a time series per label combination, and the multiplication is unforgiving.

def series_count(buckets: int, queues: int, tasks: int, instances: int) -> int:
    # +2 for _sum and _count, +1 for the implicit +Inf bucket
    return (buckets + 3) * queues * tasks * instances

series_count(14, queues=6, tasks=40, instances=20)   # 81,600 series
series_count(14, queues=6, tasks=1,  instances=20)   # 2,040 — drop the task label
series_count(8,  queues=6, tasks=40, instances=20)   # 52,800 — fewer buckets

Eighty thousand series from one metric is a real cost in a shared Prometheus. The usual resolution is to drop the task label from the histogram and keep it on a plain counter — you lose per-task quantiles and keep per-task rates, which is the right trade for most fleets. Never put job ID, tenant, or any unbounded value on a histogram.

Step 5 — Use Different Buckets for Different Queues

One bucket layout across a fleet whose durations differ by two orders of magnitude serves neither end well. Define layouts per workload class and pick by queue.

Workload Typical p50 Layout
Webhooks, cache warm 50ms 0.01 … 5 — twelve buckets, factor 1.8
Standard API-bound jobs 400ms 0.05 … 60 — the general default
Media, exports 30s 1 … 1800 — factor 2 from one second
Nightly batch 10min 60 … 14400 — coarse; only the tail matters 
LAYOUTS = {
    "webhooks": exponential_buckets(0.01, 1.8, 12),
    "default":  merged(exponential_buckets(0.05, 2, 12), (10, 30, 300)),
    "media":    merged(exponential_buckets(1, 2, 11), (300, 900)),
    "batch":    (60, 300, 900, 1800, 3600, 7200, 14400),
}

def histogram_for(queue: str) -> Histogram:
    return Histogram(f"job_duration_seconds_{queue}", "Execution time",
                     ["task"], buckets=LAYOUTS.get(queue, LAYOUTS["default"]))

Step 6 — Validate Against Real Data

What a well-fitted bucket layout looks like Two bar charts of observation counts per bucket. In the healthy layout observations are spread across the middle buckets with a small tail, and the plus-infinity bucket is nearly empty. In the degenerate layout most observations land in the plus-infinity bucket, which makes every quantile query meaningless. Check where the observations actually land well fitted +Inf ≈ 0.2% — quantiles are trustworthy defaults, wide spread +Inf ≈ 60% — every quantile query is fiction Rule of thumb: no bucket should hold more than ~30% of observations, and +Inf under 1%. 
# Fraction of observations above the highest finite boundary — should be < 1%
1 - (
  sum(rate(job_duration_seconds_bucket{le="900"}[1h]))
  / sum(rate(job_duration_seconds_count[1h]))
)

# Per-bucket share, to spot a boundary that swallows everything
sum by (le) (increase(job_duration_seconds_bucket[1h]))

Step 7 — Know When a Histogram Is the Wrong Tool

Histograms are aggregatable across instances, which is why they are the default for a fleet. They are not always the right choice.

A summary computes quantiles client-side and stores them directly. That gives exact quantiles per instance with no bucket design at all — but summary quantiles cannot be aggregated across instances, so the fleet-wide p99 is unobtainable. Use one when you have a single instance, or when the per-instance number is genuinely what you want.

Native histograms (Prometheus 2.40+, stable from 3.0) store an exponential bucket layout automatically with configurable resolution and a fraction of the series cost. They remove the design problem entirely, and where your Prometheus and client libraries both support them they are the better answer — a single job_duration_seconds series replaces the fifteen the classic layout needs. The caveats are ecosystem support in dashboards and long-term storage, so check the whole path before migrating.

For anything where you need the individual slow job rather than the distribution — "which export took twelve minutes?" — no histogram helps. That is a trace or a log with a duration field, and pairing the histogram with exemplars is what links the two: the quantile tells you the distribution moved, and the exemplar points at one request that proves it. That pairing is covered in distributed tracing for async jobs.

Step 8 — Instrument Queue Time with Its Own Layout

Job duration and queue time are different distributions and deserve different boundaries, even though teams routinely give them the same ones. Duration is bounded by the work; queue time is bounded by capacity, and during a backlog it can be orders of magnitude larger than any job's runtime.

QUEUE_TIME = Histogram(
    "job_queue_time_seconds", "Enqueue to execution start",
    ["queue"],
    buckets=(
        0.1, 0.5, 1, 2, 5,        # healthy: sub-second to a few seconds
        10, 30,                   # ← the SLO threshold lives here
        60, 300, 900,             # degraded
        3600, 14400,              # backlog territory — you want resolution here
    ),
)

The top of this layout matters more than the top of the duration layout. A duration histogram that saturates at 900 seconds tells you a job is slow, which you probably already knew. A queue-time histogram that saturates at 900 seconds during a four-hour backlog tells you nothing at all about how bad the backlog was, at exactly the moment you need to describe it in an incident review. Extend the top boundaries well past what "normal" requires — the buckets cost almost nothing when empty, and they are the only record you will have of how far outside normal an incident went.

Where histogram cardinality comes from A multiplication chain. Fourteen buckets plus three extra series, multiplied by six queues, forty task names and twenty instances, yields over eighty thousand time series from a single metric. Dropping the task label reduces it to about two thousand while keeping per-queue quantiles. One metric, eighty thousand series 17 buckets +sum,count x 6 queues x 40 task names x 20 instances = 81,600 series Drop the task label from the histogram and keep it on a counter: 2,040 series, same alerts. You lose per-task quantiles and keep per-task rates — the right trade for most fleets. Never put job id, tenant, or any unbounded value on a histogram.

Verification

  1. +Inf holds under 1% of observations. If it is higher, the top boundary is too low and quantiles are under-reported.
  2. No single bucket dominates. A bucket holding more than about 30% of observations means that region needs finer boundaries.
  3. SLO thresholds are exact boundaries. job_duration_seconds_bucket{le="30"} must exist and return data for a 30-second objective.
  4. Cardinality is within budget. count({__name__=~"job_duration_seconds_bucket"}) should match your calculation, not exceed it by an order of magnitude.

Gotchas & Edge Cases

Changing buckets breaks historical comparison. Old and new series have different le sets, so a quantile query spanning the change mixes incompatible data. Change buckets at a natural boundary and note it as a dashboard annotation.

histogram_quantile above the top boundary returns that boundary. It does not return +Inf and does not error — it silently reports a number that looks plausible. This is the failure mode that makes an under-sized top boundary so dangerous.

Buckets must be sorted and unique. Client libraries mostly enforce this, but a merged layout assembled from several sources can produce duplicates that are silently dropped.

Per-instance quantiles are not fleet quantiles. histogram_quantile(0.99, rate(...)) without a sum by (le) computes per-instance quantiles, which is almost never what you meant.

Retry attempts inflate the distribution. If the histogram records every attempt, a retry storm shifts the apparent p99 even though no single job got slower. Record attempts separately, as in retry strategies & backoff.

Related