Setting Retry Budgets and Max Attempts

A per-job attempt ceiling limits what one message can cost; a retry budget limits what the whole fleet can do to a dependency. This guide implements both, extending the policy design in Retry Strategies & Backoff under Queue Fundamentals & Architecture.

Problem Statement

Your search-indexing workers are configured with five attempts and exponential backoff, which behaves perfectly when a single job fails. When the search cluster itself goes down, all 40,000 queued jobs enter their retry curves simultaneously. Backoff spreads them in time, but the total retry volume — 200,000 additional requests — still arrives at a search cluster that is trying to recover, and it keeps arriving for the full length of the retry window. You need a mechanism that notices the failure rate is systemic rather than per-job, and stops spending attempts on a dependency that is clearly down.

Prerequisites

  • A retry policy already in place with exponential backoff and jitter, so per-attempt spacing is solved.
  • Redis (or any shared counter store) reachable from every worker, used here for the rolling budget.
  • Metrics on task success and retry counts, per the practice in Prometheus metrics for workers.
  • A working dead-letter queue, because a budget that trips must send work somewhere visible.

Step 1 — Derive Max Attempts from the Recovery Window

Do not pick an attempt count directly. Pick the length of time you are willing to keep trying, then solve for the count that fits it. With base delay b and factor 2, total elapsed time after n attempts is approximately b × (2ⁿ − 1).

def attempts_for_window(window_s: float, base: float = 2.0, cap: float = 600.0) -> int:
    """Smallest attempt count whose cumulative backoff covers `window_s`."""
    elapsed, attempts = 0.0, 0
    while elapsed < window_s:
        elapsed += min(cap, base * (2 ** attempts))
        attempts += 1
    return attempts

attempts_for_window(60)      # 5  — a one-minute blip
attempts_for_window(900)     # 9  — a 15-minute deploy window
attempts_for_window(14400)   # 30 — a four-hour outage: almost never worth it

The window should come from data, not intuition: take the p95 duration of your dependency's past incidents. Most internal services recover inside 15 minutes, which is eight or nine attempts on a 2s base. Third-party APIs often need longer windows but fewer attempts, because their base delay starts higher.

Attempts required to cover a given recovery window A stepped bar chart. Covering a one-minute recovery window needs five attempts, five minutes needs seven, fifteen minutes needs nine, one hour needs eleven, and four hours needs thirty. The chart annotates that beyond about fifteen minutes each extra attempt buys progressively less coverage. Recovery window → attempts needed (base 2s, factor 2) 30 15 0 attempts 1 min 5 5 min 7 15 min 9 1 hour 11 4 hours 30 use a DLQ and replay Past ~15 minutes, quarantining and replaying beats holding attempts open.

Beyond roughly fifteen minutes, retries stop being the right tool. Quarantining to a dead-letter queue and replaying after the incident is cheaper, more observable, and does not keep messages invisible for hours.

Step 2 — Differentiate Ceilings by Job Cost

A uniform max_retries across every task treats a 5ms cache warm and a $0.40 LLM call identically. Split the ceiling by what an attempt costs.

Job class Cost of an attempt Attempts Rationale
Idempotent internal call negligible 6–9 Cheap to retry; window matters more than count
Third-party API (metered) cents 3–4 Each attempt bills; honour Retry-After
Paid inference / rendering tens of cents 1–2 One retry covers transport blips; more is waste
Non-idempotent side effect correctness risk 1 + dedup key Retry only behind an idempotency guard
Batch reconciliation minutes of CPU 2 Prefer restarting the batch from a checkpoint

Step 3 — Add a Fleet-Wide Retry Budget

The budget answers a different question than the ceiling: what fraction of total traffic is allowed to be retries? The industry-standard shape is a ratio over a rolling window — retries may not exceed, say, 20% of successful attempts in the last 60 seconds. Beyond that, retries are refused and the job is dead-lettered immediately, converting an invisible retry storm into a visible DLQ alert.

# retry_budget.py — token-ratio budget on a Redis rolling window
import time
import redis

r = redis.Redis(host="redis", decode_responses=True)
WINDOW_S = 60
RATIO = 0.2          # retries may be at most 20% of successes
MIN_ALLOWANCE = 10   # always permit a trickle so low-traffic queues can retry

def _bucket(name: str) -> str:
    return f"budget:{name}:{int(time.time() // WINDOW_S)}"

def record_success(name: str) -> None:
    key = _bucket(name) + ":ok"
    pipe = r.pipeline()
    pipe.incr(key)
    pipe.expire(key, WINDOW_S * 2)
    pipe.execute()

def try_spend_retry(name: str) -> bool:
    """True if this retry fits inside the budget; False means dead-letter now."""
    ok = int(r.get(_bucket(name) + ":ok") or 0)
    allowance = max(MIN_ALLOWANCE, int(ok * RATIO))
    used = r.incr(_bucket(name) + ":retry")
    r.expire(_bucket(name) + ":retry", WINDOW_S * 2)
    return used <= allowance

Wire it into the task's failure path. The budget check runs before the delay is computed, so a tripped budget short-circuits the whole curve.

@shared_task(bind=True, max_retries=6, acks_late=True)
def index_document(self, doc_id: str):
    try:
        search.index(doc_id)
        record_success("index_document")
    except (ConnectionError, TimeoutError) as exc:
        if not try_spend_retry("index_document"):
            # Budget exhausted: the dependency is systemically unhealthy.
            # Fail fast so the message reaches the DLQ and the alert fires.
            raise DeadLetter("retry budget exhausted for index_document") from exc
        raise self.retry(exc=exc)
Rolling retry budget gating the retry path Successful jobs increment a success counter in the current sixty-second window, which sets the retry allowance at twenty percent of that count. Each failed job spends one unit from the allowance and is retried. When the allowance is exhausted, subsequent failures bypass the retry path and go straight to the dead-letter queue, raising an alert. Retries spend from a budget that successes refill Successful jobs 1,000 in this 60s window Allowance = 20% × 1,000 200 retries available resets each window Failed job transient exception spend 1 Retry with backoff budget had room allowance exhausted Dead-letter immediately alert fires, dependency gets air A per-job ceiling caps one message; the budget caps the fleet. Ship both.

The MIN_ALLOWANCE floor matters more than it looks. Without it, a queue with zero successes — every job failing — computes an allowance of zero and dead-letters everything instantly, including the first probe that might have discovered the dependency had recovered.

Step 4 — Expose the Budget State

A budget that trips silently is indistinguishable from a bug. Export both the allowance and the spend so a dashboard can show the headroom, and alert on the ratio rather than on either number alone.

# Retry ratio: how much of current throughput is retries
sum(rate(celery_task_retries_total[5m])) by (task)
  / clamp_min(sum(rate(celery_task_succeeded_total[5m])) by (task), 0.01)

# Page when the budget is trapping work: sustained trips, not a single spike
- alert: RetryBudgetExhausted
  expr: sum(rate(retry_budget_denied_total[5m])) by (task) > 0
  for: 10m
  labels: { severity: critical }
  annotations:
    summary: "Retry budget exhausted for {{ $labels.task }} — work is dead-lettering"

Step 5 — Scope Budgets to the Dependency, Not the Task

The unit a budget protects is the thing that can be overwhelmed. Ten task types that all call the same search cluster need one shared budget keyed on search, not ten independent budgets that each allow 20% and collectively allow 200%. Conversely, a single task that fans out to three dependencies needs three budget checks, because the search cluster being down says nothing about the mail provider's health.

DEPENDENCY_OF = {
    "index_document": "search",
    "reindex_tenant": "search",       # same downstream → same budget key
    "send_receipt":   "mail",
    "sync_crm":       "crm",
}

def try_spend_retry_for(task_name: str) -> bool:
    return try_spend_retry(DEPENDENCY_OF.get(task_name, task_name))

Getting this wrong is the most common budget bug, and it is invisible until an incident: each task type stays inside its own limit while the dependency absorbs the sum of all of them. Name budget keys after the system you are trying to protect, review them whenever a new task starts calling an existing dependency, and treat a task with no mapping as its own budget so a missing entry fails safe rather than silently sharing someone else's allowance.

Budget keys must name the system being protected On the left, three task types each hold their own twenty-percent retry budget against the same search cluster, so it can receive sixty percent of throughput as retries. On the right, all three share a single budget keyed on the dependency, so total retry pressure stays within the twenty percent that was intended. Three budgets or one? Per-task budgets — 3 x 20% index_document reindex_tenant bulk_import search cluster 60% retry load every task stays inside its own limit while the dependency absorbs the sum One budget keyed on the dependency index_document reindex_tenant bulk_import budget: "search" 20% total, shared retry pressure on search is bounded regardless of how many task types call it

Verification

  1. Ceiling honours the window. Fail a task deterministically and time the gap between the first failure and the dead-letter. It should land within 10% of the window you solved for in step 1.
  2. Budget trips under systemic failure. Point the dependency at a black hole and drive 1,000 jobs. The retry counter must plateau at roughly RATIO × successes and the DLQ must start filling — not the retry queue.
  3. Budget recovers. Restore the dependency. Within two windows the allowance must rebuild and normal retries resume without a restart.
  4. Low-traffic queues still retry. Send a single failing job to an otherwise idle queue; MIN_ALLOWANCE must let it retry rather than dead-lettering on attempt one.
# Watch allowance versus spend during a game day
watch -n5 'redis-cli --scan --pattern "budget:index_document:*" | \
  xargs -I{} sh -c "printf \"%s \" {}; redis-cli get {}"'

Gotchas & Edge Cases

Budget keys must be time-bucketed, not global. A single counter with no window never resets, so the budget appears exhausted forever after one incident. Bucket by floor(now / window) and set a TTL of two windows.

Per-worker budgets do not work. A budget enforced in worker memory scales with the number of workers, so autoscaling silently multiplies the allowance exactly when you need it constrained. The counter must be shared.

Do not count budget-denied jobs as retries. If a denial increments the retry metric, the ratio alert fires on its own mitigation and the dashboard becomes unreadable. Use a separate retry_budget_denied_total counter.

A tripped budget needs a replay plan. Dead-lettering during an outage is correct, but those messages are real work. Pair the budget with the replay workflow from replaying dead-letter messages in RabbitMQ or the SQS redrive path, and treat the post-incident replay as part of the incident.

max_retries counts retries, not attempts, in Celery. max_retries=5 allows six executions. BullMQ's attempts: 5 allows five. Mixing the two mental models across stacks produces off-by-one budgets that only show up in a postmortem.

Related