Retry Strategies & Backoff

Retries are the cheapest reliability mechanism a job system has and the easiest one to turn into an outage, which is why they belong in the same design conversation as the delivery guarantees described in Queue Fundamentals & Architecture. A retry policy is really four decisions bundled together — which failures deserve another attempt, how long to wait before it, how many attempts to allow, and where the message goes when the attempts run out — and getting any one of them wrong produces a distinctive production failure mode.

The default policy shipped by most frameworks is "retry everything three times, immediately." That is close to the worst possible choice. Immediate retries hammer a dependency that is already failing, uniform delays synchronise every worker into a thundering herd, and retrying non-retryable errors burns the attempt budget that a genuinely transient failure needed. This guide covers the mechanics that fix each of those, with runnable configuration for Celery, BullMQ, Sidekiq, and SQS.

Why Naive Retries Amplify Failures

A failing downstream dependency does not fail in isolation. When a database starts rejecting connections, every in-flight job fails at roughly the same moment, and every one of those jobs retries at roughly the same moment. The retry traffic arrives as a spike on top of the original load, which extends the outage that caused the retries — a feedback loop usually called a retry storm.

The arithmetic is unforgiving. A fleet processing 500 jobs per second against a degraded dependency, with three immediate retries, generates 2,000 requests per second at the dependency: the original attempt plus three retries, all inside a few hundred milliseconds. The dependency now has four times its normal load precisely when it has the least capacity to serve it. Recovery becomes impossible until something sheds load, and the only thing that can shed load is the retry policy itself.

The second amplifier is synchronisation. Even with a delay, a fixed one is worse than useless at scale: every job that failed at t=0 retries at exactly t=30s, so you have converted one spike into a series of identical spikes. Randomising the delay — jitter — is what actually spreads the load, and it matters more than the shape of the backoff curve. A retry policy without jitter is a scheduled outage generator.

The third amplifier is the interaction with at-least-once delivery semantics. Broker-level redelivery and application-level retries are two independent retry loops. If a worker retries a task five times internally, then crashes without acknowledging, the broker redelivers the message and the internal loop starts over. Total attempts become the product of the two ceilings, not the sum. Any retry design must decide which of the two layers owns the attempt budget.

Classifying Failures Before Retrying Them

Not every exception deserves a second attempt. A connection reset, a 503, a lock timeout, a throttling response — these are transient: the same input will probably succeed later. A validation error, a 404 on a deleted resource, a JSON decode failure, a permissions denial — these are permanent: the same input will fail identically forever, and retrying it wastes the attempt budget and delays the message reaching a dead-letter queue where a human can see it.

The safe default is a deny-list of retryable exceptions rather than an allow-list of fatal ones. If an error type is not explicitly known to be transient, dead-letter it. Getting an unfamiliar error into the DLQ within seconds is a much better outcome than discovering three days later that a code path has been silently retrying and failing 40,000 times an hour. Retrying only transient errors by exception type works through the concrete exception taxonomies for Python, Node, and Ruby stacks.

# celeryconfig / tasks.py — explicit transient set, everything else dead-letters
from celery import Task
from kombu.exceptions import OperationalError
import requests

TRANSIENT = (
    ConnectionError,                    # socket-level failures
    TimeoutError,                       # our own client timeouts
    OperationalError,                   # broker/DB connectivity
    requests.exceptions.ConnectionError,
    requests.exceptions.Timeout,
)

class TransientOnly(Task):
    # autoretry_for is evaluated by Celery's task wrapper: anything not listed
    # propagates immediately and is routed by the broker to the DLQ.
    autoretry_for = TRANSIENT
    retry_backoff = 2          # base delay in seconds, doubled each attempt
    retry_backoff_max = 600    # never wait more than 10 minutes
    retry_jitter = True        # randomise within the window — critical at scale
    max_retries = 5
    acks_late = True           # ack only after success, so crashes redeliver

A 5xx from an HTTP dependency is worth one nuance: 500 is usually transient, but a 501 or 422 never is. Match on status code ranges, not on "it was an HTTPError."

Backoff Curves and Jitter Algorithms

Exponential backoff multiplies the delay by a constant factor after each failure, typically two. The purpose is to give a struggling dependency geometrically more room while keeping the first retry fast enough that a one-off blip resolves in under a second. The three parameters that matter are the base delay, the growth factor, and the ceiling — without a ceiling, attempt eight of a 2× curve waits over four minutes and attempt twelve waits more than an hour.

Jitter randomises each delay so that jobs which failed together do not retry together. The three standard variants behave differently under load:

Full jitter wins in nearly every published simulation of contended retries, and it is what the AWS SDKs use by default. Its one drawback is that a retry can fire almost immediately, which is undesirable when the failure mode is a rate limit with a known reset window — that case wants the delay floor that equal jitter provides, or an explicit Retry-After honour path.

import random

def backoff_delay(attempt: int, base: float = 1.0, cap: float = 600.0,
                  mode: str = "full", previous: float = 0.0) -> float:
    """Delay in seconds before `attempt` (0-indexed). Mirrors AWS's published variants."""
    exp = min(cap, base * (2 ** attempt))
    if mode == "full":
        return random.uniform(0, exp)
    if mode == "equal":
        return exp / 2 + random.uniform(0, exp / 2)
    if mode == "decorrelated":
        return min(cap, random.uniform(base, max(base, previous) * 3))
    return exp  # no jitter

Because the delay must survive a worker crash, it has to be expressed as a broker-level delay rather than a sleep() inside the task. Sleeping holds the worker slot, holds the message invisible, and evaporates on restart. Every mature framework has a native mechanism instead: Celery's Task.retry(countdown=…) re-publishes the message with an ETA, BullMQ stores a delayed state in a Redis sorted set, and SQS uses per-message VisibilityTimeout extensions. Those mechanics are the same ones behind scheduled and delayed jobs.

// BullMQ — backoff is a queue-level default, overridable per job
const { Queue, Worker } = require('bullmq');

const queue = new Queue('payments', {
  defaultJobOptions: {
    attempts: 6,                       // total attempts, not extra retries
    backoff: { type: 'exponential', delay: 1000 },  // 1s, 2s, 4s, 8s, 16s
    removeOnComplete: 1000,            // keep the last 1k for inspection
    removeOnFail: false,               // failed jobs stay for triage
  },
});

// A custom strategy adds jitter, which the built-in exponential type lacks.
new Worker('payments', processor, {
  settings: {
    backoffStrategy: (attemptsMade) => {
      const exp = Math.min(600_000, 1000 * 2 ** attemptsMade);
      return Math.floor(Math.random() * exp);   // full jitter
    },
  },
});

Attempt Budgets and Where the Ceiling Lives

An attempt ceiling is a promise about the worst case: how much compute a single poison message can consume, and how long a genuinely transient failure has to heal. Those two goals conflict, and the resolution is to size the ceiling from the total retry window rather than from a comfortable-looking count. With a 2× curve and a 1s base, six attempts span about a minute; nine attempts span about eight minutes; twelve span over an hour. Pick the window that matches the dependency's realistic recovery time, then derive the count.

The harder question is which layer owns the ceiling. Application-level retries (Celery max_retries, BullMQ attempts) are aware of the exception and can implement classification, but they are invisible to the broker. Broker-level redelivery (SQS maxReceiveCount, RabbitMQ dead-letter exchanges) is crash-proof and drives the DLQ, but it cannot distinguish a validation error from a timeout. The workable pattern is to let the application own classification and delay while the broker owns the hard ceiling and quarantine, with the broker's ceiling set slightly above the application's so it only fires when the application layer is bypassed by a crash.

Attempt budgets should also be shared, not per-message. A per-message ceiling limits the damage one message can do, but says nothing about the fleet: 50,000 messages each retrying five times is still 250,000 requests at a dependency that is down. A retry budget caps retries as a fraction of total traffic — for example, allow retries to be at most 10% of successful calls, measured over a rolling window, and fail fast beyond that. This is the queue-side analogue of a circuit breaker, and it is what stops a dependency outage from becoming a self-inflicted denial of service. Setting retry budgets and max attempts covers the implementation with a Redis-backed counter.

Framework and Broker Comparison

Sidekiq's default curve deserves a specific mention because it surprises people: attempt⁴ + 15 + rand(30) × (attempt + 1) spans about 21 days over 25 attempts. That is an excellent default for background email but far too long for anything a user is waiting on, and it is the reason Sidekiq's dead set fills up so slowly. Override it per worker class with sidekiq_retry_in when the job has a real deadline.

SQS has no application-visible backoff at all: the only lever is the visibility timeout, which you extend to delay the next receive. That makes the visibility timeout do double duty as both a processing lease and a retry delay, and it is why SQS consumers usually implement backoff in the consumer library rather than at the broker.

Failure Modes & Recovery

Retry storms during a dependency outage. Symptom: dependency error rate spikes, then plateaus at a multiple of normal traffic; queue depth climbs while throughput drops. Cause: uniform or absent backoff. Fix: full jitter plus a fleet-wide retry budget. Immediate mitigation during an incident is to raise the base delay via config and restart workers rolling, which drops retry pressure within one backoff window.

Poison messages consuming the whole budget. Symptom: a small set of job IDs accounts for most retries; DLQ stays empty despite constant failures. Cause: a permanent error is inside the retryable set, so it burns all attempts before quarantine. Fix: tighten the exception classification and check that the framework's catch-all is not Exception.

Delay inflation from stacked layers. Symptom: jobs complete hours after enqueue, with logs showing dozens of attempts. Cause: application retries nested inside broker redelivery, multiplying the ceilings. Fix: set the broker ceiling at application ceiling + 1 or 2, and log the broker's receive count alongside the application's attempt number so the two are distinguishable.

Duplicate side effects from redelivery. Symptom: duplicate emails, double charges, repeated webhooks. Cause: retries are inherently at-least-once; a job that half-succeeded before failing repeats its completed steps. Fix: make handlers idempotent with a deduplication key, as described in preventing duplicate job execution with idempotency.

Lost retries after deploy. Symptom: delayed retries vanish when workers restart. Cause: the delay was implemented as an in-process sleep or timer rather than a broker-side scheduled message. Fix: always re-publish with an ETA or delay; never hold a retry in worker memory.

Performance Tuning

• Base delay: 0.5–2s for internal dependencies, 5–10s for third-party APIs with rate limits. Below 500ms the retry usually lands inside the same failure window and is wasted.
• Growth factor: 2 is right almost always. Use 1.5 when you need more attempts inside a short window; use 3 when the dependency has long recovery cycles.
• Ceiling: cap the per-attempt delay at 5–10 minutes for user-facing work, up to an hour for batch. An uncapped curve turns a retry into an abandonment.
• Attempts: 3–5 for classified transient errors; 1–2 for expensive operations such as paid API calls or LLM inference; 8+ only when each attempt is cheap and the dependency is known to flap.
• Retry budget: cap retries at 10–20% of successful throughput per rolling minute. Beyond that, fail fast to the DLQ and let the alert do its job.
• Queue isolation: publish retries to a dedicated queue with its own worker pool when retry volume can starve first-attempt jobs. This uses the same routing mechanics as priority queues and job fairness.
• Observability: export attempt number as a histogram bucket, not just a counter — the shape tells you whether retries are succeeding on attempt two (healthy) or exhausting the budget (broken).

FAQ

Should retries be handled by the application or the broker? Both, with different jobs. The application owns classification and delay because it is the only layer that knows what the exception means. The broker owns the hard ceiling and dead-lettering because it is the only layer that survives a worker crash. Set the broker ceiling one or two above the application ceiling so it fires only when the application loop is bypassed.

How much jitter is enough? Full jitter — a uniform random draw between zero and the current exponential window — is the safe default and the one AWS's own SDKs use. It maximises spread at the cost of occasionally retrying almost immediately. If the failure is a rate limit with a known reset, use equal jitter or honour the Retry-After header instead so no attempt lands inside the closed window.

Do retries break exactly-once processing? Retries make duplicate execution certain rather than merely possible, so they demand idempotent handlers. Use a deduplication key derived from the job's business identity, record it transactionally with the side effect, and treat a key collision as success. Without that, every retry policy is also a duplicate-side-effect policy.

Why did my retries stop happening after a deploy? Almost always because the delay lived in worker memory — a sleep, a timer, or an in-process scheduler — instead of being re-published to the broker with an ETA. Rolling a deployment discards those. Re-publish delayed retries as real messages so they survive restarts and rebalances.

Related

• Queue Fundamentals & Architecture — the delivery semantics and topology this retry model builds on.
• Dead-Letter Queues & Poison-Message Handling — where a message goes when the attempt budget runs out.
• Exponential backoff with jitter in Celery — the concrete Celery implementation, including custom retry_backoff curves.
• Setting retry budgets and max attempts — fleet-wide retry caps with a Redis rolling counter.
• Retrying only transient errors by exception type — exception taxonomies for Python, Node, and Ruby workers.