Exponential Backoff with Jitter in Celery

This walkthrough implements the backoff model described in Retry Strategies & Backoff, part of Queue Fundamentals & Architecture, using Celery's own retry machinery rather than a hand-rolled loop.

Problem Statement

A Celery task that calls a payment provider fails for ninety seconds during the provider's deploy. Every one of the 3,000 tasks in flight raises ConnectionError, and every one retries immediately three times, so the provider receives 12,000 requests in under two seconds while it is already degraded. When the provider recovers, the same pattern repeats on the next blip. You want each task to wait progressively longer between attempts, with the waits randomised so the fleet never retries in lockstep, and you want the delay to survive a worker restart.

Prerequisites

  • Celery 5.3+ with a Redis or RabbitMQ broker (retry_backoff and retry_jitter are task options added in 4.2 and stable since).
  • acks_late = True on the tasks in question, so an unacknowledged message is redelivered if the worker dies mid-retry.
  • A broker that supports delayed delivery — Redis via countdown/ETA, or RabbitMQ with a delayed-message exchange or per-queue TTL routing.
  • Worker logging at INFO so Task retry lines with the computed countdown are visible.

Step 1 — Turn On Backoff and Jitter Declaratively

The shortest correct configuration is three task options. retry_backoff=2 sets the base delay in seconds and doubles it per attempt; retry_backoff_max clamps the curve; retry_jitter=True converts each computed delay into a uniform random draw between zero and that delay — full jitter.

# tasks.py
from celery import shared_task
import requests

@shared_task(
    bind=True,
    autoretry_for=(requests.exceptions.ConnectionError,
                   requests.exceptions.Timeout),
    retry_backoff=2,          # 2s, 4s, 8s, 16s, 32s … before jitter
    retry_backoff_max=600,    # clamp any single wait at 10 minutes
    retry_jitter=True,        # uniform draw in [0, computed] — full jitter
    max_retries=5,
    acks_late=True,
)
def capture_payment(self, charge_id: str) -> dict:
    resp = requests.post(
        f"https://payments.internal/v1/charges/{charge_id}/capture",
        timeout=(3.05, 10),   # connect, read — always bound both
    )
    resp.raise_for_status()
    return resp.json()

With retry_jitter=True, attempt three does not wait exactly 8 seconds: it waits somewhere in [0, 8). Across 3,000 tasks that spreads the third attempt evenly across an 8-second window instead of concentrating it in one millisecond.

Celery exponential windows and the jittered delay drawn inside each Four horizontal bars represent the exponential windows for attempts one to four: two, four, eight and sixteen seconds. Inside each bar a marker shows where the randomly drawn actual delay falls, illustrating that full jitter spreads attempts across the whole window rather than firing at its end. retry_backoff=2 with retry_jitter=True bar = exponential window · marker = the delay Celery actually drew attempt 1 window 0–2s · drew 0.7s attempt 2 window 0–4s · drew 2.9s attempt 3 window 0–8s · drew 1.6s attempt 4 window 0–16s · drew 12s 0s 8s 16s Without jitter every task fires at the right edge of its window — one spike per attempt.

Step 2 — Understand What Celery Computes

retry_backoff is not simply "delay = base × 2ⁿ". Celery computes base * (2 ** retries) where retries starts at zero for the first retry, clamps the result to retry_backoff_max, and — when retry_jitter is on — passes it through random.randrange(0, computed). Two consequences matter in production. First, a jittered delay can be 0, so the first retry may be effectively immediate. Second, the clamp applies before the jitter, so with a 600s ceiling every attempt past the seventh draws uniformly from [0, 600), which is a very wide spread — usually what you want during a long outage.

# What Celery does internally, reproduced so you can reason about the numbers
import random

def celery_countdown(retries, backoff=2, backoff_max=600, jitter=True):
    countdown = backoff * (2 ** retries)
    countdown = min(countdown, backoff_max)
    return random.randrange(countdown) if jitter else countdown

for attempt in range(6):
    print(attempt, celery_countdown(attempt))
# 0 1      ← may be near-zero: jitter draws from [0, 2)
# 1 3
# 2 5
# 3 11
# 4 27
# 5 44

If a near-zero first retry is unacceptable — a rate-limited third-party API, for instance — switch to equal jitter with an explicit countdown, covered in the next step.

Step 3 — Take Manual Control with self.retry

Declarative options cover the common case. When you need a delay floor, a non-2× growth factor, or a delay taken from the response (Retry-After), call self.retry() explicitly and compute the countdown yourself.

import random
from celery import shared_task
from celery.exceptions import Retry
import requests

BASE, CAP = 5.0, 900.0

def equal_jitter(attempt: int) -> float:
    """Half the exponential window, plus a random draw over the other half:
    guarantees a floor while still breaking synchronisation."""
    window = min(CAP, BASE * (2 ** attempt))
    return window / 2 + random.uniform(0, window / 2)

@shared_task(bind=True, max_retries=6, acks_late=True)
def sync_crm_contact(self, contact_id: str):
    try:
        resp = requests.post("https://crm.example.com/v2/contacts",
                             json={"id": contact_id}, timeout=(3.05, 15))
        if resp.status_code == 429:
            # Provider told us exactly when to come back — obey it, and never
            # let the header push us past the task's own deadline.
            wait = min(float(resp.headers.get("Retry-After", 60)), CAP)
            raise self.retry(countdown=wait)
        resp.raise_for_status()
    except Retry:
        raise                                  # re-raise Celery's control-flow exception
    except (requests.ConnectionError, requests.Timeout) as exc:
        raise self.retry(exc=exc, countdown=equal_jitter(self.request.retries))

The except Retry: raise clause is the detail that bites people: self.retry() works by raising Retry, so a broad except Exception below it will swallow the retry and mark the task as failed.

Step 4 — Keep the Delay in the Broker, Not the Worker

Celery implements countdown by re-publishing the message with an ETA. The worker that raised the retry acknowledges nothing and moves on; a future worker picks the message up when the ETA passes. This is what makes the delay survive a restart — and it is also why a time.sleep() inside a task is never an acceptable substitute. Sleeping pins the worker's concurrency slot, keeps the message unacknowledged, extends the visibility timeout pressure, and loses the retry entirely if the process is replaced during a deploy.

One broker-specific caveat: with RabbitMQ, ETA messages are held in the worker's memory once prefetched, so a long countdown combined with a high prefetch can hold thousands of messages hostage on one node. Either keep worker_prefetch_multiplier = 1 for queues with long retries, or route retries through a delayed-message exchange so RabbitMQ holds them instead.

# celeryconfig.py — safe defaults for queues with long retry curves
worker_prefetch_multiplier = 1     # do not hoard ETA messages in worker RAM
task_acks_late = True              # redeliver if the worker dies
broker_transport_options = {
    "visibility_timeout": 3600,    # Redis: must exceed your longest countdown
}
task_reject_on_worker_lost = True  # requeue rather than silently dropping

The Redis visibility_timeout line is the single most common misconfiguration in Celery retry setups. If your maximum countdown exceeds it, Redis makes the message visible again while the delay is still pending, and the task runs twice.

How a Celery retry travels back through the broker Three vertical lifelines: worker A, the broker, and worker B. Worker A receives the task and fails, then re-publishes the message with an ETA to the broker instead of sleeping. The broker holds the message for the countdown period, then delivers it to worker B, which succeeds and acknowledges. A retry is a new message, not a paused worker Worker A Broker Worker B deliver capture_payment ConnectionError re-publish with eta = now + 2.9s held 2.9s deliver to whichever worker is free succeeds → ack

Step 5 — Instrument the Retry Distribution

A retry policy you cannot observe is a guess. Export attempt count as a histogram and the computed countdown as a summary, then confirm the shape matches the curve you configured. The signal you want is most retries succeeding on attempt one or two, with a thin tail reaching the ceiling.

# celery_signals.py
from celery.signals import task_retry, task_failure
from prometheus_client import Counter, Histogram

RETRIES = Counter("celery_task_retries_total", "Retries by task and reason",
                  ["task", "exception"])
ATTEMPTS = Histogram("celery_task_attempts", "Attempt number at completion",
                     ["task"], buckets=(1, 2, 3, 4, 5, 6, 8, 12))

@task_retry.connect
def on_retry(sender=None, reason=None, **kwargs):
    RETRIES.labels(task=sender.name, exception=type(reason).__name__).inc()

@task_failure.connect
def on_failure(sender=None, **kwargs):
    ATTEMPTS.labels(task=sender.name).observe(sender.request.retries + 1)

Pair that with the metrics practice in Prometheus metrics for workers, and alert when the retry rate exceeds a fixed fraction of task throughput rather than on any absolute count.

Attempt-number histogram for a healthy retry policy Bars showing how many tasks completed on each attempt number. The overwhelming majority succeed on attempt one, a small fraction on attempt two, and a very thin tail extends to attempts three through six. A second annotated shape shows the unhealthy pattern where a large spike sits at the maximum attempt, indicating retries that never succeed. Where tasks finish tells you if the curve works 1 18,422 2 1,088 3 4 5 6 max a spike here means retries never succeed attempt at which the task completed

Verification

Confirm three things before calling it done.

  1. The delays are actually random. Run twenty tasks against a deliberately broken endpoint and grep the worker log: grep 'Retry in' worker.log | head -20. The countdowns must differ from each other within the same attempt number. Identical values mean retry_jitter is off or was overridden by an explicit countdown.
  2. The curve grows and clamps. celery -A proj inspect scheduled shows ETA-held messages; their ETAs should spread further apart on later attempts and never exceed retry_backoff_max from the failure time.
  3. A restart does not lose retries. Enqueue a failing task, wait for its first retry to be scheduled, then restart the worker. The task must still execute after the countdown — proof the delay lives in the broker and not in worker memory.
# Attempt-number histogram from the exporter — a healthy shape is front-loaded
curl -s localhost:9808/metrics | grep celery_task_attempts_bucket
# celery_task_attempts_bucket{task="capture_payment",le="1"} 18422
# celery_task_attempts_bucket{task="capture_payment",le="2"} 19510
# celery_task_attempts_bucket{task="capture_payment",le="4"} 19604
# celery_task_attempts_bucket{task="capture_payment",le="+Inf"} 19611

Gotchas & Edge Cases

retry_backoff is ignored when you pass countdown. An explicit self.retry(countdown=…) overrides the declarative curve entirely, jitter included. Mixing both in one task is the most common reason "jitter isn't working."

Redis visibility_timeout must exceed the longest countdown. With Celery on Redis, a countdown longer than the transport's visibility timeout causes the message to be redelivered while the retry is still pending — the task runs twice and the duplicate is invisible in the retry metrics. Set visibility_timeout to at least retry_backoff_max plus the task's own runtime.

max_retries=None means infinite, not default. It is easy to type while intending "use the default." Combined with a broad autoretry_for, it produces a task that never dead-letters and quietly consumes a worker slot forever.

Countdown does not compose with expires. If the task was published with expires, a long backoff can push the next attempt past the expiry, and Celery discards the message silently. Either drop expires on retryable tasks or make it comfortably larger than the total retry window.

ETA messages are prefetched into worker RAM on RabbitMQ. A large worker_prefetch_multiplier with long countdowns concentrates delayed work on a single node, so a restart there loses far more than a fair share of the retries. Keep prefetch at 1 for these queues, as covered in tuning prefetch and consumer concurrency.

Related