Retrying Only Transient Errors by Exception Type

Classification is the first decision in the retry policy described in Retry Strategies & Backoff, part of Queue Fundamentals & Architecture: everything downstream — the delay curve, the attempt ceiling, the budget — only makes sense once you know which failures deserve another attempt.

Problem Statement

A worker processing webhook deliveries retries every exception five times with backoff. Half of the retry volume comes from payloads that reference deleted accounts, which raise KeyError and will fail identically forever. Those jobs occupy worker slots for twenty minutes each before dead-lettering, delaying real work behind them, and their failures drown the retry metrics so a genuine outage looks like normal noise. You want permanent failures to reach the dead-letter queue on the first attempt, and retries reserved for faults that can actually heal.

Prerequisites

  • A task framework whose retry trigger you can scope by exception type (Celery's autoretry_for, an explicit try/except in BullMQ, sidekiq_options retry: with a custom error filter).
  • A dead-letter destination already wired, so a non-retried failure is quarantined rather than lost.
  • Structured logging that records the exception class on failure — the taxonomy is only maintainable if you can see what is actually being raised.

Step 1 — Split Failures into Three Buckets, Not Two

The usual "retryable / not retryable" split loses an important middle case: failures that are transient but must not be retried on the normal curve, because the dependency has told you when to come back.

Bucket Examples Policy
Transient connection reset, read timeout, 502/503/504, deadlock, OperationalError Retry with exponential backoff and jitter
Throttled HTTP 429, provider quota error, TooManyRequestsException Retry after the server-specified delay; do not spend the normal curve
Permanent validation error, KeyError, 400/401/403/404, decode failure, schema mismatch No retry — dead-letter immediately

The permanent bucket should be the default for anything unrecognised. Fail-fast on an unknown exception surfaces it in the DLQ within seconds, where a human sees it and adds a rule. Fail-slow buries it under twenty minutes of pointless retries.

Three-bucket exception taxonomy and its routing An incoming exception is matched against three buckets. Transient network and timeout errors go to the exponential backoff path. Throttling errors go to a delayed path that honours the Retry-After header. Permanent errors such as validation and decode failures, plus anything unrecognised, go straight to the dead-letter queue. One exception, three destinations Exception raised matched by class Transient reset · timeout · 503 · deadlock Throttled 429 · quota exceeded Permanent + unknown 400 · 404 · decode · KeyError Backoff + jitter spends the budget Honour Retry-After one long wait Dead-letter now visible in seconds Default-deny: an exception you have not classified is treated as permanent.

Step 2 — Write the Taxonomy Once, Import It Everywhere

Scattering except clauses across fifty tasks guarantees drift. Define the buckets in one module and have every task reference them, so adding a newly-discovered transient error is a one-line change that applies fleet-wide.

# retry_policy.py — the single source of truth for what is retryable
import http.client
import socket
import redis
import requests
from sqlalchemy.exc import OperationalError, DBAPIError

TRANSIENT = (
    socket.timeout,
    ConnectionResetError,
    ConnectionRefusedError,
    http.client.RemoteDisconnected,
    requests.exceptions.ConnectionError,
    requests.exceptions.Timeout,
    redis.exceptions.ConnectionError,
    redis.exceptions.TimeoutError,
    OperationalError,          # includes deadlocks and connection drops
)

class Throttled(Exception):
    """Raised by clients when a dependency asks us to slow down."""
    def __init__(self, retry_after: float):
        self.retry_after = retry_after
        super().__init__(f"throttled, retry after {retry_after}s")

class PermanentFailure(Exception):
    """Never retried — the message is quarantined on the first raise."""

def classify_http(resp) -> None:
    """Translate an HTTP response into the taxonomy. Status ranges, not exception types."""
    if resp.status_code == 429:
        raise Throttled(float(resp.headers.get("Retry-After", 60)))
    if resp.status_code in (408, 425) or 500 <= resp.status_code < 600:
        if resp.status_code in (501, 505):        # server will never implement it
            raise PermanentFailure(f"unsupported: {resp.status_code}")
        raise ConnectionError(f"transient upstream: {resp.status_code}")
    if 400 <= resp.status_code < 500:
        raise PermanentFailure(f"client error: {resp.status_code} {resp.text[:200]}")

The DBAPIError family deserves care: a deadlock is transient and a constraint violation is permanent, but both can surface as the same wrapper class. Inspect the driver error code rather than the wrapper — 1213 for a MySQL deadlock, 40001 and 40P01 for PostgreSQL serialization failure and deadlock respectively.

Step 3 — Wire It Into the Task

from celery import shared_task
from retry_policy import TRANSIENT, Throttled, PermanentFailure, classify_http
import requests

@shared_task(
    bind=True,
    autoretry_for=TRANSIENT,      # ONLY these auto-retry
    retry_backoff=2, retry_jitter=True, retry_backoff_max=600,
    max_retries=6, acks_late=True,
)
def deliver_webhook(self, endpoint: str, payload: dict):
    try:
        resp = requests.post(endpoint, json=payload, timeout=(3.05, 10))
        classify_http(resp)
    except Throttled as exc:
        # Server-specified wait: bypass the curve, and do not count it as a fault.
        raise self.retry(exc=exc, countdown=min(exc.retry_after, 900), max_retries=3)
    except PermanentFailure:
        # No retry: propagate so the broker routes the message to the DLQ.
        raise

In BullMQ the same shape is a try/catch that decides whether to rethrow or to mark the job as unrecoverable:

const { UnrecoverableError } = require('bullmq');

async function processor(job) {
  const res = await fetch(job.data.endpoint, { method: 'POST', body: JSON.stringify(job.data.payload) });
  if (res.status === 429) {
    const wait = Number(res.headers.get('retry-after') ?? 60) * 1000;
    await job.moveToDelayed(Date.now() + wait, job.token);   // requeue, no attempt spent
    return;
  }
  if (res.status >= 400 && res.status < 500) {
    // UnrecoverableError stops retries immediately and moves the job to `failed`.
    throw new UnrecoverableError(`client error ${res.status}`);
  }
  if (!res.ok) throw new Error(`transient upstream ${res.status}`);  // normal retry path
}

Sidekiq expresses the same policy through sidekiq_retry_in, where returning :kill sends the job straight to the dead set:

class DeliverWebhookJob
  include Sidekiq::Job
  sidekiq_options retry: 6, queue: :webhooks

  sidekiq_retry_in do |count, exception, _jobhash|
    case exception
    when Faraday::ConnectionFailed, Faraday::TimeoutError
      (2**count) + rand(0..(2**count))     # transient: backoff with jitter
    when RateLimited
      exception.retry_after                # throttled: obey the server
    else
      :kill                                # permanent: dead set, no further attempts
    end
  end
end

Step 4 — Make Unknown Exceptions Loud

Default-deny only works if you notice what you denied. Count failures by exception class and alert when an unclassified class appears at volume — that is the signal to extend the taxonomy.

from celery.signals import task_failure
from prometheus_client import Counter
from retry_policy import TRANSIENT

FAILURES = Counter("task_failures_total", "Failures by class and bucket",
                   ["task", "exception", "bucket"])

@task_failure.connect
def record(sender=None, exception=None, **kwargs):
    bucket = ("transient" if isinstance(exception, TRANSIENT)
              else "throttled" if type(exception).__name__ == "Throttled"
              else "permanent")
    FAILURES.labels(sender.name, type(exception).__name__, bucket).inc()
# An unclassified exception class appearing at volume — extend the taxonomy
topk(5, sum by (exception) (rate(task_failures_total{bucket="permanent"}[30m])))
Failure mix by exception class after classification Horizontal bars show failure rates per exception class. Connection timeouts and resets are classified transient, rate-limit errors throttled, and validation and key errors permanent. One unclassified class, SchemaMismatch, is highlighted as a candidate to add to the taxonomy. Where the failures actually come from ReadTimeout transient · 42/min · retried ConnectionReset transient · 25/min · retried Throttled throttled · 17/min · Retry-After honoured ValidationError permanent · 12/min · dead-lettered SchemaMismatch unclassified · 21/min · classify me 0 50/min

Step 5 — Test the Taxonomy, Not Just the Happy Path

Classification rules rot silently because nothing exercises them until production does. A small table-driven test pins the behaviour of every class you have decided about, so a dependency upgrade that renames an exception fails CI instead of filling the retry queue.

import pytest
from retry_policy import TRANSIENT, PermanentFailure, Throttled, classify_http

class FakeResponse:
    def __init__(self, status, headers=None, text=""):
        self.status_code, self.headers, self.text = status, headers or {}, text

@pytest.mark.parametrize("status,expected", [
    (200, None),
    (429, Throttled),
    (503, ConnectionError),      # transient bucket
    (504, ConnectionError),
    (400, PermanentFailure),
    (404, PermanentFailure),
    (501, PermanentFailure),     # server will never implement it
])
def test_http_classification(status, expected):
    resp = FakeResponse(status, {"Retry-After": "30"})
    if expected is None:
        classify_http(resp)      # must not raise
    else:
        with pytest.raises(expected):
            classify_http(resp)

def test_every_transient_entry_is_an_exception_class():
    # Catches a typo that would make autoretry_for silently match nothing.
    assert all(isinstance(e, type) and issubclass(e, BaseException) for e in TRANSIENT)

The last assertion is worth more than it looks. A tuple entry that is a string or a module — an easy mistake when the taxonomy is edited under incident pressure — makes autoretry_for match nothing at all, quietly converting every transient failure into an immediate dead-letter. The test costs one line and catches a class of outage that is otherwise diagnosed at three in the morning.

Wrapper exceptions hide the classification you need A single DBAPIError wrapper class arrives at the classifier carrying two very different underlying causes: a deadlock, which is transient and should retry, and a unique-constraint violation, which is permanent and should dead-letter. Matching on the wrapper class treats both identically; matching on the vendor error code inside it separates them correctly. Match the error code, not the wrapper class DBAPIError one class, two meanings 40P01 · deadlock detected transient — will succeed on retry 23505 · unique violation permanent — fails identically forever backoff + retry dead-letter now Listing DBAPIError as transient sends the bottom path into the retry loop forever.

Verification

  1. Permanent failures die on attempt one. Enqueue a job with a payload that raises PermanentFailure and confirm the DLQ receives it within one visibility cycle, with retries = 0 in the message metadata.
  2. Transient failures use the full curve. Point a task at an unreachable host; the log must show attempts growing with the backoff curve up to the ceiling.
  3. Throttling does not consume the normal budget. Return a 429 with Retry-After: 120 and check that exactly one delayed retry is scheduled at ~120s, not a 2s backoff step.
  4. No silent catch-alls remain. grep -rn "except Exception" tasks/ should return only handlers that re-raise or explicitly classify.

Gotchas & Edge Cases

A broad autoretry_for=(Exception,) defeats everything. It is the framework default in many templates and silently overrides a carefully written taxonomy. Audit for it first.

Retryable wrappers hide permanent causes. ORM and HTTP libraries wrap driver errors, so an unretryable constraint violation can arrive as a "retryable" DBAPIError. Match on the vendor error code, not the wrapper class.

Timeouts are ambiguous, not transient. A client timeout on a non-idempotent request may mean the work succeeded and the response was lost. Retry it only behind an idempotency key, as covered in preventing duplicate job execution with idempotency.

429 without Retry-After needs a floor. Fall back to a generous fixed delay (30–60s) rather than the normal curve, or you will keep pushing on a closed window and extend the throttle.

Exception classes change with dependency upgrades. A library that renamed or re-parented an error class quietly moves those failures into the "unknown → permanent" bucket. The unclassified-exception alert from step 4 is what catches that on the day of the upgrade rather than a quarter later.

Related