Handling SIGTERM in Celery Workers

This is the Celery-specific implementation of the drain model in Graceful Shutdown & Worker Deployments, part of Backend Frameworks & Worker Scaling — what each signal does, and how to make a long task stop safely before the grace period expires.

Problem Statement

Every deploy produces a handful of duplicate invoice emails. The workers run under Kubernetes with the default 30-second grace period, and the invoice task takes 40–90 seconds. When the rollout starts, SIGTERM arrives, Celery keeps working, the grace period expires, SIGKILL lands mid-task, and the broker redelivers the unacknowledged message to a new worker that starts the task from the beginning — including the part that already sent the email. You want workers to stop accepting new tasks immediately, finish what they hold, and exit before Kubernetes kills them.

Prerequisites

  • Celery 5.3+ running under a supervisor that sends SIGTERM (Kubernetes, systemd, Docker) rather than SIGKILL.
  • task_acks_late = True, otherwise a killed task is lost rather than redelivered — a different and worse failure.
  • Known p99 task durations per queue; the whole design is sized from them.
  • Idempotent handlers, because graceful shutdown reduces redelivery but never eliminates it.

Step 1 — Know What Each Signal Does

Celery's shutdown behaviour is signal-dependent, and the difference between warm and cold is the difference between finishing work and abandoning it.

Signal Celery name Behaviour
SIGTERM (first) warm shutdown Stop consuming new messages, finish running tasks, then exit
SIGTERM (second) cold shutdown Terminate running tasks immediately, ack nothing
SIGQUIT cold shutdown Immediate termination, same as a second TERM
SIGINT (Ctrl-C) warm, then cold on repeat Interactive equivalent of the above
SIGUSR1 — Dump a traceback for every worker thread; invaluable for diagnosing a stuck drain
Celery warm and cold shutdown states A running worker receives the first SIGTERM and enters warm shutdown, where it stops consuming but keeps executing in-flight tasks. From there it either exits cleanly when those tasks acknowledge, or drops into cold shutdown if a second signal arrives or the grace period expires, in which case tasks are terminated without acknowledgement and the broker redelivers them. One signal finishes the work; two abandon it running consuming + executing SIGTERM warm shutdown no new fetches in-flight tasks continue tasks ack clean exit no redelivery 2nd signal / SIGKILL cold shutdown tasks killed → redelivered Supervisors that "restart harder" by re-sending TERM convert every warm shutdown into a cold one. SIGUSR1 dumps stack traces — use it when a drain never finishes.

Celery's warm shutdown waits indefinitely by default. That is safe in isolation and dangerous under an orchestrator, because the orchestrator's grace period, not Celery, decides when the process dies.

Step 2 — Bound Task Runtime So the Drain Can Finish

Give tasks a soft time limit below the drain budget. A soft limit raises SoftTimeLimitExceeded inside the task, which you can catch to checkpoint and re-enqueue; a hard limit kills the child process.

# celeryconfig.py
task_acks_late = True
task_reject_on_worker_lost = True    # requeue instead of dropping on SIGKILL
worker_prefetch_multiplier = 1       # fewer held messages to release on shutdown

task_soft_time_limit = 55            # raise inside the task at 55s
task_time_limit = 75                 # hard kill at 75s, still inside a 120s grace
task_annotations = {
    "app.exports.generate_report": {"soft_time_limit": 600, "time_limit": 660},
}

The invariant to preserve: task_time_limit < drain budget < terminationGracePeriodSeconds. When a task class genuinely needs more than the grace period, it goes on its own queue and its own deployment with a larger budget, rather than raising the limit for everything.

Step 3 — Catch the Shutdown and Checkpoint

Celery emits worker_shutting_down when the warm shutdown begins. Combine it with a module-level flag and a checkpointed loop so long tasks can stop at a safe point.

from celery.signals import worker_shutting_down
from celery.exceptions import SoftTimeLimitExceeded
import redis

r = redis.Redis(decode_responses=True)
SHUTTING_DOWN = False

@worker_shutting_down.connect
def on_shutdown(sig, how, exitcode, **kwargs):
    global SHUTTING_DOWN
    SHUTTING_DOWN = True
    # `how` is "Warm" or "Cold" — log it so postmortems can tell them apart.
    print(f"worker shutting down: how={how} signal={sig}")

@shared_task(bind=True, acks_late=True, soft_time_limit=600)
def generate_report(self, report_id: str, total_rows: int):
    key = f"report:{report_id}:cursor"
    cursor = int(r.get(key) or 0)
    try:
        while cursor < total_rows:
            cursor = write_chunk(report_id, cursor, size=500)
            r.set(key, cursor, ex=86_400)      # commit progress after the write
            if SHUTTING_DOWN:
                # Yield the worker slot; a fresh worker resumes from the cursor.
                raise self.retry(countdown=5, max_retries=None)
    except SoftTimeLimitExceeded:
        raise self.retry(countdown=5, max_retries=None)
    r.delete(key)
    return {"report_id": report_id, "rows": cursor}

self.retry() re-publishes the message, so the resumed task is a normal delivery rather than a broker-driven redelivery — meaning it does not consume the redelivery budget or trip dead-letter thresholds. The cursor makes the resume cheap: only the current chunk is repeated.

Checkpointed task resuming on a different worker Worker A processes chunks and commits a cursor after each one. When SIGTERM sets the shutting-down flag, the task re-enqueues itself and worker A exits cleanly inside the grace period. Worker B picks up the re-enqueued message, reads the stored cursor, and continues from that chunk rather than starting over. Checkpoint, yield, resume — no repeated work worker A chunk 1 chunk 2 chunk 3 cursor=500 · 1000 · 1500 committed after each write SIGTERM self.retry() re-published with cursor worker B chunk 4 chunk 5 resumes at cursor=1500 Worker A exits in seconds, not in the time the whole report would take. Commit the cursor AFTER the durable write: a crash repeats one chunk, never skips one.

Step 4 — Align the Orchestrator

spec:
  terminationGracePeriodSeconds: 120     # > task_time_limit (75) + exit overhead
  containers:
    - name: worker
      args: ["celery", "-A", "app", "worker", "-Q", "invoices",
             "--concurrency=8", "--prefetch-multiplier=1",
             "--max-tasks-per-child=500"]
      lifecycle:
        preStop:
          exec:
            # Give the readiness/metrics endpoints a moment to deregister before
            # the runtime sends SIGTERM to PID 1.
            command: ["/bin/sh", "-c", "sleep 5"]

--max-tasks-per-child is worth setting on any worker with a memory leak history: it recycles pool children between tasks, which is a graceful restart, whereas an OOM kill is the least graceful shutdown there is.

Step 5 — Log the Shutdown Path

@worker_shutting_down.connect
def emit_shutdown_metric(sig, how, exitcode, **kwargs):
    from prometheus_client import Counter
    Counter("celery_worker_shutdowns_total", "Shutdowns by kind",
            ["how"]).labels(how=str(how)).inc()

A dashboard panel of warm versus cold shutdowns per deploy is the single most useful signal here: cold shutdowns should be zero, and any non-zero value is a grace period that needs raising or a task that needs checkpointing.

Step 6 — Handle the Pool You Actually Run

Signal behaviour differs by execution pool, and a pattern that drains cleanly under prefork can hang forever under gevent.

Pool How tasks run Shutdown behaviour to plan for
prefork (default) Child processes Parent stops fetching; children finish their task. A child blocked in C code ignores the soft time limit, so the hard limit is the real bound.
gevent / eventlet Greenlets in one process Soft time limits work only at yield points; a CPU-bound greenlet never yields and blocks the whole drain.
solo Inline in the main process No concurrency, no separate child to signal — the task must finish or be killed.
threads Thread pool Threads cannot be interrupted; the hard time limit does not apply, so checkpointing is the only stop mechanism. 
# Pool-appropriate limits — the same numbers are not correct everywhere
worker_pool = "prefork"
task_soft_time_limit = 55        # honoured via SIGUSR1 to the child in prefork
task_time_limit = 75             # SIGKILL to the child — prefork only

# For gevent, a hard limit cannot preempt a greenlet, so lean on checkpoints:
# worker_pool = "gevent"
# task_soft_time_limit = 55
# task_time_limit = None         # would not be enforceable anyway

The practical rule: under prefork the hard time limit is a genuine backstop, so the drain budget can be derived from it. Under gevent, threads, or solo, nothing can preempt a running task, so the only bound is the task's own cooperation — which means the checkpoint pattern from step 3 stops being an optimisation and becomes the mechanism. Teams that switch pools for throughput reasons frequently inherit deploy-time duplicates that were not there before, purely because the enforcement that used to bound the drain silently stopped applying. Choosing between pools on their own merits is covered in choosing Celery prefork vs gevent pools.

Warm and cold shutdowns per deploy Bars for six consecutive deploys. The first three show a mix of warm and cold shutdowns, with cold shutdowns indicating workers killed before draining. After the grace period is raised, the final three deploys show only warm shutdowns. Cold shutdowns should be zero v41 v42 v43 v44 v45 v46 grace period raised to 120s warm cold — every one is a redelivery

Verification

  1. Warm shutdown completes. kill -TERM <pid>; the log must show warm shutdown (MainProcess) followed by task completions and shutdown complete, with no SIGKILL.
  2. Drain fits the budget. Time from SIGTERM to process exit under load must be less than terminationGracePeriodSeconds with at least 25% margin.
  3. Checkpointed tasks resume. Start a long report, restart the worker, and confirm the row count in the output equals the total with no duplicated chunk beyond the one in flight.
  4. No redelivery spike on rollout. kubectl rollout restart under load; the count of executions with request.retries > 0 should stay near its baseline.

Gotchas & Edge Cases

A second SIGTERM cancels the drain. Some supervisors and impatient operators send it. Configure the orchestrator to send exactly one and to wait; on Kubernetes that is automatic, on systemd set KillSignal=SIGTERM with TimeoutStopSec above the drain.

worker_prefetch_multiplier > 1 inflates the release. Each prefetched message is released unacknowledged at shutdown and redelivered elsewhere. On a frequently-deployed fleet this is the dominant source of duplicate executions; see tuning prefetch and consumer concurrency.

acks_late = False loses work instead of duplicating it. The default acknowledges on receipt, so a killed task is simply gone. That trades a visible duplicate for an invisible loss, which is almost always the worse bargain.

Signal handlers registered in a task do not survive the pool. With the prefork pool, tasks run in child processes; a handler installed at import time in the parent is inherited, but one installed inside a task body is not reliably in place for the next one. Register at module import.

--max-tasks-per-child recycles mid-drain. A child that hits its limit during warm shutdown is replaced, and the replacement has nothing to do — harmless, but it makes the shutdown log look like a restart loop. Read the how=Warm field rather than inferring from process churn.

Redis visibility timeout must exceed the grace period. With the Redis transport, a message released at shutdown is only re-delivered after visibility_timeout. If that is an hour, a deploy silently stalls a portion of the queue for an hour.

Related