Graceful Shutdown & Worker Deployments

Deploying a web server is a solved problem: drain connections, stop accepting, exit. Deploying a worker fleet is not, because a worker holds work that has no client waiting on a socket and no load balancer tracking its state — which is why deployment safety belongs alongside the scaling models in Backend Frameworks & Worker Scaling. A worker killed mid-job either loses that job or, if the broker redelivers it, runs it twice.

Every deploy is therefore a controlled test of your redelivery and idempotency guarantees. Teams that discover this during an incident usually discover it as duplicate charges, duplicate emails, or a half-written export. The mechanics below make a restart boring: signal handling that stops intake but finishes work, a drain window sized from real job durations, and orchestrator settings that agree with both.

What Actually Happens on SIGTERM

A worker receives SIGTERM and has a bounded time before SIGKILL arrives. The correct sequence in that window is: stop fetching new messages, finish or checkpoint what is in flight, acknowledge completed work, release the rest so it is redelivered promptly, then exit. Frameworks implement roughly this, with important differences in defaults.

Shutdown timeline from SIGTERM to exit A horizontal timeline. At zero seconds SIGTERM arrives and the worker stops fetching new messages while three in-flight jobs continue. Two finish and are acknowledged within the drain window. The third is still running when the window closes, so it is released without acknowledgement for redelivery. SIGKILL arrives at the end of the grace period. The drain window is the whole design SIGTERM SIGKILL grace period — terminationGracePeriodSeconds intake stops here — no new fetches job A finishes at +9s → ack job B finishes at +22s → ack job C still running when the window closes release C unacknowledged → broker redelivers to another worker 0s 15s 30s Grace period must exceed p99 job duration, or every deploy manufactures redeliveries.

The second SIGTERM (or SIGINT) is usually the escalation path — Celery calls it "cold shutdown" and terminates in-flight tasks immediately. Orchestrators that send repeated signals, or supervisors configured to "restart harder," can therefore convert a graceful shutdown into an abrupt one without anyone intending it.

Framework Signal semantics Default drain behaviour
Celery TERM = warm shutdown, second TERM/QUIT = cold Waits indefinitely for in-flight tasks unless --soft-time-limit is set
Sidekiq TSTP = stop fetching, TERM = shutdown 25s timeout, then pushes unfinished jobs back to Redis
BullMQ worker.close() Waits for active jobs; stalled jobs recovered by the stalled-check interval
RQ TERM = warm, second = cold Waits for the current job; no built-in timeout
Dramatiq TERM = graceful Waits for in-flight, then requeues

Sidekiq's two-signal model is the most explicit and worth copying conceptually: TSTP first (stop fetching, keep working), then TERM after a pause. That separation lets a deployment quiesce a fleet before it starts terminating anything.

Sizing the Drain Window

The grace period must exceed the p99 duration of the jobs a worker may be holding. Anything shorter guarantees that the slowest tail of every deploy is killed mid-flight and redelivered — which is survivable if your handlers are idempotent and expensive if they are not.

# drain_window.py — derive the grace period from observed durations
def grace_period(p99_seconds: float, checkpoint: bool = False) -> int:
    """Grace = time to finish the slowest in-flight job + shutdown overhead.
    With checkpointing you only need to reach the next checkpoint."""
    base = 30 if checkpoint else p99_seconds
    return int(base + 15)     # +15s for ack flush, connection close, process exit

grace_period(45)                    # 60  — typical API-bound jobs
grace_period(1800)                  # 1815 — a 30-minute ETL: too long to wait
grace_period(1800, checkpoint=True) # 45   — checkpoint every 30s instead

Jobs longer than a few minutes should not be handled by extending the grace period to match; nobody wants a thirty-minute deploy. Make them resumable instead — checkpoint progress, and on redelivery skip the completed portion. That converts an unbounded drain requirement into a bounded one and is the only approach that scales to hour-long work.

Jobs that cannot be checkpointed and cannot be safely repeated need the third option: refuse to start them near a known deploy window, using a "quiescing" flag the worker checks before accepting a long task.

Coordinating with the Orchestrator

On Kubernetes, three timeouts must agree, and by default they do not. The pod's terminationGracePeriodSeconds bounds everything; the container's preStop hook runs before SIGTERM and is the right place to stop intake; and the readiness probe removes the pod from any service endpoints — irrelevant for workers, which have no inbound traffic, but relevant for the metrics endpoint scraping them.

# worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
spec:
  replicas: 12
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 2      # never drain more than 2 workers at once
      maxSurge: 2            # bring replacements up before removing capacity
  template:
    spec:
      # Must exceed p99 job duration + ack flush. Kubernetes SIGKILLs at this
      # boundary regardless of what the process is doing.
      terminationGracePeriodSeconds: 120
      containers:
        - name: worker
          image: registry.example.com/worker:2026.07.26
          lifecycle:
            preStop:
              exec:
                # Stop fetching immediately; the grace period then covers only
                # the jobs already in flight, not newly-claimed ones.
                command: ["/bin/sh", "-c", "kill -TSTP 1 && sleep 5"]
          env:
            - name: CELERY_WORKER_SHUTDOWN_TIMEOUT
              value: "100"   # inside the 120s pod budget, with margin

The relationship to remember: preStop duration + drain time must be less than terminationGracePeriodSeconds, and the whole thing must be less than any external timeout (a CI deploy gate, a load balancer drain, an ArgoCD sync window). When those disagree, the shortest one wins and the design silently degrades to a hard kill. Zero-downtime worker deploys on Kubernetes works through the full manifest, including PodDisruptionBudgets for node drains.

Rolling Strategy: Capacity During the Deploy

A rolling deploy of workers removes processing capacity for the length of the drain, and a fleet at 90% utilisation cannot afford to lose 25% of itself for two minutes without the queue growing. The arithmetic is simple and usually skipped: with N workers, maxUnavailable M, drain time D, and B batches, the deploy costs roughly M × D × B worker-seconds of lost capacity.

Capacity during a rolling worker deploy Two capacity lines over the length of a deploy. Without surge, capacity dips below the required threshold during each batch and the queue backlog grows. With maxSurge set, replacement workers start before old ones drain, so total capacity stays at or above the requirement throughout. Rolling deploy: surge keeps the queue flat 14 12 0 capacity required no surge — every batch dips below the line maxSurge — replacements start first, capacity never dips deploy start deploy end Surge costs a few minutes of extra pods; a backlog costs latency across every queue.

maxSurge is cheap insurance: for the two or three minutes of a deploy you run a couple of extra workers, and total capacity never dips. The exception is a fleet constrained by a downstream connection limit — a database with a hard max_connections, for instance — where surging adds connections you do not have. In that case reduce maxUnavailable to 1 and accept a longer deploy instead, or scale connection pooling first, as described in tuning the Sidekiq Redis connection pool.

Checkpointing Long Jobs

The alternative to a long grace period is a job that can stop early and resume where it left off. Checkpointing turns an unbounded drain requirement into a bounded one: the worker only needs enough time to reach the next checkpoint, typically a few seconds, no matter how long the whole job runs.

# resumable.py — a batch job that survives being stopped mid-flight
import signal
import redis

r = redis.Redis(decode_responses=True)
_stopping = False

def _on_term(signum, frame):
    """Do not exit here — just raise the flag. The loop decides where it is
    safe to stop, which is the only place that knows the invariants."""
    global _stopping
    _stopping = True

signal.signal(signal.SIGTERM, _on_term)

@shared_task(bind=True, acks_late=True, max_retries=None)
def export_rows(self, export_id: str, total: int, batch: int = 500):
    key = f"export:{export_id}:cursor"
    cursor = int(r.get(key) or 0)

    while cursor < total:
        rows = fetch(export_id, offset=cursor, limit=batch)
        write_chunk(export_id, rows)
        cursor += len(rows)
        # Commit the cursor AFTER the chunk is durably written: a crash between
        # the two repeats one batch, which the chunk writer must tolerate.
        r.set(key, cursor, ex=86_400)

        if _stopping:
            # Re-enqueue ourselves and exit cleanly inside the drain window.
            raise self.retry(countdown=0)

    r.delete(key)
    return {"export_id": export_id, "rows": cursor}

Two properties make this work. The signal handler never terminates anything itself — it sets a flag that the loop reads at a point where state is consistent. And the cursor is committed after the durable write, so the worst case on an abrupt kill is repeating one batch rather than losing or skipping rows. Chunk writes must therefore be idempotent, which is the same requirement redelivery already imposes.

Choose the checkpoint interval so that a checkpoint costs well under a second and occurs every 30–60 seconds of work. More frequent checkpoints add I/O for no benefit; less frequent ones erode the guarantee, because the drain window has to cover the gap between them.

Verifying a Deploy Is Safe

Treat deploy safety as a property you test, not one you assume. Three checks catch nearly everything, and all of them can run in staging.

# 1. Does the worker actually stop fetching on SIGTERM, and finish in-flight work?
kubectl exec deploy/celery-worker -- kill -TERM 1
kubectl logs deploy/celery-worker --tail=50 | grep -E "warm shutdown|Restoring|shutdown complete"

# 2. How many messages get redelivered by a full rolling restart under load?
#    Redelivery count should be ~0 with acks_late + a correct grace period.
kubectl rollout restart deploy/celery-worker
watch -n2 'redis-cli LLEN celery; redis-cli ZCARD unacked_index'

# 3. Does any job exceed the grace period? Compare p99 duration to the budget.
curl -s localhost:9808/metrics | grep 'celery_task_runtime_seconds{quantile="0.99"}'

The second check is the one that matters most: a rolling restart under production-like load, with duplicate-execution counters instrumented. If the deploy produces redeliveries, either the grace period is too short or something is killing workers before they drain — and both are far cheaper to discover in staging than through a customer report. Record the number as a release-gate metric and alert if it regresses, in the same way the practices in Observability & Monitoring for Job Queues treat any other reliability signal.

Three timeouts, and the shortest one wins Nested bars showing the orchestrator grace period as the outermost bound, the framework drain timeout inside it, and the preStop hook at the start. When any inner value exceeds the outer bound, the process is killed regardless of what the inner setting says. These three numbers must agree terminationGracePeriodSeconds: 120 — the hard wall preStop 5s framework drain timeout: 100s slack preStop + drain must fit inside the grace period, with room for ack flush and process exit. A CI deploy gate or load-balancer drain shorter than any of these silently becomes the real limit.

Failure Modes & Recovery

Duplicate side effects after every deploy. Symptom: a small burst of duplicate emails or charges correlated with release times. Cause: jobs killed mid-flight and redelivered without idempotency. Fix: idempotency keys on every non-idempotent handler; see preventing duplicate job execution with idempotency.

Jobs stuck invisible for the full visibility timeout. Symptom: after a deploy, throughput drops and queue depth stalls for exactly the visibility timeout. Cause: workers exited without releasing unacknowledged messages, so the broker waits for the lease to expire. Fix: explicitly nack/reject with requeue on shutdown, and keep the visibility timeout close to real job durations.

SIGKILL before drain completes. Symptom: worker logs end abruptly with no "shutdown complete" line. Cause: terminationGracePeriodSeconds shorter than the drain. Fix: raise the grace period above p99 duration, or checkpoint long jobs.

Scheduler duplication during rollout. Symptom: periodic tasks fire twice around deploys. Cause: two schedulers (Celery Beat, Sidekiq Cron) briefly running together during a surge. Fix: run schedulers as a Recreate strategy singleton with leader election, per preventing duplicate scheduled jobs with leader election.

Prefetched messages lost to the drain. Symptom: a burst of redeliveries far larger than the number of running jobs. Cause: a high prefetch means each worker holds dozens of unstarted messages that all get released at once. Fix: prefetch_multiplier=1 on any fleet that deploys frequently.

Deploying the Scheduler and the Workers Differently

A worker fleet and a periodic scheduler have opposite deployment requirements, and treating them as one deployment is a recurring source of duplicate work. Workers are horizontally redundant: running two extra during a rollout is harmless. A scheduler is a singleton: two instances running simultaneously means every periodic task fires twice.

# beat-deployment.yaml — the scheduler is the one component that must NOT surge
spec:
  replicas: 1
  strategy:
    type: Recreate          # terminate the old pod fully before starting the new one
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: beat
          args: ["celery", "-A", "app", "beat", "--pidfile="]

Recreate accepts a few seconds with no scheduler — which simply delays a periodic tick — in exchange for never running two. That is the correct trade for a component whose failure mode is duplication rather than latency. Belt and braces is a lock held in Redis or the database, so even a misconfigured rollout cannot produce two active schedulers; the leader-election pattern in preventing duplicate scheduled jobs with leader election covers the implementation.

The same asymmetry applies to any singleton sidecar in a worker fleet — a queue-depth exporter that writes to a shared gauge, a partition rebalancer, a nightly compaction job. Enumerate them explicitly before the first rolling deploy, because each one is a component where "two running briefly" is not benign.

Performance Tuning

  • Grace period: p99 job duration + 15s, capped at ~120s. Longer than that means the job needs checkpointing, not a longer window.
  • maxUnavailable: 10–20% of the fleet. Lower for tightly-utilised fleets; 1 for fleets bounded by downstream connection limits.
  • maxSurge: match maxUnavailable when capacity is tight and the downstream can take the extra connections.
  • Prefetch: 1 for frequently-deployed fleets. Every prefetched message is a message that gets released and redelivered on shutdown.
  • Checkpoint interval: aim for 30–60s of work between checkpoints. Shorter wastes I/O; longer erodes the benefit.
  • Node drains: a PodDisruptionBudget with minAvailable set to your required capacity stops a node-pool upgrade from draining half the fleet at once. Without one, the deploy strategy you tuned is bypassed entirely by node maintenance.
  • Deploy timing: avoid deploying into a scheduled burst. If the nightly rollup starts at 02:00, do not ship at 01:55 — the drain will collide with the enqueue spike.

FAQ

How long should the grace period be? Longer than the p99 duration of the jobs a worker can hold, plus ten to fifteen seconds for acknowledgement flushing and process exit. If that number exceeds two minutes, the answer is not a longer grace period — it is checkpointing the job so a restart resumes rather than repeats.

Can I deploy workers and the application code that enqueues them independently? You have to, and the constraint is message compatibility rather than timing. During any rollout, old workers consume messages published by new producers and vice versa, so a task signature change must be additive: add the new parameter with a default, ship the workers, then ship the producers that pass it. Removing a parameter — or renaming a task — is a two-release operation for the same reason. Treating a queue as a versioned interface between two independently-deployed services is the mental model that makes this obvious.

Do I need idempotency if shutdown is graceful? Yes. Graceful shutdown reduces redeliveries; it cannot eliminate them, because a node can lose power, an OOM killer can fire, and a grace period can expire. Every deploy strategy assumes at-least-once delivery underneath it, so handlers must tolerate re-execution.

Should workers stop fetching before they stop working? Always, and they should be separate steps. Stopping intake first means the drain window only has to cover jobs already in flight rather than jobs claimed a moment before exit. Sidekiq exposes this as TSTP then TERM; Celery and BullMQ achieve it through a preStop hook or an explicit close() before the signal.

How do I know a deploy actually caused a duplicate? Instrument it rather than inferring it. Stamp each execution with the message ID and the attempt number, and count executions where the attempt number is greater than one. Plot that counter against deploy annotations: a spike aligned to a rollout is a drain problem, a steady background rate is a timeout or crash problem, and a spike with no deploy nearby is usually a visibility timeout shorter than the job's real duration. Without the counter every conversation about deploy safety is speculative.

What about long-running jobs that cannot be checkpointed? Isolate them. Give them a dedicated worker pool with a long grace period and a slow deploy cadence, so the rest of the fleet can ship several times a day without waiting on them. Pair that with an admission check that refuses to start such a job when a deploy is imminent.

Related