Zero-Downtime Worker Deploys on Kubernetes

This is the manifest-level implementation of the drain model in Graceful Shutdown & Worker Deployments, part of Backend Frameworks & Worker Scaling — the four settings that decide whether a rollout is invisible or produces a wave of redeliveries.

Problem Statement

Your worker Deployment uses defaults: a 30-second termination grace period, maxUnavailable: 25%, no preStop hook, and no PodDisruptionBudget. Jobs average 20 seconds with a p99 of 90. Every rollout therefore kills roughly one job per pod mid-flight, and every cluster autoscaler event does the same thing without any deploy at all. Queue depth spikes for a few minutes after each release, and the on-call rotation has learned to ignore it — which means a real incident during a deploy window would also be ignored.

Prerequisites

  • A worker image whose process handles SIGTERM gracefully; on Celery that is covered in handling SIGTERM in Celery workers.
  • Measured p50/p99 task durations per queue — every number below is derived from them.
  • kubectl access to edit the Deployment, plus permission to create a PodDisruptionBudget.
  • Metrics on queue depth and duplicate executions, so the improvement is measurable rather than assumed.

Step 1 — Size the Grace Period from Real Durations

terminationGracePeriodSeconds is a hard wall: when it expires, the container gets SIGKILL regardless of what it is doing. Set it above the p99 duration of any task that pod can hold, plus overhead for acknowledgement flushing and process exit.

# p99 task runtime per queue, straight from the exporter
curl -s localhost:9808/metrics \
  | grep 'celery_task_runtime_seconds{quantile="0.99"'
# celery_task_runtime_seconds{queue="invoices",quantile="0.99"} 88.4
# celery_task_runtime_seconds{queue="webhooks",quantile="0.99"} 3.1
# → invoices needs ~120s; webhooks is fine at 30s. Split them.

Two queues with a 30× duration difference should not share a Deployment. Splitting them lets the webhook fleet roll in seconds while the invoice fleet takes its two minutes, instead of every deploy paying the slowest queue's cost.

Kubernetes pod termination sequence for a worker A pod termination timeline. The API server marks the pod terminating and runs the preStop hook, during which the worker stops fetching. SIGTERM then arrives and in-flight tasks finish. The process exits before the termination grace period expires, so SIGKILL is never sent. A second bar shows a misconfigured pod where the grace period expires first. preStop + SIGTERM must both fit inside the grace period Correct: grace 120s, p99 task 88s preStop SIGTERM → in-flight tasks finish → ack → exit unused headroom 120s Broken: grace 30s (the default), same tasks preStop draining… SIGKILL — task killed, message redelivered elsewhere every pod in the rollout produces at least one duplicate execution 0s 60s 120s

Step 2 — Quiesce in preStop, Finish in the Grace Period

The preStop hook runs before SIGTERM. Use it to stop fetching so the grace period only has to cover work already in flight, not work claimed a second before shutdown.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker-invoices
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # drain one pod at a time on a latency-sensitive queue
      maxSurge: 2           # replacements start first, so capacity never dips
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: worker
          image: registry.example.com/worker:2026.07.26
          args: ["celery", "-A", "app", "worker", "-Q", "invoices",
                 "-c", "8", "--prefetch-multiplier=1"]
          lifecycle:
            preStop:
              exec:
                # Stop consuming immediately; SIGTERM follows once this returns.
                command: ["sh", "-c", "celery -A app control cancel_consumer invoices && sleep 5"]
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits:   { memory: "1Gi" }

Note what is absent: no readiness probe gating traffic, because workers receive none. A readiness probe on a worker is still useful for surfacing health in kubectl get pods, but it plays no part in draining — which surprises teams who assume the web-service playbook transfers unchanged.

Step 3 — Keep Capacity Flat with Surge

A rolling update removes capacity for the duration of each pod's drain. With maxSurge the replacement is running before the old pod finishes, so total throughput never dips.

# Watch capacity and backlog together during a rollout
kubectl rollout restart deploy/worker-invoices
watch -n2 'kubectl get pods -l app=worker-invoices --no-headers | wc -l; \
           redis-cli LLEN invoices'

Surge costs extra pods for the length of the deploy. The exception is a fleet bounded by a downstream connection limit — Postgres max_connections, a Redis maxclients, a third-party API concurrency cap — where extra pods mean connection errors. In that case set maxSurge: 0, maxUnavailable: 1, and accept a slower rollout.

Step 4 — Protect Against Node Drains

A rolling update is only one way pods die. Cluster autoscaling, node upgrades, and spot reclamation all evict pods through a completely different path, one that ignores your Deployment strategy. A PodDisruptionBudget is what applies a limit there.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: worker-invoices
spec:
  # Keep at least 80% of the fleet running through any voluntary disruption.
  minAvailable: 8
  selector:
    matchLabels:
      app: worker-invoices

Without a PDB, kubectl drain on a node hosting six of your ten workers evicts all six simultaneously, each with its own full drain — a far larger capacity loss than any deploy you tuned for. This is the single most commonly missing piece in otherwise careful worker manifests.

Node drain with and without a disruption budget Without a PodDisruptionBudget, draining a node evicts all six workers hosted on it at once and fleet capacity drops from ten to four. With minAvailable set to eight, the eviction is serialised so no more than two workers are unavailable at any moment and capacity stays at or above eight. A node drain bypasses your rollout strategy No PDB — six pods evicted at once capacity 10 → 4 instantly six simultaneous drains, six chances to exceed the grace period minAvailable: 8 — eviction serialised never below 8 running The drain waits for replacements; the node upgrade takes longer and nothing is lost. Set minAvailable from required throughput, not from a round percentage.

Step 5 — Give Long Queues Their Own Deployment

Splitting by queue is what makes fast fleets deploy fast. Each Deployment gets a grace period matched to its own workload.

# webhooks: 3s p99 → quick rollout, aggressive surge
terminationGracePeriodSeconds: 30
rollingUpdate: { maxUnavailable: 3, maxSurge: 3 }
---
# exports: 20-minute jobs → checkpointing, not a 20-minute grace period
terminationGracePeriodSeconds: 90
rollingUpdate: { maxUnavailable: 1, maxSurge: 1 }

For the exports fleet the grace period is deliberately far below the job duration: those tasks checkpoint and re-enqueue, so 90 seconds is enough to reach the next checkpoint. Extending the grace period to cover a twenty-minute job would make every deploy twenty minutes long and would still fail on the job that takes twenty-five.

Step 6 — Make the Rollout Observable

A deploy that "looks fine" in kubectl rollout status can still be shedding work, because rollout status only knows about pod readiness. Annotate deploys into your metrics backend and watch three series across the window: queue depth, oldest-message age, and executions with attempt greater than one.

# A deploy annotation Grafana can overlay on the queue panels
apiVersion: v1
kind: ConfigMap
metadata:
  name: deploy-annotation
data:
  hook.sh: |
    curl -s -X POST "$GRAFANA_URL/api/annotations" \
      -H "Authorization: Bearer $GRAFANA_TOKEN" \
      -H 'Content-Type: application/json' \
      -d "{\"tags\":[\"deploy\",\"$DEPLOY_TARGET\"],\"text\":\"rollout $IMAGE_TAG\"}"

The pattern to look for is a small, symmetric bump in depth that recovers within a couple of minutes — that is capacity churn and is expected. What is not expected is a step change in oldest-message age, which means some portion of the queue stopped being consumed, or a rise in the retry-attempt counter, which means jobs were killed mid-flight. Both are actionable and neither is visible without the annotation to align them against. Treat "no duplicate executions attributable to the last ten deploys" as a release-quality metric alongside error rate; the alerting practices in Observability & Monitoring for Job Queues apply unchanged.

One deployment per queue, sized to its own durations Two worker deployments side by side. The webhooks fleet has a three-second p99 duration, a thirty-second grace period and rolls three pods at a time. The exports fleet has twenty-minute checkpointed jobs, a ninety-second grace period and rolls one pod at a time, so the fast fleet is not forced to inherit the slow one's deploy time. Splitting by duration makes deploys fast again worker-webhooks p99 job: 3s grace: 30s · maxUnavailable 3 · maxSurge 3 whole rollout: under a minute ships several times a day without ceremony worker-exports p99 job: 20 min, checkpointed every 30s grace: 90s · maxUnavailable 1 · maxSurge 1 rollout: several minutes, and that is fine grace covers the checkpoint, not the whole job Combined into one deployment, every webhook deploy would pay the export fleet's drain time.

Verification

  1. No SIGKILL in a rollout. kubectl rollout restart under load, then check each terminated pod's last log lines for a clean shutdown message rather than an abrupt end.
  2. Capacity stays flat. Sample running pod count every two seconds during the rollout; with surge configured it should never fall below the replica count.
  3. Backlog does not grow. Queue depth at the end of the rollout must be within noise of where it started.
  4. A node drain is survivable. kubectl drain <node> --ignore-daemonsets on a node hosting several workers: the PDB must serialise the eviction, and depth must stay flat.
  5. Duplicates stay at baseline. Count executions with attempt > 1 before and after; a rollout should not move the number.

Gotchas & Edge Cases

preStop runs inside the grace period, not before it. The clock starts when the pod is marked for deletion, so a sleep 30 in preStop consumes 30 of your 120 seconds. Keep it to the few seconds needed to stop consuming.

maxUnavailable as a percentage rounds down. With 3 replicas, 25% is 0, and Kubernetes silently bumps it to 1 — a third of the fleet. On small fleets always use absolute numbers.

HPA and rollouts interact badly. An autoscaler scaling down during a rollout evicts pods the rollout is also replacing, doubling the churn. Use a stabilizationWindowSeconds on scale-down and prefer queue-depth-driven scaling that reacts to the backlog rather than to CPU; see horizontal worker scaling.

No PDB means the Kubernetes autoscaler ignores your careful settings. This is worth repeating because it produces incidents that look random: capacity drops with no deploy, no alert, and no obvious cause.

Spot and preemptible nodes give you far less notice. A spot reclamation typically offers 30 seconds to two minutes regardless of your grace period, so a 120-second drain simply does not complete. Run long-queue workers on on-demand capacity and keep the short-queue fleets on spot, rather than assuming the grace period applies everywhere.

Init containers delay every replacement. A slow migration or asset-sync init container adds its runtime to each new pod, stretching the rollout well past the drain time and widening the window where capacity is degraded.

Related