Draining BullMQ Workers Before Shutdown

This is the Node implementation of the drain model in Graceful Shutdown & Worker Deployments, under Backend Frameworks & Worker Scaling — how BullMQ's lock-based delivery behaves when a process exits, and what to call so it exits cleanly.

Problem Statement

Your Node workers process image transcodes averaging 25 seconds. On every deploy, PM2 sends SIGINT, the process exits immediately, and BullMQ's stalled-job checker eventually notices the lock expired and re-runs those jobs on another worker — thirty seconds later, from the beginning. Users see duplicate uploads in their media library because the transcode's side effect ran twice. You want the process to stop taking new jobs, finish the ones it holds, release its Redis locks, and exit before the orchestrator loses patience.

Prerequisites

  • BullMQ 4+ and Redis 6+. The lock semantics described here changed subtly across BullMQ 3 and 4, so pin the version.
  • Known p99 job duration — lockDuration and the grace period are both derived from it.
  • A process manager that sends a signal and waits: Kubernetes, systemd, or PM2 with kill_timeout raised from its 1.6-second default.
  • Idempotent processors, because a stalled job is re-run from the start rather than resumed.

Step 1 — Understand Locks and Stalled Jobs

BullMQ has no broker-side visibility timeout. A worker holds a lock on the job in Redis, renewed every lockDuration / 2 while the job runs. If the process dies, the lock is not renewed; the stalled-job checker on another worker notices, and moves the job back to wait — where it runs again from the beginning.

Lock renewal versus a stalled job after an abrupt exit The upper timeline shows a worker renewing its job lock every fifteen seconds while processing, then completing and releasing the lock. The lower timeline shows the same job on a worker that is killed mid-processing: the lock is not renewed, it expires after thirty seconds, and the stalled-job checker moves the job back to the wait state where another worker restarts it from the beginning. The lock is the only thing holding the job Healthy: lock renewed every lockDuration / 2 processing — renew at 15s, 30s, 45s … complete → lock freed Killed: no renewal, lock expires, job stalls processing… SIGKILL here lock TTL counting down stalled → re-run from the start The whole job repeats — not just the unfinished part. 0s 30s 60s maxStalledCount bounds the loop: after N stalls the job fails permanently.

The consequence is that draining matters more in BullMQ than in brokers with server-side leases: an abrupt exit does not just delay the job, it repeats work already done.

Step 2 — Call close() and Await It

worker.close() stops the worker from taking new jobs and resolves once active jobs finish. The failure mode is not awaiting it, or awaiting it after the process has already been told to exit.

// worker.js
const { Worker, QueueEvents } = require('bullmq');

const connection = { host: process.env.REDIS_HOST, port: 6379, maxRetriesPerRequest: null };

const worker = new Worker('transcode', processor, {
  connection,
  concurrency: 4,
  lockDuration: 60_000,        // > p99 job duration; renewed every 30s
  stalledInterval: 30_000,     // how often this worker checks for stalled jobs
  maxStalledCount: 2,          // after 2 stalls the job fails instead of looping
});

let shuttingDown = false;

async function shutdown(signal) {
  if (shuttingDown) return;    // a second signal must not cancel the drain
  shuttingDown = true;
  console.log(`${signal} received — draining`);
  try {
    // false = do NOT force; wait for active jobs to finish and locks to release.
    await worker.close(false);
    await connection.quit?.();
    console.log('drain complete');
    process.exit(0);
  } catch (err) {
    console.error('drain failed', err);
    process.exit(1);
  }
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

worker.close(true) forces immediate closure and abandons active jobs — the equivalent of a cold shutdown. Use it only as a last resort inside a watchdog, never as the default.

Step 3 — Add a Watchdog So a Stuck Job Cannot Hang the Pod

If one job never finishes, close(false) never resolves and the orchestrator eventually sends SIGKILL — losing every other in-flight job as collateral. A watchdog forces closure just before that deadline so the rest drain cleanly.

const GRACE_MS = Number(process.env.DRAIN_GRACE_MS ?? 100_000); // < terminationGracePeriodSeconds

async function shutdown(signal) {
  if (shuttingDown) return;
  shuttingDown = true;

  const forced = setTimeout(async () => {
    console.warn('drain exceeded budget — forcing close');
    await worker.close(true);   // abandon what is left; those jobs will stall and re-run
    process.exit(1);            // non-zero so the deploy surfaces it
  }, GRACE_MS);
  forced.unref();

  await worker.close(false);
  clearTimeout(forced);
  process.exit(0);
}

Exiting non-zero on a forced close is deliberate: it makes the event visible in pod restart reasons and dashboards instead of blending into normal shutdowns.

Step 4 — Extend Locks from Long Jobs

For jobs longer than lockDuration, extend the lock from inside the processor. BullMQ exposes this through job.extendLock, and the pattern doubles as a natural checkpoint.

async function processor(job) {
  const total = job.data.frames;
  for (let i = job.data.cursor ?? 0; i < total; i += 100) {
    await transcodeChunk(job.data.assetId, i, 100);

    // Renew the lock and publish progress — progress is durable, so a re-run
    // after a stall can skip the frames already done.
    await job.extendLock(job.token, 60_000);
    await job.updateProgress({ cursor: i + 100 });

    if (shuttingDown) {
      // Re-queue with the cursor so the next worker resumes rather than restarts.
      await job.moveToDelayed(Date.now() + 1000, job.token);
      return;
    }
  }
}

updateProgress is the only durable place to record a cursor without a second store, and it is readable from job.progress on the re-run. Without it, a stalled transcode restarts at frame zero, which is exactly the duplicate work the drain exists to avoid.

Step 5 — Configure the Process Manager to Wait

BullMQ's careful drain is worthless if the supervisor kills the process after 1.6 seconds.

// ecosystem.config.js (PM2)
module.exports = {
  apps: [{
    name: 'transcode-worker',
    script: 'worker.js',
    instances: 4,
    kill_timeout: 120_000,      // default is 1600ms — the usual culprit
    listen_timeout: 10_000,
    wait_ready: false,
  }],
};
# Kubernetes equivalent
spec:
  terminationGracePeriodSeconds: 120   # > DRAIN_GRACE_MS + exit overhead
  containers:
    - name: worker
      command: ["node", "worker.js"]   # NOT `npm start` — npm swallows signals

Running the process directly rather than through npm start matters: npm forwards signals inconsistently, so SIGTERM frequently never reaches Node and the drain never begins. Use node worker.js or an init shim such as tini as PID 1.

Step 6 — Choose lockDuration and Concurrency Together

These two settings interact in a way that is easy to miss. lockDuration bounds how long a job may run without a renewal; concurrency bounds how many jobs one process holds. Node's single-threaded event loop ties them together: a CPU-bound job blocks the loop, which blocks the renewal timer, which lets the lock expire on a job that is actively running.

Job shape Concurrency lockDuration Why
I/O-bound (HTTP, DB) 10–50 30s Renewals interleave freely between awaits
Mixed 4–10 60s Some blocking; give renewals more slack
CPU-bound in-process 1–2 120s+ The loop stalls; long lock is the only safety
CPU-bound in a worker thread 4–8 60s Offloading keeps the main loop responsive for renewals

The pathological case is CPU-bound work at high concurrency: a 20-second synchronous transcode with lockDuration: 30000 and concurrency: 20 will renew nothing for twenty seconds, and any job whose lock was already half-expired stalls while it is still running. The result is two workers processing the same job concurrently — worse than a duplicate after a crash, because both are live and writing at once. Push CPU-heavy work into worker_threads or a child process, or drop concurrency to 1 and scale out with more processes instead.

Drain, watchdog, and the grace-period boundary A shutdown timeline. On SIGTERM the worker calls close and active jobs continue. Most finish well inside the budget. A watchdog timer set below the orchestrator grace period forces closure for anything still running, so the remaining jobs drain cleanly rather than every job being killed at once. A watchdog bounds the drain from the inside SIGTERM watchdog 100s SIGKILL 120s job 1 completes, lock released job 2 completes job 3 stuck — forced closed at the watchdog, stalls and re-runs Without the watchdog, close() never resolves and jobs 1 and 2 are killed too. Exit non-zero on a forced close so the event is visible in pod restart reasons.

Verification

  1. Drain finishes cleanly. kill -TERM <pid> under load; logs must show drain complete and the process must exit 0 with no stalled-job warnings afterwards.
  2. No stalls follow a deploy. redis-cli LLEN bull:transcode:wait should not jump after a rollout, and the stalled event count must stay at zero.
  3. Locks are released, not expired. Watch bull:transcode:lock:* keys disappear at shutdown rather than lingering until TTL.
  4. The watchdog fires when it should. Deploy a deliberately hanging job, send SIGTERM, and confirm the forced close happens inside the grace period with a non-zero exit.
// Observability: count stalls so a regression is visible, not anecdotal
const events = new QueueEvents('transcode', { connection });
events.on('stalled', ({ jobId }) => {
  console.warn(`stalled job ${jobId}`);
  stalledCounter.inc();      // alert on any sustained non-zero rate
});
Concurrency and lockDuration must be chosen together Four job shapes with their safe concurrency and lock duration. I/O-bound work tolerates high concurrency with a short lock because renewals interleave freely. CPU-bound work in the main process blocks the event loop, so renewals stop and the lock must be long or concurrency must be one. The event loop renews the lock — do not block it I/O-bound concurrency 10–50 · lock 30s mixed concurrency 4–10 · lock 60s CPU in-process concurrency 1–2 · lock 120s+ — renewals stall CPU in a thread concurrency 4–8 · lock 60s — loop stays free A lock that expires while the job runs means two workers process it concurrently.

Gotchas & Edge Cases

lockDuration shorter than the job duration causes stalls with no deploy at all. The job keeps running while another worker picks it up, producing genuine concurrent duplicate execution. Set lockDuration above p99 duration or extend it from inside the processor.

Every worker runs the stalled checker. stalledInterval applies per worker, so a large fleet checks frequently in aggregate. That is usually fine, but a very short interval combined with a marginal lockDuration turns transient GC pauses into stall storms.

close() does not stop a QueueScheduler or QueueEvents connection. Close those explicitly, or the process stays alive with an open Redis handle and the orchestrator eventually kills it.

Concurrency multiplies the drain. With concurrency: 20 and 30-second jobs, a drain can take 30 seconds — but with unlucky timing it holds twenty jobs, any one of which can be the slow one. Size the grace period against the p99 of the slowest job class on that worker, not against the average.

maxStalledCount decides whether a poison job loops forever. Leave it at 1–2 so a job that repeatedly stalls fails and becomes visible rather than cycling. Pair it with the quarantine practices in dead-letter queues and poison-message handling.

Related