BullMQ Flows for Parent-Child Jobs

Flows turn a queue into a small dependency graph, which is exactly what a fan-out-then-aggregate workload needs. This guide extends BullMQ for Node.js Ecosystems, part of Backend Frameworks & Worker Scaling.

Problem Statement

A video upload must be transcoded into four renditions, then a manifest is written that references all four. Today one job does everything serially: twelve minutes of work on one worker, no parallelism, and a failure in the last rendition repeats the first three. You want the four transcodes to run concurrently across the fleet and the manifest job to run automatically once — and only once — every child has succeeded.

Prerequisites

  • BullMQ 4+ and Redis 6.2+ (flows use Lua scripts and sorted sets throughout).
  • Distinct queues for parent and child work, so each can be scaled and tuned independently.
  • Idempotent processors: a child can be retried, and the parent can be re-run after a partial failure.
  • An understanding of your failure policy — flows make it explicit, which is the point.

Step 1 — Model the Graph

FlowProducer.add takes a tree. The parent only becomes available once every child has completed.

const { FlowProducer } = require('bullmq');
const flow = new FlowProducer({ connection });

async function submitTranscode(assetId) {
  return flow.add({
    name: 'write-manifest',
    queueName: 'manifests',
    data: { assetId },
    opts: { attempts: 3, backoff: { type: 'exponential', delay: 2000 } },
    children: [
      { name: 'transcode', queueName: 'transcodes', data: { assetId, rendition: '1080p' } },
      { name: 'transcode', queueName: 'transcodes', data: { assetId, rendition: '720p'  } },
      { name: 'transcode', queueName: 'transcodes', data: { assetId, rendition: '480p'  } },
      { name: 'transcode', queueName: 'transcodes', data: { assetId, rendition: 'audio' } },
    ],
  });
}
Fan-out children, then an aggregating parent A flow producer creates four child transcode jobs on the transcodes queue, which run concurrently across the worker fleet. The parent manifest job sits in a waiting-children state until all four report success, at which point it moves to waiting and is picked up by a worker on the manifests queue. The parent waits; the children do not flow.add() one call transcode 1080p transcode 720p transcode 480p transcode audio queue: transcodes · run concurrently write-manifest waiting-children → waiting queue: manifests · runs once, after all four Twelve minutes serial becomes about four — the longest child, plus the parent.

Step 2 — Read the Children's Results in the Parent

Each child's return value is stored and made available to the parent, which is what makes the aggregate step possible without a separate store.

const { Worker } = require('bullmq');

new Worker('manifests', async (job) => {
  // Keys are `${queueName}:${childJobId}`; values are the children's return values.
  const childValues = await job.getChildrenValues();
  const renditions = Object.values(childValues);

  if (renditions.length !== 4) {
    // Defensive: a child that returned undefined is indistinguishable from a
    // missing one, so make children always return something meaningful.
    throw new Error(`expected 4 renditions, got ${renditions.length}`);
  }

  await writeManifest(job.data.assetId, renditions);
  return { assetId: job.data.assetId, renditions: renditions.length };
}, { connection, concurrency: 4 });
// The child must return a value, or the parent sees undefined for that key.
new Worker('transcodes', async (job) => {
  const { assetId, rendition } = job.data;
  const key = await transcode(assetId, rendition);
  return { rendition, key, bytes: await sizeOf(key) };   // ends up in the parent
}, { connection, concurrency: 2 });

Keep child return values small. They are stored in Redis and loaded together by the parent, so returning a whole payload rather than a reference recreates the sizing problem discussed in message size limits and serialization.

Step 3 — Decide What a Failed Child Means

By default a child that exhausts its attempts leaves the parent stuck in waiting-children forever — the flow neither completes nor fails, which is the single most confusing behaviour teams hit.

Policy Configuration Use when
All-or-nothing Default: parent waits; fail it explicitly on child failure The aggregate is meaningless without every part
Best effort failParentOnFailure: false and handle gaps in the parent Partial results are still useful
Fail fast failParentOnFailure: true on the child A single failure invalidates the whole flow immediately
Ignore ignoreDependencyOnFailure: true The child is optional enrichment 
children: [
  { name: 'transcode', queueName: 'transcodes', data: { assetId, rendition: '1080p' },
    opts: { attempts: 3, failParentOnFailure: true } },      // required
  { name: 'thumbnail', queueName: 'thumbnails', data: { assetId },
    opts: { attempts: 2, ignoreDependencyOnFailure: true } }, // nice to have
]

Choosing explicitly is the whole value here: the default "wait forever" is almost never what anyone intends, and it produces flows that sit invisibly until someone audits the waiting-children count.

Step 4 — Retry Only What Failed

The reason to decompose in the first place is that a retry should re-run one child, not the whole graph. BullMQ retries children independently, and the parent stays waiting while that happens.

// Failed children retry on their own schedule; the parent is untouched.
new Worker('transcodes', processor, {
  connection,
  concurrency: 2,
  settings: {
    backoffStrategy: (attemptsMade) =>
      Math.floor(Math.random() * Math.min(120_000, 2000 * 2 ** attemptsMade)),
  },
});

To resume a flow whose child exhausted its attempts after you have fixed the cause, retry the child rather than resubmitting the flow — resubmitting duplicates the three renditions that already succeeded.

const { Queue } = require('bullmq');
const transcodes = new Queue('transcodes', { connection });
const failed = await transcodes.getFailed(0, 100);
for (const job of failed) await job.retry();     // parent unblocks when all pass

Step 5 — Watch the waiting-children State

waiting-children is the flow health metric A line showing the count of parent jobs in the waiting-children state. In normal operation it hovers near zero as flows complete. After a change that causes children to fail permanently, the count climbs steadily because parents never complete and never fail, so no existing alert notices. Parents stuck in waiting-children healthy — flows complete children failing permanently — parents never fail either bad deploy Neither the failed-job counter nor queue depth moves here — this is the only signal. Alert on both the count and the age of the oldest waiting parent. 
const manifests = new Queue('manifests', { connection });
setInterval(async () => {
  const counts = await manifests.getJobCounts('waiting-children', 'waiting', 'failed');
  metrics.gauge('bullmq.waiting_children', counts['waiting-children'], { queue: 'manifests' });
}, 15_000);

A rising waiting-children count with a flat failed count is the specific signature of a child failure policy that was never configured.

Step 6 — Know When a Flow Is the Wrong Tool

Flows model dependencies, not workflows. They handle "these must finish before that" well, and everything else poorly.

Reach for something else when you need conditional branching (run B only if A returned a particular result), loops or dynamic fan-out sized at runtime, human approval steps, or waits measured in days. Those are workflow-engine problems — Temporal, Step Functions, or an explicit state machine in your own database — and forcing them into a flow produces graphs nobody can reason about.

The specific smell is a flow whose shape depends on data: if the number of children or their types is computed from a child's output, the graph cannot be declared up front, and you are simulating a workflow engine with job dependencies. A cheaper intermediate step is a parent that enqueues a new flow when it completes, keeping each individual graph static and declarative while allowing the overall process to be multi-stage.

Step 7 — Size the Fan-Out Deliberately

The number of children per flow is a capacity decision, not a modelling one. Four renditions per asset is trivial; four thousand rows per import is a different system. Two limits bite at scale.

The first is Redis. Every child carries its own job hash, its dependency entry on the parent, and its stored return value, so a flow with 5,000 children is roughly 5,000 times the bookkeeping of one — and creating it is a single large Lua invocation that blocks Redis for the duration. Keep fan-out in the tens or low hundreds and chunk beyond that: a parent with 50 children, each of which processes 100 rows, is far kinder than one with 5,000.

The second is the parent's aggregation step. getChildrenValues loads every child's return value into memory at once, so the parent's footprint scales with fan-out multiplied by return-value size. Returning a 20KB summary from each of 500 children means a 10MB object in the parent before it does any work.

// Chunked fan-out: bounded children, bounded aggregation
const CHUNK = 100;
const chunks = chunkArray(rowIds, CHUNK);            // 5,000 rows → 50 children
await flow.add({
  name: 'finalise-import', queueName: 'imports', data: { importId },
  children: chunks.map((ids, i) => ({
    name: 'import-chunk', queueName: 'import-chunks',
    data: { importId, ids, chunk: i },
    // Return a reference and a count, never the processed rows themselves.
    opts: { attempts: 3 },
  })),
});

The rule of thumb worth remembering: children should be sized so that one child is a meaningful unit of retry. Too fine, and the bookkeeping dominates the work; too coarse, and a retry repeats far more than it needs to.

Chunked fan-out keeps Redis and the parent bounded On the left, five thousand children each carry their own job hash, dependency entry and return value, and the parent must load all five thousand return values to aggregate. On the right, fifty children each process a hundred rows, so bookkeeping and parent memory stay small while the same work is done. Size children so one child is a unit of retry 5,000 children, one row each 5,000 job hashes + dependency entries one huge Lua call blocks Redis parent loads 5,000 return values at once a retry re-runs one row — very fine-grained 50 children, 100 rows each 50 hashes — bookkeeping is negligible creation is a small, fast Lua call parent aggregates 50 small summaries a retry re-runs 100 rows — acceptable

Verification

  1. Children run concurrently. Wall-clock time for a flow should approximate the slowest child plus the parent, not the sum of children.
  2. The parent runs exactly once. Under load, assert one manifest write per asset — a re-run indicates the parent was resubmitted rather than retried.
  3. A failed child blocks correctly. Force one child to exhaust its attempts and confirm the parent behaves per the configured policy rather than hanging silently.
  4. Retrying a child unblocks the parent. After the forced failure, retry the child and confirm the parent proceeds without re-running the successful siblings.

Gotchas & Edge Cases

getChildrenValues returns only completed children. A child that returned undefined is indistinguishable from one that is missing. Always return a value from child processors.

Flow trees are created transactionally, but not atomically with your database. If a flow must exist if and only if a row commits, apply the transactional outbox pattern.

Removing a parent orphans children. Deleting the parent job leaves children that run and complete with nothing to report to. Remove the whole tree via the flow API.

Deep trees multiply Redis round trips. Each level adds dependency bookkeeping; a three-level tree with fan-out at each level is much more expensive than the job count suggests. Prefer wide and shallow.

Child concurrency is per queue, not per flow. Ten flows each with four children put forty jobs on one queue, which will saturate a worker pool sized for four. Size the child pool for aggregate concurrency, as in configuring BullMQ concurrency limits.

Related