Choosing Celery Prefork vs Gevent Pools

The execution pool decides how many tasks a worker runs at once, how it fails, and whether time limits mean anything — which makes it one of the highest-leverage settings in Celery Architecture & Configuration, part of Backend Frameworks & Worker Scaling.

Problem Statement

Your webhook-delivery workers run the default prefork pool with concurrency 8 on 4-core nodes. Each task spends 400ms waiting on an HTTP response and about 5ms computing, so eight processes consume eight cores' worth of memory to keep almost nothing busy. Throughput is capped at roughly 20 requests per second per node, and scaling out means paying for RAM to hold idle processes. Meanwhile your image-resize workers, on the same configuration, genuinely need the isolation prefork provides.

Prerequisites

• A breakdown of where task time goes: CPU versus network or disk wait.
• Knowledge of which libraries the tasks use, since C extensions and gevent interact badly.
• Memory measurements per worker process — the pool choice changes the arithmetic entirely.
• The ability to run separate worker deployments per queue, which is what makes per-workload pools possible.

Step 1 — Know What Each Pool Actually Is

Step 2 — Measure Where the Time Goes

The decision follows from one number: the fraction of task duration spent waiting rather than computing.

import time, resource
from celery.signals import task_prerun, task_postrun

_start = {}

@task_prerun.connect
def _pre(task_id=None, **kw):
    ru = resource.getrusage(resource.RUSAGE_SELF)
    _start[task_id] = (time.monotonic(), ru.ru_utime + ru.ru_stime)

@task_postrun.connect
def _post(task_id=None, sender=None, **kw):
    wall0, cpu0 = _start.pop(task_id, (None, None))
    if wall0 is None:
        return
    ru = resource.getrusage(resource.RUSAGE_SELF)
    wall = time.monotonic() - wall0
    cpu = (ru.ru_utime + ru.ru_stime) - cpu0
    # cpu/wall near 1.0 → CPU-bound (prefork). Near 0 → I/O-bound (gevent).
    CPU_RATIO.labels(task=sender.name).observe(cpu / wall if wall else 0)

A ratio below about 0.2 means the task is waiting four fifths of the time and gevent will multiply throughput several-fold. Above 0.7, gevent buys nothing and prefork's parallelism is what you need.

Step 3 — Configure Prefork for CPU Work

# celeryconfig.py — CPU-bound workers
worker_pool = "prefork"
worker_concurrency = 4                  # ≈ cores; more only adds context switching
worker_max_tasks_per_child = 500        # recycle to bound memory growth
worker_max_memory_per_child = 400_000   # KB — restart a child above 400 MB
worker_prefetch_multiplier = 1
task_soft_time_limit = 55
task_time_limit = 75                    # enforceable: the child is killed

worker_max_memory_per_child is the setting that quietly saves prefork fleets: a task with a slow leak recycles its child instead of growing until the OOM killer removes the whole worker.

Step 4 — Configure Gevent for I/O Work

pip install "celery[gevent]"
celery -A app worker --pool=gevent --concurrency=200 -Q webhooks -n webhooks@%h

# The monkey patch MUST run before anything imports socket, ssl or urllib3.
# Putting it at the top of the module Celery loads first is the only reliable place.
from gevent import monkey
monkey.patch_all()

import app  # noqa: E402  — import order matters here more than lint style

Two constraints come with it. Hard time limits do not work: task_time_limit relies on signalling a child process, and greenlets are not processes, so a runaway task runs forever. And any C extension that blocks without releasing the GIL — some database drivers, some compression and image libraries — blocks the entire pool rather than one task.

# Gevent-appropriate settings
worker_pool = "gevent"
worker_concurrency = 200        # bounded by the downstream, not by cores
task_soft_time_limit = 30       # cooperative only — the task must yield to see it
task_time_limit = None          # unenforceable under gevent; do not pretend otherwise

Step 5 — Size Concurrency from the Bottleneck

def gevent_concurrency(target_rps: float, p95_seconds: float,
                       downstream_limit: int) -> int:
    """Little's Law, capped by whatever the downstream will actually accept."""
    needed = math.ceil(target_rps * p95_seconds)
    return min(needed, downstream_limit)

gevent_concurrency(200, 0.4, downstream_limit=500)   # 80
gevent_concurrency(200, 0.4, downstream_limit=50)    # 50 — the API is the limit

Setting concurrency to 1,000 because greenlets are cheap simply moves the queue from the broker into the downstream's connection backlog, where it is invisible. The binding constraint is almost always a pool or a quota — the same reasoning as tuning prefetch and consumer concurrency.

Step 6 — Run Both, One per Workload

Step 7 — Understand How Each Pool Fails

Pool choice changes the blast radius of a failure, which matters more than throughput when something goes wrong.

Under prefork, a segfault in a C library kills one child; the parent notices and replaces it, and only that child's task is lost — which task_reject_on_worker_lost turns into a redelivery rather than a loss. Under gevent, the same segfault takes the process down with every in-flight greenlet, so a single bad input can destroy two hundred unrelated tasks at once. That asymmetry is the strongest argument for prefork whenever the task touches native code you do not control.

Memory failures differ too. A prefork leak is bounded by max_memory_per_child, which recycles quietly. A gevent leak accumulates in one long-lived process with no equivalent guard, so it ends in an OOM kill that takes the whole pool. If you run gevent, add an external memory watchdog that restarts the worker gracefully well before the cgroup limit — the graceful path in handling SIGTERM in Celery workers — rather than letting the kernel choose the moment.

Step 8 — Cost the Decision in Nodes, Not in Feelings

Pool choice has a directly computable cost, and putting numbers on it usually ends the debate faster than any argument about elegance.

def nodes_required(target_rps: float, p95_seconds: float, pool: str,
                   cores_per_node: int = 4, ram_gb: int = 8,
                   mem_per_proc_mb: int = 180) -> int:
    """How many nodes to sustain target_rps, given the pool's concurrency model."""
    in_flight = math.ceil(target_rps * p95_seconds)
    if pool == "prefork":
        per_node = min(cores_per_node,
                       (ram_gb * 1024) // mem_per_proc_mb)   # cores or RAM, whichever binds
    else:                                                    # gevent
        per_node = 250                                       # bounded by the downstream
    return math.ceil(in_flight / per_node)

nodes_required(200, 0.4, "prefork")   # 20 nodes — 80 in flight, 4 per node
nodes_required(200, 0.4, "gevent")    # 1 node   — 80 greenlets on one core

For the webhook workload in the problem statement, that is a twentyfold difference in infrastructure for identical throughput. For an image-resize workload the calculation inverts entirely: greenlets give no CPU parallelism, so gevent would need the same twenty nodes and lose the hard time limits and crash isolation that prefork provides.

Run the numbers per queue and record them alongside the pool setting. The output is not just a node count — it is the justification that stops the setting being changed on intuition six months later, and it makes the review trigger obvious: re-run it when p95 duration or the CPU ratio moves by more than a factor of two.

Verification

1. The CPU ratio matches the pool. Tasks on gevent queues should show a ratio below 0.2; anything above 0.5 belongs on prefork.
2. Throughput improved measurably. Compare requests per second per node before and after; an I/O-bound workload should improve several-fold, not marginally.
3. Monkey patching happened early. python -c "import app; import socket; print(socket.socket)" must show the gevent-patched class.
4. Time limits behave as documented. Run a deliberately infinite task on each pool: prefork must kill it at the hard limit, gevent must not — confirm you are not relying on a guarantee you do not have.

Gotchas & Edge Cases

Late monkey patching silently does nothing. If socket is imported before patch_all(), the pool runs but every call blocks, and concurrency collapses to one. Nothing warns you.

Blocking C extensions freeze the whole gevent pool. psycopg2 in its default mode, some image libraries, and any time.sleep inside a C call will stall every greenlet. Use psycogreen or move that work to prefork.

Hard time limits are prefork-only. Documentation and configuration accept task_time_limit under gevent; it simply has no effect. Long-running tasks there must cooperate.

Prefork forks after import. Anything created at import time — database connections, file handles, random seeds — is shared across children in an undefined state. Create connections inside the task or in worker_process_init.

Mixing pools in one worker is impossible. Monkey patching is process-wide, so a single worker cannot serve a gevent queue and a prefork queue. Separate deployments are not a stylistic choice here; they are the only way to run both.

Threads help only for GIL-releasing extensions. The threads pool is genuinely useful for NumPy, image codecs, and drivers that release the GIL, and nearly useless for pure Python.

Related

• Celery Architecture & Configuration — the broader configuration surface these settings sit in.
• Celery task routing with task_routes — separating workloads so they can use different pools.
• Handling SIGTERM in Celery workers — how each pool behaves during a drain.
• Tuning prefetch and consumer concurrency — the other half of the concurrency decision.