Kafka vs RabbitMQ for Task Queues

This comparison extends Message Broker Comparison under Queue Fundamentals & Architecture to the specific question teams keep asking: we already run Kafka for events — should our background jobs go there too?

Problem Statement

Your platform team runs Kafka for the event pipeline and RabbitMQ for background jobs, and maintaining two systems is expensive. The proposal is to move jobs onto Kafka. Before agreeing, you need to know what actually changes for job workloads: how a single slow job affects everything behind it, how retries and dead-lettering work without per-message acknowledgement, and whether consumer scaling is still a matter of adding workers.

Prerequisites

  • Familiarity with your job mix: durations, retry rates, and whether ordering matters per key.
  • Current RabbitMQ (or SQS) metrics — prefetch, consumer count, redelivery rate — as the baseline.
  • Awareness that "we already run Kafka" is an operational argument, not a technical one; both sides deserve honest weighting.

Step 1 — The Structural Difference

RabbitMQ is a queue: a message is delivered to one consumer, acknowledged individually, and removed. Kafka is a log: messages are appended to a partition, consumers track an offset, and nothing is removed on consumption. That single difference drives every practical consequence below.

Per-message ack versus a partition offset On the left, RabbitMQ delivers messages to consumers which acknowledge each one independently, so a slow message does not delay others. On the right, a Kafka partition is consumed in order by one consumer, and a slow message at the current offset blocks every message behind it in that partition until it completes. Removal by acknowledgement vs progress by offset RabbitMQ — independent acks m1 m2 slow m4 m5 m4 and m5 complete while "slow" is still running Add consumers freely; each takes whatever is next. Kafka — one offset per partition m1 m2 slow m4 m5 offset stuck here m4, m5 wait — head-of-line blocking inside the partition Consumers cannot exceed the partition count. Neither is better: a log gives replay and ordering, a queue gives independent completion. Job workloads with variable durations mostly want independent completion.

Step 2 — Compare on What Job Systems Need

Capability RabbitMQ Kafka
Acknowledgement Per message Per-partition offset
Effect of one slow job None on others Blocks the partition behind it
Max useful consumers Unbounded One per partition
Retry with delay Native (DLX + TTL, delayed exchange) Manual: retry topics per delay tier
Dead-letter Built in Build it: a separate topic plus your own routing
Per-message priority Yes (x-max-priority) No
Replay after a bug Only from a DLQ copy Native — rewind the offset
Ordering guarantee Per queue, single consumer Per partition key, strong
Throughput ceiling ~50k msg/s per queue typical Millions/s, horizontally
Message retention after ack Removed Kept for the retention window

The rows that decide most job workloads are the first three. Background jobs have wildly variable durations — a 50ms webhook and a 40-second PDF render in the same stream — and per-message acknowledgement is what stops the slow one from delaying everything behind it.

Step 3 — See What Retries Cost on Kafka

Kafka has no per-message retry. The standard workaround is a ladder of retry topics, each with a consumer that waits before republishing.

// A Kafka retry ladder: one topic per delay tier, plus a terminal DLQ topic.
@KafkaListener(topics = "jobs.receipts", groupId = "receipts")
public void handle(ConsumerRecord<String, String> rec, Acknowledgment ack) {
    try {
        process(rec.value());
    } catch (TransientException e) {
        int attempt = attemptOf(rec) + 1;
        String next = switch (attempt) {
            case 1 -> "jobs.receipts.retry.5s";
            case 2 -> "jobs.receipts.retry.1m";
            case 3 -> "jobs.receipts.retry.10m";
            default -> "jobs.receipts.dlq";
        };
        // Republish elsewhere so THIS partition's offset can advance.
        producer.send(new ProducerRecord<>(next, rec.key(), rec.value()));
    }
    ack.acknowledge();   // always advance; the retry lives in another topic now
}

Three consequences follow. Ordering is lost for any message that retries, because it rejoins the stream later and on a different topic. Every delay tier needs its own topic, consumer group, and dashboards. And the ack.acknowledge() on the failure path means the original partition advances even though the work has not succeeded — correct here, but it makes "offset lag" a much weaker health signal than queue depth. On RabbitMQ the same behaviour is a queue argument:

channel.queue_declare(queue="receipts", durable=True, arguments={
    "x-dead-letter-exchange": "dlx",
    "x-dead-letter-routing-key": "receipts.failed",
    "x-message-ttl": 600_000,
})

Step 4 — Check the Scaling Ceiling

Kafka consumer parallelism is capped by partition count. Twelve partitions means at most twelve useful consumers; the thirteenth idles. Repartitioning is possible but disruptive — it changes key-to-partition mapping and therefore ordering.

# Kafka: consumers cannot exceed partitions
kafka-topics.sh --describe --topic jobs.receipts | grep PartitionCount
# PartitionCount: 12  → 12 is your maximum concurrency for this topic

# RabbitMQ: concurrency is just consumer count
rabbitmqctl list_queues name consumers messages_ready
# receipts  47  120390     → scale to 200 consumers if the downstream allows

For a job system that scales elastically with backlog — the horizontal worker scaling model — this ceiling is the sharpest edge. You must provision partitions for peak concurrency in advance, which means over-provisioning at all other times.

Step 5 — Use the Right Tool per Workload

Which workloads suit a log and which suit a queue Two columns. Kafka suits high-volume event streams, ordered per-key processing, replayable pipelines, and multiple independent consumer groups reading the same data. RabbitMQ suits variable-duration background jobs, per-message retry and priority, elastic consumer scaling, and workloads needing a dead-letter queue out of the box. Pick per workload, not per platform Kafka fits · High-volume event streams (100k+/s) · Strict ordering per key · Replay after a consumer bug · Many consumer groups on one stream · Uniform, short processing times RabbitMQ fits · Variable job durations in one stream · Per-message retry, delay and priority · Elastic consumer scaling with backlog · Dead-lettering without extra machinery · Complex routing (topic, fanout, headers) A common good answer: events on Kafka, jobs triggered by those events on a queue.

The hybrid is often the honest answer. Kafka carries the event stream; a small consumer translates the events that need work into jobs on a queue that has per-message semantics. You keep replay and ordering where they matter and independent completion where that matters.

Step 6 — Weigh the Operational Cost

Kafka's operational surface is larger: brokers, controllers or ZooKeeper, partition rebalancing, consumer-group coordination, retention and compaction policy, and a rebalance storm every time a consumer group becomes unstable. RabbitMQ's is smaller but not trivial — quorum queues, memory watermarks, and partition handling across a multi-node broker all need attention.

The decisive question is whether you are adding a system or reusing one. Adding Kafka for jobs alone is rarely justified. Reusing an existing, well-run Kafka for a job workload whose shape actually suits a log — uniform durations, ordering per key, no per-message retry needs — can be entirely reasonable, and saves a second system to operate.

Step 7 — Cost the Migration Before Committing

If the analysis still favours a move, price it as a project rather than a configuration change. The work is rarely where teams expect: the publish call is a one-line change, and everything around it is not.

Producer changes ........... small — swap the client, add a partition key
Consumer changes ........... medium — offset commits, manual ack semantics
Retry ladder ............... large — 3–4 topics, consumers, dashboards, alerts
Dead-letter path ........... medium — topic, routing, replay tooling
Priority / delay features .. large — no native equivalent; redesign required
Scheduled jobs ............. large — Kafka has no delayed delivery at all
Observability rebuild ...... medium — lag replaces depth; every alert changes
Runbook rewrite ............ medium — rebalances, partition skew, retention

Two items deserve extra scrutiny. Scheduled and delayed jobs have no Kafka equivalent, so anything relying on scheduled and delayed jobs needs a separate scheduler that publishes at the right moment — a component you did not previously operate. And the observability rebuild is deceptively large: every dashboard, alert threshold, and runbook step written against queue depth has to be re-derived against consumer lag, which behaves differently under the retry ladder because failed work advances the offset anyway.

Run the migration on one workload first, chosen because it genuinely suits a log — uniform durations, ordering per key, low retry rate. If that workload does not feel better on Kafka after a month of production, the rest will not either, and you will have learned it at the cost of one topic instead of a quarter.

A Kafka retry ladder, built by hand A message failing on the main topic is republished to a five-second retry topic, then a one-minute topic, then a ten-minute topic, and finally a dead-letter topic. Each hop is a separate topic with its own consumer group and dashboards, and the message loses its position in the original partition ordering. What RabbitMQ gives you as a queue argument jobs.receipts main topic retry.5s own consumer retry.1m own dashboards retry.10m own alerts dlq topic Ordering is lost at the first hop. Each topic is a component to operate. Offset lag stops meaning "work pending".

Verification

  1. Measure head-of-line blocking honestly. Replay a production sample onto a test Kafka topic and record p99 end-to-end latency; compare it against the same sample on a queue.
  2. Count required partitions. Take peak concurrency from your current consumer count. If it exceeds a partition count you are willing to run permanently, Kafka is the wrong fit.
  3. Prototype the retry ladder. Build the retry topics and confirm the operational load — dashboards, alerts, and lag per tier — is acceptable before committing.
  4. Test the rebalance. Kill a consumer under load; measure how long the group is stalled. On a job system that pause is a latency spike.

Gotchas & Edge Cases

Offset commit semantics are not acknowledgement. Committing an offset means "I have read up to here", not "this work succeeded". Committing before processing loses work on a crash; committing after risks reprocessing. Both are at-least-once at best, so idempotency is required either way — see preventing duplicate job execution with idempotency.

Consumer-group rebalances stop the world. Every scale event, deploy, or crash triggers a rebalance during which the group processes nothing. Cooperative sticky assignment reduces but does not remove it.

Kafka retention is time-based, not consumption-based. A slow consumer can fall behind the retention window and silently skip records. Alert on lag against retention, not just on lag.

RabbitMQ classic mirrored queues are deprecated. Use quorum queues for durability; the failover behaviour and memory profile differ enough to require re-testing.

Partition skew concentrates load. Keying by tenant sends one large customer's entire stream to a single partition, so one consumer saturates while eleven idle. Kafka will not rebalance that for you — the key has to change, which means re-partitioning and losing the ordering the key existed to provide.

"We already run it" cuts both ways. An existing Kafka is cheaper to add a topic to and more expensive to debug when a job workload behaves unlike an event stream. Weigh the on-call cost, not just the provisioning cost.

Related