Sizing Redis Memory for Queue Backlogs

A Redis-backed queue trades durability characteristics for speed, and the bill arrives as memory. This guide puts numbers on the trade described in In-Memory vs Persistent Queue Storage, part of Backend Frameworks & Worker Scaling.

Problem Statement

Your Redis instance has 4GB and normally uses 300MB. During a two-hour downstream outage the workers stopped acknowledging, the backlog grew to 900,000 messages, Redis hit maxmemory, and — because the policy was allkeys-lru — it evicted queued jobs to make room. Nobody noticed until customers reported missing exports. You need to know how much memory the queue can actually need, and to make the failure mode "reject new work" rather than "silently delete queued work".

Prerequisites

  • Access to INFO memory and MEMORY USAGE on the Redis instance.
  • Message-size statistics: median and p99 serialised payload bytes.
  • Arrival rate per queue, and a realistic worst-case outage duration.
  • The ability to change maxmemory-policy, which requires understanding what else shares the instance.

Step 1 — Measure Real Per-Message Cost

The wire size of your payload is not the memory cost. Redis adds structural overhead per entry, and the data structure matters.

# Real bytes for one queued job, structure overhead included
redis-cli MEMORY USAGE celery                     # whole list
redis-cli LLEN celery                             # entries
# → 184320000 bytes / 512000 entries ≈ 360 B per 280 B payload
def bytes_per_message(payload_bytes: int, structure: str = "list") -> int:
    """Payload plus Redis structural overhead. Measure to confirm — these are
    typical values for 64-bit Redis 7 with default encodings."""
    overhead = {
        "list": 60,          # quicklist node share + ziplist header amortised
        "zset": 110,         # skiplist node + score + hash entry
        "hash": 90,          # hash entry + key
        "stream": 140,       # entry + consumer-group bookkeeping
    }[structure]
    return payload_bytes + overhead

bytes_per_message(280)                  # 340 B  — plain list
bytes_per_message(280, "zset")          # 390 B  — a delayed or priority queue
bytes_per_message(4096, "hash")         # 4186 B — a payload-in-hash design

Sorted sets — used by BullMQ, by delayed jobs, and by priority queues — cost roughly twice a list's overhead per entry. That is invisible at a thousand messages and decisive at a million.

Step 2 — Compute the Worst-Case Backlog

Sizing for steady state guarantees an outage takes the instance down. Size for the backlog that accumulates while nothing is being consumed.

def worst_case_backlog_gb(arrival_per_sec: float, outage_hours: float,
                          bytes_per_msg: int, safety: float = 1.5) -> float:
    messages = arrival_per_sec * 3600 * outage_hours
    return messages * bytes_per_msg * safety / 1024**3

worst_case_backlog_gb(150, 2, 340)      # 0.51 GB — two-hour outage at 150/s
worst_case_backlog_gb(150, 8, 340)      # 2.05 GB — an overnight failure
worst_case_backlog_gb(1500, 2, 390)     # 5.9 GB  — high volume, sorted set

The outage duration is a business decision disguised as an engineering one: it is how long you are willing to accumulate work rather than shed it. Two hours is a reasonable default for a system with on-call coverage; eight is right if failures can start on a Friday evening.

Memory during an outage, and where the limits sit Memory usage starts at a low steady state and climbs linearly once consumers stop acknowledging. The chart marks the point where the instance reaches maxmemory and begins evicting under an LRU policy, and shows the headroom that a correctly sized instance keeps above the worst-case backlog. Size for the outage, not for Tuesday afternoon 4 GB 0 maxmemory — evictions begin worst-case backlog steady state 300 MB consumers stop under allkeys-lru, queued jobs are deleted here With noeviction, writes fail instead — an error you can see and alert on.

Step 3 — Never Let a Queue Instance Evict

This is the single most important setting on a Redis instance backing a queue.

# Queue instances MUST refuse writes rather than delete data.
redis-cli CONFIG SET maxmemory-policy noeviction
redis-cli CONFIG SET maxmemory 6gb
Policy Behaviour at the ceiling Suitable for a queue?
noeviction New writes fail with an error Yes — the only correct choice
allkeys-lru Deletes any key, including queued jobs No — silent data loss
volatile-lru Deletes only keys with a TTL No — dedup keys vanish first
allkeys-random Deletes arbitrary keys No

noeviction turns a silent failure into a visible one: producers get an OOM command not allowed error, which surfaces as a failed enqueue you can retry, alert on, and reason about. Do not share a queue instance with a cache that needs an eviction policy — the policy is instance-wide, and one of the two workloads will get the wrong behaviour.

Step 4 — Leave Headroom Above the Number

maxmemory is not the instance size. Three things consume memory that the calculation above does not cover.

def instance_size_gb(dataset_gb: float, has_replica: bool = True,
                     bgsave_enabled: bool = True) -> float:
    size = dataset_gb
    size *= 1.25 if bgsave_enabled else 1.0     # copy-on-write during a fork
    size += 0.5                                 # client buffers, replication backlog
    size *= 1.15                                # allocator fragmentation
    return round(size, 1)

instance_size_gb(2.0)     # 3.6 GB for a 2 GB dataset

The copy-on-write factor is the one that surprises people: during a background save, pages modified while the fork is running are duplicated, so a write-heavy instance can briefly need substantially more than its dataset size. On a queue that is draining a large backlog, nearly every page is being modified.

Step 5 — Alert Before the Ceiling, Not At It

# Memory headroom as a fraction — page well before it reaches zero
1 - (redis_memory_used_bytes / redis_memory_max_bytes) < 0.25

# Fragmentation ratio: above ~1.5 means real memory far exceeds the dataset
redis_memory_used_rss_bytes / redis_memory_used_bytes > 1.5

# Rejections mean producers are already failing
rate(redis_rejected_connections_total[5m]) > 0
- alert: RedisQueueMemoryHeadroomLow
  expr: 1 - (redis_memory_used_bytes / redis_memory_max_bytes) < 0.25
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "Redis queue instance under 25% headroom — backlog may be growing"
    runbook: "https://runbooks.internal/redis-queue-memory"

Twenty-five percent headroom sounds generous and is not: at 150 messages per second and 340 bytes each, a 1GB gap is filled in about five and a half hours — and much faster during the burst that usually accompanies a recovery.

Step 6 — Have a Plan for the Ceiling

What to do as memory runs out Four mitigation steps ordered by cost. First, drain faster by scaling consumers. Second, shed low-priority producers so arrivals slow. Third, offload payloads to object storage using the claim-check pattern. Fourth, spill the backlog to durable storage and replay it later. Each step is annotated with what it costs. Mitigations, cheapest first 1 · Scale consumers — drain faster Costs: more load on whatever the workers call. Useless if that is the thing that is down. 2 · Shed low-priority producers — slow arrivals Costs: deferred work. Needs a producer-side switch that already exists. 3 · Offload payloads to object storage (claim check) Costs: a code change — prepare it before the incident, not during. 4 · Spill the backlog to durable storage and replay later Costs: bespoke tooling and a replay window. The last resort, and it must be rehearsed.

Steps three and four are not incident-time actions unless they were built beforehand. The realistic pre-work is to keep payloads small — the claim-check pattern applied before you need it — and to have a producer-side kill switch for bulk work that can be flipped without a deploy.

Step 7 — Decide Whether Redis Is Still the Right Store

At some backlog size the calculation stops favouring memory. A million messages at 340 bytes is 340MB and entirely reasonable; fifty million is 17GB and an expensive way to hold a queue that a disk-backed broker would carry for a fraction of the cost.

The crossover depends on three things: how large the backlog can legitimately get, how much the durability difference matters, and whether the throughput actually requires in-memory speed. A queue that peaks at 100,000 messages and needs sub-millisecond enqueue is squarely Redis territory. One that regularly holds millions of messages for hours, at a few thousand operations per second, is better served by RabbitMQ with lazy queues or by SQS, both of which page to disk without you sizing for it.

The intermediate option worth knowing about is Redis Streams with capped length, or a two-tier design where Redis holds the working set and a durable store holds the overflow. Both add complexity; both are cheaper than provisioning memory for a backlog you hope never happens. The full comparison of these trade-offs is in in-memory vs persistent queue storage and Redis persistence: AOF vs RDB for queues.

Dataset size is not instance size A stacked bar. The two-gigabyte dataset is the base. Copy-on-write during a background save adds five hundred megabytes. Client and replication buffers add five hundred more. Allocator fragmentation adds a further fifteen percent, bringing the required instance to about three point six gigabytes. A 2 GB dataset needs a 3.6 GB instance dataset 2.0 GB COW 0.5 buffers 0.5 frag 15% provision 3.6 GB, set maxmemory below it Copy-on-write bites hardest while draining a backlog, when nearly every page is being modified. Size against used_memory_rss, not used_memory.

Verification

  1. Measured cost matches the model. Enqueue 100,000 real messages and compare MEMORY USAGE against the per-message estimate; a large discrepancy means an unexpected encoding.
  2. The ceiling rejects rather than evicts. In staging, fill to maxmemory and confirm producers receive an OOM error and no queued job disappears.
  3. Headroom survives the worst case. Simulate the outage duration you sized for and confirm memory stays below the alert threshold.
  4. Fragmentation is bounded. After a large drain, mem_fragmentation_ratio should return under 1.5; persistently higher suggests activating activedefrag.

Gotchas & Edge Cases

used_memory excludes fragmentation. The number that matters for a container limit is used_memory_rss. Sizing against used_memory alone under-provisions by 15–40%.

Small-object encodings collapse under growth. Redis uses compact encodings for small collections and switches to the general form past a threshold, so memory per entry can jump sharply as a queue grows. Measure at realistic size, not at ten entries.

Lua scripts and client output buffers are not free. A large ZRANGE or a script returning thousands of entries builds its reply in memory before sending it, and several workers doing that concurrently can add hundreds of megabytes of transient usage that no dataset calculation predicts.

Replica buffers count. A replica that falls behind accumulates a replication backlog in the primary's memory. On a large drain this can be hundreds of megabytes.

Cluster mode does not divide a single queue. All keys for one queue share a hash slot and therefore one node, so cluster mode adds nodes without adding capacity for that queue. Partition the queue explicitly, as in partitioning queues by tenant ID.

Delayed and dead-letter sets are part of the total. Scheduled jobs, retry sets, and result backends all live in the same instance. Size for their peak too, not just the ready queue.

Related