Diagnosing Sidekiq Memory Bloat

Growing worker memory is the most common Sidekiq operational complaint, and most of the time it is not a leak. This guide extends Sidekiq Performance Tuning, part of Backend Frameworks & Worker Scaling, to diagnosing which of the three causes you actually have.

Problem Statement

Your Sidekiq processes start at 220MB RSS and reach 1.4GB within eight hours, at which point the container hits its memory limit and is OOM-killed — taking every in-flight job with it. A nightly restart hides the symptom, but the restart occasionally lands mid-job and produces duplicate side effects. You need to know whether this is heap fragmentation, retained objects, or simply a workload that genuinely needs that much memory, because the fix differs completely for each.

Prerequisites

  • Sidekiq 6.5+ with the concurrency setting visible, and Ruby 3.0+.
  • Container memory limits and the ability to read RSS over time per process.
  • A staging environment where you can replay production job mix.
  • derailed_benchmarks or memory_profiler available in that environment.

Step 1 — Distinguish the Three Causes

Cause Signature Fix
Heap fragmentation RSS grows then plateaus; Ruby heap stays flat Switch to jemalloc
Retained objects (leak) Both RSS and live object count grow without bound Find and release the retained references
Legitimate high-water mark RSS jumps with a specific job class, then holds Reduce per-job allocation or accept it
Three shapes of Sidekiq memory growth Three RSS curves over twelve hours. Fragmentation rises steeply at first and then flattens well below the container limit. A genuine leak rises steadily and crosses the limit. A workload high-water mark steps up sharply when a large job runs and then stays flat at the new level. The shape of the curve names the cause 1.5G 0 container limit fragmentation — plateaus leak — crosses the limit high-water mark — steps and holds 0h 12h Plot RSS alongside live object count: if RSS grows and objects do not, it is fragmentation.

Step 2 — Rule Out Fragmentation First

Ruby's default allocator fragments badly under the allocate-and-free pattern that a job queue produces. Switching to jemalloc frequently removes 30–50% of apparent bloat with no code change at all, so it is the cheapest thing to try.

# Dockerfile — install jemalloc and preload it
RUN apt-get update && apt-get install -y libjemalloc2 && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
ENV MALLOC_CONF="dirty_decay_ms:1000,muzzy_decay_ms:1000"
# Confirm it is actually loaded — this is the step people skip
ruby -e 'puts `cat /proc/#{Process.pid}/maps`.include?("jemalloc")'

If RSS drops substantially and then plateaus, the problem was fragmentation and you are done. If the curve still climbs without bound, continue.

Step 3 — Separate Retained from Transient Allocation

A leak retains objects across jobs. Measure the live heap between jobs rather than during one.

# config/initializers/sidekiq_memory.rb
require "objspace"

class MemoryProbeMiddleware
  def call(worker, job, queue)
    GC.start(full_mark: true, immediate_sweep: true)
    before = ObjectSpace.count_objects[:TOTAL] - ObjectSpace.count_objects[:FREE]
    rss_before = rss_kb

    yield

    GC.start(full_mark: true, immediate_sweep: true)
    after = ObjectSpace.count_objects[:TOTAL] - ObjectSpace.count_objects[:FREE]
    # Retained AFTER a full GC is the number that matters. Transient allocation
    # during the job is irrelevant — it is collected.
    StatsD.gauge("sidekiq.retained_objects", after - before,
                 tags: ["job:#{worker.class.name}"])
    StatsD.gauge("sidekiq.rss_delta_kb", rss_kb - rss_before,
                 tags: ["job:#{worker.class.name}"])
  end

  private

  def rss_kb
    File.read("/proc/self/status")[/VmRSS:\s+(\d+)/, 1].to_i
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware { |chain| chain.add MemoryProbeMiddleware }
end

Run this on a sample of jobs, not all of them: a full GC per job is expensive. A job class that consistently retains thousands of objects after collection is your leak.

Step 4 — Find What Is Holding the References

# In a staging console, replay the suspect job and dump what survived
require "memory_profiler"

report = MemoryProfiler.report do
  100.times { SuspectJob.new.perform(sample_args) }
end
report.pretty_print(to_file: "suspect_job.txt", retained_strings: 20)

The usual culprits in Ruby job code are narrow and recur constantly:

  • Class-level accumulators. @@seen ||= [] or a memoised hash on the class keeps growing for the life of the process.
  • Global caches without bounds. A Hash used as a cache with no eviction, particularly one keyed by record ID.
  • Logger or instrumentation buffers. Subscribers that append events and are never flushed.
  • Connection or client objects created per job and registered somewhere global, so each job adds one.
  • Symbols or string interning from untrusted input — to_sym on user data grows a table that is never collected.

Step 5 — Understand How Concurrency Multiplies Memory

Sidekiq runs jobs as threads in one process, so all threads share one heap. Concurrency does not multiply the base image, but it does multiply peak transient allocation.

def peak_rss_estimate(base_mb, per_job_peak_mb, concurrency)
  # Threads share the heap, so the base is paid once — but the peak is
  # concurrent, and Ruby's heap does not shrink back readily.
  base_mb + per_job_peak_mb * concurrency
end

peak_rss_estimate(220, 40, 10)   # 620 MB
peak_rss_estimate(220, 40, 25)   # 1220 MB — the same job mix, tighter budget

This is why raising concurrency to improve throughput so often produces OOM kills a week later: the high-water mark rises with concurrency and the heap does not return the memory. If one job class allocates heavily, isolate it on a lower-concurrency process rather than lowering concurrency everywhere.

Step 6 — Bound Memory Without Hiding the Cause

Graceful memory-triggered restart versus an OOM kill The upper timeline shows memory climbing until the kernel OOM-kills the process, losing every in-flight job with no drain. The lower timeline shows a watchdog noticing the threshold first, sending TSTP to stop fetching, waiting for in-flight jobs to finish, and exiting cleanly so the supervisor starts a fresh process. Restart on your terms, not the kernel's no watchdog memory climbing, 25 jobs in flight OOM kill — 25 lost no drain, no ack, 25 redeliveries and their duplicate side effects with a watchdog memory climbing TSTP · drain clean exit zero jobs lost; the supervisor starts a fresh process Set the watchdog threshold ~15% below the container limit so the drain has room. 
# A memory watchdog: stop fetching, let in-flight jobs finish, then exit.
Thread.new do
  limit_kb = Integer(ENV.fetch("SIDEKIQ_MAX_RSS_KB", "1_200_000"))
  loop do
    sleep 30
    rss = File.read("/proc/self/status")[/VmRSS:\s+(\d+)/, 1].to_i
    next if rss < limit_kb
    Sidekiq.logger.warn("RSS #{rss}KB over #{limit_kb}KB — draining")
    Process.kill("TSTP", Process.pid)     # stop fetching new jobs
    sleep 30                              # let in-flight work finish
    Process.kill("TERM", Process.pid)     # graceful shutdown
    break
  end
end

Restarting is a mitigation, not a fix. Keep the retained-object metric from step 3 in a dashboard so the underlying leak stays visible after the restart makes it survivable — otherwise the workaround becomes permanent and the next concurrency increase rediscovers it.

Step 7 — Reduce Allocation in the Jobs Themselves

Once the cause is a genuine high-water mark rather than a leak, the fix is allocating less. Three patterns account for most of it in job code.

Loading whole result sets is the largest single cause: Order.where(...).each materialises every row before iterating. find_each batches at 1,000 by default and keeps peak memory flat regardless of table size. The same applies to file processing — File.read on a 200MB export costs 200MB, while File.foreach costs a line.

Serialisation is the second. Parsing a large JSON payload creates a full object graph that survives as long as any reference to it does; if only three fields are needed, extract them and let the parsed structure go out of scope before the expensive part of the job begins. This is also an argument for keeping payloads small in the first place, as in the claim-check pattern — a job that fetches only the fields it needs never allocates the rest.

The third is retained per-job state: instance variables on a worker object are fine, because Sidekiq creates a new instance per job, but anything assigned to a constant, a class variable, or a thread-local outlives the job and accumulates across the process's whole lifetime.

Concurrency multiplies the peak, not the base Bars showing estimated peak RSS at concurrency ten, fifteen, twenty and twenty-five. The base image cost is shared across threads and stays constant, while per-job peak allocation multiplies with concurrency. At twenty-five threads the estimate crosses the container memory limit. Estimated peak RSS by concurrency container limit 1.2 GB 10 15 20 25 over the limit — OOM shared base (paid once) per-job peak x concurrency

Verification

  1. RSS plateaus. After the fix, RSS over twelve hours should flatten below the container limit rather than trending upward.
  2. Retained objects are flat. The per-job retained metric should hover near zero for every job class.
  3. No OOM kills. Container restart reasons should show clean exits or none, not OOMKilled.
  4. Concurrency is safe. Raise concurrency by 50% in staging and confirm the peak estimate from step 5 still fits the limit.

Gotchas & Edge Cases

RSS includes shared pages. Multiple Sidekiq processes on one host share copy-on-write memory, so summing RSS overstates actual usage. Use PSS when sizing hosts.

A nightly restart masks everything. It also makes the leak undiagnosable, because the curve never has time to reveal its shape. Diagnose in staging with restarts disabled.

Killing on memory without draining loses jobs. A watchdog that sends KILL, or a container limit reached without warning, discards in-flight work — the same problem as an abrupt deploy, covered in graceful shutdown & worker deployments.

Ruby's heap rarely shrinks. Once the process has grown to serve a peak, it holds that memory even when idle. A plateau at a high level after a genuine spike is therefore expected behaviour, not evidence of a leak — judge by whether it keeps climbing, not by the absolute number.

GC.compact is not a general fix. It can help fragmentation on Ruby 3.x, but it is expensive and pauses the process; measure before adopting it, and never call it per job.

Memory growth can be a queue-mix change, not a code change. A process whose RSS jumped after a deploy that touched nothing memory-related is usually now receiving a different job mix — a new task class routed onto its queue, or a shift in argument sizes. Check what changed in routing before profiling the code; the fix may be a queue split rather than an allocation fix.

Redis client buffers grow with large payloads. A very large job argument inflates the client's buffer, and with a big connection pool that is multiplied — another reason to keep payloads small, and to size the pool per tuning the Sidekiq Redis connection pool.

Related