Tuning the Sidekiq Redis Connection Pool

This guide drills into one knob from Sidekiq Performance Tuning, set within the wider Backend Frameworks & Worker Scaling context: the Redis connection pool that every Sidekiq thread shares, and why getting its size wrong is one of the most common production stalls.

The classic symptom is a log full of Waited 1.0 sec for connection or ConnectionPool::TimeoutError, while CPU and Redis itself sit nearly idle. Throughput plateaus far below what the worker count suggests it should be. The cause is almost always a connection pool too small for the concurrency: 30 threads contending for 5 connections means 25 threads block on every Redis call. Conversely, an oversized pool on the client side multiplied across web dynos can exhaust Redis's maxclients limit. This page explains the relationship between Sidekiq concurrency and pool size, gives the sizing formulas for the server (worker) and client (enqueuer) sides separately, shows the RedisConnection configuration, and covers how to confirm the pool is right.

Prerequisites

  • Sidekiq 7.x (the pool defaults changed across versions; the formulas below note the differences).
  • Knowledge of your worker concurrency setting (concurrency in sidekiq.yml) and your web/enqueuer process count.
  • Access to Redis to inspect CONFIG GET maxclients and INFO clients.
  • An understanding that the server (Sidekiq worker process) and the client (your web app or any process that only enqueues) have completely separate pools with different sizing needs.

Step 1 — Understand the concurrency-to-pool relationship

Each Sidekiq worker thread checks a connection out of the shared pool whenever it touches Redis — fetching a job, acking, scheduling a retry, or any Sidekiq.redis call in your job code. If the pool has fewer connections than active threads, threads queue for a connection and block up to the pool timeout (default 1 second) before raising.

The rule on the server side: the pool must be at least as large as concurrency, plus headroom for Sidekiq's internal threads (the heartbeat, scheduler, and any job code that itself uses Redis concurrently).

# A worker with concurrency 25 needs >= 25 connections just for job threads,
# plus a few for Sidekiq's own background threads. Under-sizing here is the
# #1 cause of ConnectionPool::TimeoutError under load.

In Sidekiq 7, the server pool defaults to concurrency + 5 when you do not set it explicitly, which is the sane baseline. You only need to override it if your job code opens additional concurrent Redis operations per job.

Step 2 — Apply the server pool sizing formula

For the worker process, size the pool to concurrency plus a fixed buffer. The buffer covers Sidekiq's internal threads and any incidental Redis use.

# config/initializers/sidekiq.rb
concurrency = ENV.fetch("SIDEKIQ_CONCURRENCY", 25).to_i

Sidekiq.configure_server do |config|
  config.concurrency = concurrency
  config.redis = {
    url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0"),
    # server pool: one connection per worker thread + buffer for internals
    size: concurrency + 5,
    pool_timeout: 1,   # seconds a thread waits for a free connection before raising
  }
end

If your jobs spawn their own threads that each call Sidekiq.redis, add those to the size. A job that fans out 4 parallel Redis calls across 25 concurrent jobs would need 25 + (25 * 3) + 5, not 25 + 5. Most jobs do not do this, so the simple formula holds.

Step 3 — Size the client pool separately and smaller

The client side — your Rails web process, a cron enqueuer, an API server — only pushes jobs. It does not run worker threads, so it needs far fewer connections: roughly one per concurrent request thread that enqueues, not per worker. Sizing the client pool like the server wastes Redis connections.

# config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0"),
    # client pool: match your web server's thread count (e.g. Puma threads),
    # NOT the worker concurrency. Enqueueing is brief, so a small pool suffices.
    size: ENV.fetch("RAILS_MAX_THREADS", 5).to_i,
    pool_timeout: 1,
  }
end

The total connections Redis sees is (server_pool * worker_processes) + (client_pool * web_processes). Account for every process type when checking against maxclients — a fleet of 20 web dynos each holding a 25-connection client pool is 500 connections doing almost nothing.

Step 4 — Use a shared RedisConnection / redis_pool for batched enqueues

When a single process enqueues large batches (see Sidekiq batches & job workflows), reuse one connection for the whole burst with Sidekiq.redis rather than checking out a connection per perform_async. This keeps the client pool from being exhausted by a tight enqueue loop.

# enqueue a large fan-out using a single pooled connection
Sidekiq.redis do |conn|
  conn.pipelined do |pipe|
    image_ids.each do |id|
      # build raw job payloads and push in one round trip
      pipe.lpush("queue:default", Sidekiq.dump_json(
        "class" => "ResizeImageJob", "args" => [id], "jid" => SecureRandom.hex(12)
      ))
    end
  end
end

Pipelining the enqueue collapses thousands of round trips into one, so the burst holds a single connection briefly instead of thrashing the pool.

Step 5 — Tune pool_timeout and align Redis maxclients

pool_timeout controls how long a thread waits for a connection before raising. Raising it hides under-sizing by trading errors for latency — fix the size first, then keep the timeout short so genuine exhaustion surfaces fast. On the Redis side, ensure maxclients exceeds your computed total connections with margin.

# check Redis's client ceiling and current usage
redis-cli CONFIG GET maxclients          # e.g. "10000"
redis-cli INFO clients | grep connected_clients

# raise the ceiling if your fleet's total pool approaches it
redis-cli CONFIG SET maxclients 20000    # also set in redis.conf to persist

Leave generous headroom: spikes during deploys (old and new processes briefly coexist) can momentarily double connection counts.

Verification

Confirm the pool is correctly sized rather than guessing.

Check connection counts before and after starting workers — the delta should match your formula:

# count connections from the Sidekiq host; should be ~ (concurrency + 5) per worker process
redis-cli INFO clients | grep connected_clients

Watch the logs under load for pool timeouts — their presence means the server pool is too small:

# any of these lines means the pool is undersized for the concurrency
bundle exec sidekiq 2>&1 | grep -E "ConnectionPool::TimeoutError|Waited .* for connection"

Assert the configured size programmatically:

# rails console on a worker process
pool = Sidekiq.redis_pool      # the ConnectionPool instance
puts pool.size                 # should equal concurrency + buffer
puts pool.available            # idle connections right now; 0 under saturation

If available sits at 0 while jobs are waiting, the pool is the bottleneck — increase size (and confirm Redis maxclients has room).

Gotchas & edge cases

  • Sizing the client pool like the server pool. Web processes only enqueue and need few connections; copying concurrency + 5 to the client config multiplies wasted connections across every web replica and can hit maxclients.
  • Forgetting per-process multiplication. The size is per process. Ten worker pods at concurrency + 5 each is ten times that many real Redis connections — easy to overshoot maxclients.
  • Raising pool_timeout to mask exhaustion. A longer timeout converts errors into silent latency, making the real undersizing harder to diagnose. Keep it short and fix size.
  • Job code that opens extra Redis connections. A job using a separate Redis client (a rate limiter, a cache) consumes connections outside Sidekiq's pool against the same maxclients. Budget for those too. Persistence settings also matter under load — see In-memory vs persistent queue storage.
  • Deploy-time connection spikes. Rolling deploys briefly run old and new processes together, doubling connection demand for seconds. Provision maxclients headroom so deploys don't trip the ceiling.

FAQ

What size should the Sidekiq Redis pool be? On the server (worker) process, concurrency + 5 is the standard baseline and the default in Sidekiq 7. On the client (web/enqueuer) process, size it to your web server's thread count (e.g. Puma RAILS_MAX_THREADS), which is usually far smaller than the worker concurrency.

Why am I getting ConnectionPool::TimeoutError when Redis looks healthy? The pool, not Redis, is the bottleneck. More threads are trying to check out connections than the pool holds, so they wait past pool_timeout and raise. Increase the pool size to at least concurrency + 5 and confirm Redis maxclients has room for the higher total.

Related