Sidekiq Unique Jobs and Deduplication

Rails applications enqueue the same Sidekiq job many times without meaning to: a model callback fires on every save, a user double-clicks, a webhook is retried by its sender. Unique-job locks drop the duplicates before they run. This guide sets them up correctly and explains where they stop helping, as part of Sidekiq Performance Tuning in Backend Frameworks & Worker Scaling.

Problem Statement

A Rails application enqueues SyncAccountToCrmJob from an after_commit callback on Account. A bulk import touches each account five to ten times in a few seconds, so each account is synced up to ten times, the CRM's rate limit is exhausted, and the queue backs up for hours. A developer added sidekiq-unique-jobs with lock: :until_executed, which fixed the duplicates, but a week later some accounts stopped syncing entirely: after a deploy killed workers mid-job, their locks were never released. You want duplicates collapsed, locks that cannot block a job forever, and a design that stays correct when a duplicate does get through.

Prerequisites

  • Sidekiq 7 with either Sidekiq Enterprise (built-in unique jobs) or the open-source sidekiq-unique-jobs gem (v8).
  • A clear idea of what "the same job" means for each job class — usually the class plus some or all arguments.
  • Knowledge of each job's typical and worst-case duration.

Step 1 — Decide When the Lock Should Hold

A uniqueness lock is a Redis key derived from the job's class, queue, and arguments. The main design choice is how long it holds, which determines which duplicates are rejected:

Sidekiq unique lock types on a job timeline A job's life has three phases: waiting in the queue, running, and done. until_executing holds the lock while the job waits, so a duplicate enqueued while it waits is dropped but one enqueued during execution runs afterwards. until_executed holds it through waiting and running. while_executing allows duplicates to be enqueued but only one runs at a time. until_and_while_executing combines a queue lock with a separate runtime lock. What each lock type covers waiting in queue running done until_executing until_executed while_executing until_and_while_executing rejects duplicate enqueues serialises execution

For the CRM sync, until_executing is usually the right choice: while a sync is waiting, further enqueues are redundant because the job will read the latest data when it runs. Once it has started, a new change should trigger another sync, or the latest edits would be missed. until_executed is the tempting default but it drops updates made during execution, and it holds the lock for the longest time, which is what made the stuck-lock problem so damaging.

Step 2 — Configure Unique Jobs

Sidekiq Enterprise enables uniqueness per job with a TTL in seconds:

# config/initializers/sidekiq.rb
Sidekiq::Enterprise.unique! unless Rails.env.test?

class SyncAccountToCrmJob
  include Sidekiq::Job
  sidekiq_options queue: :crm, unique_for: 10.minutes, unique_until: :start

  def perform(account_id)
    CrmSync.new(Account.find(account_id)).call
  end
end

unique_until: :start corresponds to until_executing; the default (:success) holds until the job succeeds. unique_for is a hard TTL, so a lock never outlives it even if the worker that held it disappears.

sidekiq-unique-jobs (open source) uses the lock option:

class SyncAccountToCrmJob
  include Sidekiq::Job
  sidekiq_options queue: :crm,
                  lock: :until_executing,
                  lock_ttl: 10.minutes.to_i,
                  on_conflict: :log

  def perform(account_id) = CrmSync.new(Account.find(account_id)).call
end

Install its client and server middleware as the gem's README describes, and enable the reaper (Step 4). on_conflict: :log records dropped duplicates, which is useful when you first roll out locks to confirm they are dropping what you expect.

Step 3 — Define Uniqueness by the Right Arguments

By default the lock digest includes all arguments. If a job receives a timestamp or a request ID along with the account ID, every enqueue is "unique" and nothing is deduplicated. Restrict the digest to the arguments that identify the work:

class SyncAccountToCrmJob
  include Sidekiq::Job
  sidekiq_options lock: :until_executing, lock_ttl: 600,
                  lock_args_method: ->(args) { [args.first] }   # account_id only

  def perform(account_id, reason = nil, requested_at = nil) = ...
end

Sidekiq Enterprise computes uniqueness from the class, queue, and arguments; keep incidental data out of the arguments, or pass it through a wrapper that looks it up. Scheduled jobs are included too: perform_in(5.minutes, id) holds the lock from enqueue until the job starts (or until the TTL), which is a simple debounce pattern — enqueue with a delay, and bursts of changes collapse into one run.

Debouncing a burst with a delayed unique job During an import, an account is updated eight times in a few seconds. Each update tries to enqueue a sync job for the same account with a five-minute delay. The first attempt acquires the uniqueness lock; the next seven find the lock held and are dropped. After five minutes one sync runs, reading the account's final state, and the lock is released when it starts. Eight updates, one sync 1 locked, 7 dropped (within ~3 s) perform_in 5 minutes, lock held one sync runs reads final account state acquired lock duplicate dropped

Step 4 — Make Locks Expire and Clean Up Orphans

Every lock needs a TTL longer than the job's worst-case wait plus runtime but short enough that a lost lock heals itself. For until_executing with a five-minute delay and a typical queue wait under a minute, 10 minutes is enough; for until_executed on a job that can run 30 minutes, the TTL must cover that too.

Locks are orphaned when a worker is killed without running Sidekiq's cleanup — kill -9, OOM, a node failure — or when a job is deleted from the Web UI. Sidekiq Enterprise relies on the TTL. sidekiq-unique-jobs also provides a reaper that finds digests with no matching job in any queue, schedule, retry set, or process and deletes them:

SidekiqUniqueJobs.configure do |config|
  config.reaper          = :ruby   # or :lua
  config.reaper_count    = 1000
  config.reaper_interval = 600     # seconds
  config.reaper_timeout  = 10
end

Keep the reaper enabled in production. Monitor the number of lock digests in Redis; it should rise and fall with traffic, not grow steadily. And make sure your graceful shutdown gives jobs time to finish, so fewer locks are orphaned in the first place.

Step 5 — Keep the Job Idempotent Anyway

Uniqueness reduces duplicates; it does not guarantee they never happen. A lock can expire while a slow job is still waiting; Redis failover can lose a freshly written lock; a retried job is, by design, the same job running again. So the job itself must tolerate running twice:

def perform(account_id)
  account = Account.find(account_id)
  return if account.crm_synced_version == account.lock_version   # nothing new

  CrmSync.new(account).call
  account.update_columns(crm_synced_version: account.lock_version)
end

Idempotency keys, version checks, and upserts are covered in general terms in preventing duplicate job execution with idempotency. Treat uniqueness as a performance optimisation — fewer wasted runs, less pressure on the CRM — and idempotency as the correctness guarantee.

Step 6 — Measure What the Locks Drop

Once locks are live, measure their effect rather than assuming it. Count three things per job class: enqueue attempts, attempts dropped as duplicates, and jobs actually executed. A client middleware placed before the uniqueness middleware sees every attempt, and perform_async returning nil marks a drop:

jid = SyncAccountToCrmJob.perform_in(5.minutes, account.id)
StatsD.increment("jobs.enqueue", tags: ["class:crm_sync", "dropped:#{jid.nil?}"])
A day of CRM sync jobs, attempt to effect Of 48,000 enqueue attempts in a day, the uniqueness lock drops 41,000 as duplicates, leaving 7,000 jobs that execute. The idempotency check inside the job skips a further 100 that find nothing new, so 6,900 syncs reach the CRM. The lock removes most of the load; the idempotency check catches the rest. One day of CRM sync jobs 48,000 enqueue attempts 7,000 executed (41,000 dropped by the lock) 6,900 reached the CRM (100 skipped as unchanged) Bar widths are proportional to counts.

A drop rate near zero means the digest includes an argument that varies; a drop rate near 100% for a long time can mean a stuck lock is rejecting everything. Alert when executions for a job class fall to zero while attempts continue — that is the signature of an orphaned lock, and it is exactly the failure that stopped accounts syncing in the problem statement.

Verification

  • Enqueue the same job ten times in a loop: one job appears in the queue (or schedule), and nine conflicts are logged.
  • Enqueue a job, let it start, then enqueue again: with until_executing, the second job is accepted.
  • kill -9 a worker mid-job: the lock expires within the TTL (Enterprise) or is reaped within the reaper interval, after which enqueues are accepted again.
  • The count of lock keys in Redis stays bounded over a day.
  • During a bulk import, CRM API calls per account drop to one or two.

Gotchas & Edge Cases

Testing environments. Unique locks persist in the test Redis between examples and cause confusing failures. Disable uniqueness in tests or flush Redis between them, and test the dedup behaviour explicitly in one integration spec.

Retries keep the lock. With until_executed, a job in the retry set still holds its lock, so new enqueues are dropped for hours while retries back off. Another reason to prefer until_executing or a short TTL.

Argument types change the digest. perform_async(42) and perform_async("42") produce different digests. Normalise arguments before enqueueing.

Batches and uniqueness. Jobs dropped as duplicates are not added to a Sidekiq Pro batch, so batch callbacks can fire earlier than expected. Avoid unique locks on jobs inside batches, or account for it — see Sidekiq batch jobs and workflows.

FAQ

Is sidekiq-unique-jobs safe to use in production? Yes, with a TTL on every lock and the reaper enabled. Most production problems come from until_executed locks without TTLs and from upgrading between major versions without running the migration it provides.

Can I deduplicate across different job classes? Not with the built-in options, which include the class in the digest. Use your own SET NX key with a shared name, as in deduplicating jobs with Redis SET NX keys.

What does a dropped duplicate return to the caller? perform_async returns nil instead of a job ID. Code that stores the returned JID should handle that case.

Should the debounce delay be long or short? Long enough to cover a typical burst of changes, short enough that users do not notice the lag. For a CRM sync, one to five minutes is common; for a search-index update users expect to see quickly, 5–30 seconds. Measure how long bursts last in your import logs and pick a delay slightly above the p90.

Related