Preventing Duplicate Scheduled Jobs with Leader Election

A scheduler is a singleton by nature, and every mechanism that keeps a fleet available — redundancy, rolling deploys, failover — works against that. This guide hardens the periodic-job model from Scheduled & Delayed Jobs, part of Queue Fundamentals & Architecture.

Problem Statement

Your nightly billing run sometimes charges customers twice. The scheduler runs as a single replica, but during a rolling deploy the new pod starts before the old one has finished terminating, and for four seconds two schedulers are live. If the tick falls in that window, the job is enqueued twice — and because both messages are valid, no downstream mechanism rejects them. You want exactly one scheduler active at any instant, and a second line of defence for when that guarantee is violated anyway.

Prerequisites

• Redis 6+ (or etcd/Consul) reachable from every scheduler replica.
• NTP-synchronised clocks; leases are time-based and skew shortens them.
• The ability to make enqueue idempotent — the fallback that makes a lease failure survivable.
• A metrics pipeline, so "how often did leadership change hands" is answerable.

Step 1 — Understand Why a Single Replica Is Not Enough

replicas: 1 is an eventual guarantee, not an instantaneous one. Kubernetes with a RollingUpdate strategy deliberately overlaps old and new pods; a node partition can leave a pod running that the control plane believes is gone; and a Recreate strategy removes the overlap but leaves a gap where no scheduler exists.

Step 2 — Acquire a Lease, Not a Lock

A lock without expiry deadlocks when its holder dies. A lease expires, so leadership transfers automatically — the cost is that a leader which pauses (GC, a stalled disk) can believe it still holds a lease that has already been granted elsewhere.

import os, time, uuid, redis

r = redis.Redis(decode_responses=True)
KEY, TTL = "scheduler:leader", 15          # seconds
ME = f"{os.uname().nodename}:{os.getpid()}:{uuid.uuid4().hex[:8]}"

RENEW = r.register_script("""
    if redis.call('GET', KEYS[1]) == ARGV[1] then
      return redis.call('EXPIRE', KEYS[1], ARGV[2])
    end
    return 0
""")

def try_become_leader() -> bool:
    # SET NX EX is the whole election: first writer wins, and the key expires
    # on its own if that writer dies.
    return bool(r.set(KEY, ME, nx=True, ex=TTL))

def renew_leadership() -> bool:
    # Compare-and-extend: never extend a lease someone else now holds.
    return bool(RENEW(keys=[KEY], args=[ME, TTL]))

def release_leadership() -> None:
    if r.get(KEY) == ME:
        r.delete(KEY)                       # hand over immediately on shutdown

Renew at roughly one third of the TTL. A 15-second lease renewed every 5 gives two chances to survive a transient Redis blip before leadership moves.

Step 3 — Run the Scheduler Loop Behind the Lease

def scheduler_loop(stop_event):
    leader = False
    while not stop_event.is_set():
        leader = renew_leadership() if leader else try_become_leader()
        if leader:
            fire_due_jobs()                 # only the leader enqueues
        else:
            metrics.standby_seconds.inc(1)
        stop_event.wait(TTL / 3)
    if leader:
        release_leadership()                # explicit handover beats waiting for TTL

Releasing on shutdown turns a 15-second gap into a sub-second one during a deploy, which is the difference between a missed minute-granularity tick and an unnoticed handover.

Step 4 — Make Each Tick Idempotent Anyway

Leader election reduces duplicate ticks; it cannot eliminate them, because a paused leader can wake after its lease expired and fire before noticing. The second line of defence is a per-tick key.

def fire_due_jobs() -> None:
    now = int(time.time())
    for job in due_jobs(now):
        # The tick identity: schedule + the exact slot it belongs to.
        slot = now - (now % job.interval_seconds)
        tick_key = f"tick:{job.name}:{slot}"
        # First writer wins; a duplicate tick from a stale leader is a no-op.
        if r.set(tick_key, ME, nx=True, ex=job.interval_seconds * 3):
            enqueue(job.task, args=job.args, headers={"tick": tick_key})
        else:
            metrics.duplicate_ticks.labels(job=job.name).inc()

This is the same deduplication idea as preventing duplicate job execution with idempotency, applied at enqueue time rather than at execution time. Counting the collisions is what makes an election problem visible instead of silent.

Step 5 — Handle the Fencing Problem

A leader that stalls for longer than the TTL may resume and act while a new leader is already active. Redis leases cannot prevent that on their own; a fencing token makes the stale action detectable downstream.

def try_become_leader_fenced():
    """Monotonic token: any consumer can reject work from an older leader."""
    if not r.set(KEY, ME, nx=True, ex=TTL):
        return None
    return r.incr("scheduler:epoch")        # strictly increasing per election

def enqueue_fenced(task, args, epoch: int):
    queue.publish(task=task, args=args, headers={"epoch": epoch})

# Consumer side: refuse anything from a superseded leader.
def handle(msg):
    if int(msg.headers["epoch"]) < int(r.get("scheduler:epoch")):
        log.warning("dropping work from a stale leader epoch")
        return
    process(msg)

For most job systems the per-tick key is sufficient and fencing is unnecessary complexity. Reach for fencing when a duplicate has consequences that deduplication cannot undo — money movement, external notifications, anything with a side effect outside your database.

Step 6 — Watch Leadership as a Signal

# Leadership churn and duplicate ticks — both should be near zero
sum(increase(scheduler_leadership_changes_total[1h])) > 5
sum(increase(scheduler_duplicate_ticks_total[1h])) > 0

Step 7 — Consider What the Platform Already Offers

Before building this, check whether your platform provides it. Kubernetes offers a Lease object in coordination.k8s.io with a well-tested client-go implementation; ZooKeeper and etcd have had election recipes for years; and several job frameworks — Sidekiq Enterprise, Quartz clustering, Celery with celerybeat-redis — ship their own leader mechanisms.

Using the platform's implementation is usually right. The one caution is understanding what it guarantees: nearly all of these are lease-based, so they carry the same stale-leader caveat as the code above, and none of them make your ticks idempotent. Adopting a library removes the election code, not the per-tick key.

The case for the hand-rolled version is dependency locality: if your schedulers already talk to Redis and nothing else, a fifteen-line lease avoids introducing a coordination service and its failure modes. Pick on that basis rather than on sophistication — an election that depends on a system your scheduler otherwise never touches has a new way to fail, and it will fail at the same moment everything else does.

Step 8 — Decide Between Duplicate and Missed

Every scheduler design chooses which failure it prefers, because you cannot have neither under partition. A lease with a short TTL fails toward missed ticks: leadership moves quickly, and a tick due during the gap does not fire. A lease with a long TTL fails toward duplicates: a stalled leader keeps its claim, resumes, and fires alongside its successor.

State the preference per schedule rather than globally, because the right answer differs sharply:

Recording the choice makes the alerting obvious. A schedule that prefers missed needs an alert on gap detection — compare the last fired slot against the expected sequence and page when one is absent. A schedule that prefers duplicate needs no such alert, and adding one only creates noise. Without the explicit decision, teams tend to build both alerts, ignore both, and still be surprised by whichever failure arrives.

Verification

1. Only one leader enqueues. Run three replicas; over ten minutes exactly one should log leadership and the duplicate-tick counter must stay at zero.
2. Failover is bounded. kill -9 the leader; another replica must take over within one TTL, and the missed-tick count should be at most one.
3. A rolling deploy produces no duplicates. Restart under load with a one-minute schedule; the tick key must prevent any duplicate enqueue during the overlap.
4. Renewals are comfortable. Log renewal latency; if p99 approaches a third of the TTL, the TTL is too short for the environment.

Gotchas & Edge Cases

Renewing a lease you no longer hold. A plain EXPIRE can extend another replica's lease. The compare-and-extend script is what prevents two leaders.

TTL shorter than a GC pause. A JVM or CPython pause longer than the TTL loses leadership silently, then resumes and acts. Size the TTL above your worst observed pause, and keep the tick key as the backstop.

Redis failover loses the key. With sentinel or cluster failover, an unreplicated lease key vanishes and two replicas can both acquire. Redlock across independent instances reduces this; idempotent ticks remove the consequence.

Missed ticks are the other failure. Election prevents duplicates but a failover can skip a tick entirely. If a schedule must never be skipped, track the last successful slot per job and catch up on acquisition rather than assuming continuity.

Cron drift on the standby. Standby replicas should not compute "due" times independently from local clocks — persist the last-fired slot centrally so a new leader continues from the same sequence.

Related

• Scheduled & Delayed Jobs — the scheduling model this protects.
• Cron-style scheduling with Celery Beat — the scheduler most often deployed as a fragile singleton.
• Preventing duplicate job execution with idempotency — the execution-side counterpart of the per-tick key.
• Graceful Shutdown & Worker Deployments — why deploys create the overlap in the first place.