Cron-Style Scheduling with Celery Beat

This guide sets up reliable recurring schedules with Celery Beat, the cron counterpart to the one-shot delay queue in scheduled and delayed jobs and part of the broader Queue Fundamentals & Architecture collection. It complements the framework-level reference on Celery Beat periodic task scheduling.

The problem this solves, and its sharpest failure: you want a task to run on a cron-like cadence — every night at 02:00, every 15 minutes, the first of each month — and you want it to fire exactly once per occurrence. Celery Beat is the scheduler that emits those jobs, but Beat is a singleton: run two Beat processes (a common accident during rolling deploys or autoscaling) and every scheduled task fires twice. Getting cron right is two jobs — defining the schedule, and guaranteeing only one scheduler is live.

Prerequisites

  • A working Celery app with a broker configured (pip install celery). See setting up Celery with a Redis broker if needed.
  • At least one Celery worker to execute the jobs Beat enqueues — Beat only schedules, it does not run tasks.
  • A decision on where schedules live: static in code (beat_schedule) or dynamic in a database (DatabaseScheduler).
  • A clear reference timezone for your schedules.

Step 1: Define the schedule in code

The simplest setup is a static beat_schedule dict. Each entry names a task and a schedule. Use crontab(...) for calendar-style cadences and a plain number (or timedelta) for fixed intervals.

# celeryconfig.py
from celery import Celery
from celery.schedules import crontab
from datetime import timedelta

app = Celery("tasks", broker="redis://localhost:6379/0")

app.conf.beat_schedule = {
    "nightly-report": {
        "task": "reports.generate_nightly",
        "schedule": crontab(hour=2, minute=0),        # 02:00 every day
        "args": ("daily",),
    },
    "quarter-hour-sync": {
        "task": "sync.pull_changes",
        "schedule": crontab(minute="*/15"),           # :00 :15 :30 :45
    },
    "monthly-billing": {
        "task": "billing.run_cycle",
        "schedule": crontab(day_of_month=1, hour=0, minute=30),  # 1st @ 00:30
    },
    "heartbeat": {
        "task": "health.ping",
        "schedule": timedelta(seconds=30),            # fixed interval
    },
}

Step 2: Understand crontab fields

Celery's crontab mirrors Unix cron but with named keyword arguments, which is easier to read and less error-prone than positional cron strings. Each field accepts an integer, a comma list, a */n step, or a range.

from celery.schedules import crontab

crontab(minute=0, hour="*/3")                 # every 3 hours, on the hour
crontab(minute=30, hour=8, day_of_week="mon-fri")  # weekdays at 08:30
crontab(minute=0, hour=0, day_of_week="sun")  # Sundays at midnight
crontab(minute="0,30")                        # twice an hour

Step 3: Configure the timezone

A cron schedule is meaningless without a reference clock. By default Celery uses UTC. Set timezone explicitly so crontab(hour=2) resolves to the zone you intend, and keep enable_utc consistent. Pinning the zone in config — rather than relying on each host's local time — is what prevents schedules from shifting when a server's TZ differs or when daylight-saving changes.

app.conf.timezone = "America/New_York"   # crontab times are interpreted in this zone
app.conf.enable_utc = True               # store/transport timestamps as UTC internally

Be deliberate about daylight-saving: a crontab(hour=2) job in a DST zone can run twice or skip on transition days. For billing and other exactness-critical work, schedule in UTC and convert inside the task.

Step 4: Run exactly one Beat process

This is the step that determines correctness. Beat must be a single instance fleet-wide. There are three robust approaches.

Run Beat as its own supervised singleton. Keep Beat in a dedicated deployment with replicas pinned to 1, separate from workers, so scaling workers never spawns a second scheduler.

# one Beat process, separate from workers
celery -A tasks beat --loglevel=info --schedule=/var/lib/celery/beat-schedule
# workers run independently and may scale freely
celery -A tasks worker --concurrency=8 --loglevel=info

Use a leader lock (single-beat). Tools like single-beat wrap Beat so that, even if multiple instances start, only the one holding a Redis lock actually runs; the others stand by and take over if the leader dies. This survives the rolling-deploy window where old and new schedulers briefly overlap.

# only the lock holder runs Beat; others wait as hot standbys
BEAT_LOCK_URI=redis://localhost:6379/0 single-beat celery -A tasks beat --loglevel=info

Use the DatabaseScheduler. django-celery-beat stores schedules in the database and serializes scheduling decisions through it, which both centralizes the schedule state and helps coordinate against duplicate firings. It also lets you edit schedules at runtime without redeploying.

celery -A tasks beat --scheduler django_celery_beat.schedulers:DatabaseScheduler --loglevel=info
# with django-celery-beat, schedules are rows you can manage via the admin/ORM
from django_celery_beat.models import PeriodicTask, CrontabSchedule

schedule, _ = CrontabSchedule.objects.get_or_create(hour=2, minute=0, timezone="UTC")
PeriodicTask.objects.get_or_create(
    crontab=schedule, name="nightly-report", task="reports.generate_nightly",
)

Step 5: Make scheduled tasks idempotent

Even with one Beat, the broker is at-least-once: a job Beat enqueues can be delivered twice if a worker dies after running but before acknowledging. For recurring jobs, key the work to its intended occurrence so a duplicate is a no-op — the same discipline as preventing duplicate job execution with idempotency.

from datetime import date

@app.task
def generate_nightly(kind):
    run_key = f"nightly:{kind}:{date.today().isoformat()}"
    # claim this occurrence exactly once; SET NX returns False if already claimed
    if not r.set(run_key, "1", nx=True, ex=60 * 60 * 36):
        return "already ran for today"
    do_report(kind)

Verification

Confirm Beat sees your schedule. Start Beat at debug level; it logs each entry and the next due time on boot.

celery -A tasks beat --loglevel=debug
# look for: "Scheduler: Sending due task nightly-report (reports.generate_nightly)"

Confirm exactly one Beat is live. With a leader lock, only one process should log "Sending due task." Grep your aggregated logs for a single scheduler emitting each occurrence; two senders for one tick means a duplicate Beat is running.

Confirm a single fire per occurrence. Add a temporary counter and let a frequent schedule (e.g. */1 minute) run for a few minutes; assert the task executed once per minute, not twice.

# in the task, during testing
print(f"fired at {time.strftime('%H:%M:%S')}")   # expect one line per scheduled tick

Gotchas and edge cases

Two Beats during rolling deploys. The classic duplicate-fire cause: the new pod's Beat starts before the old one stops. A leader lock (single-beat) or DatabaseScheduler coordination closes this window; a bare replicas: 1 does not, because deploys briefly run two.

Missed runs after downtime. If Beat is down at the scheduled instant, the occurrence is skipped — Beat does not backfill. If catch-up matters, have the task detect the gap (compare against a persisted last-run marker) or use a scheduler that records run history.

Daylight-saving anomalies. Local-time schedules can double-fire or skip on DST transition days. Schedule exactness-critical jobs in UTC and convert inside the task.

Beat persistence file corruption. The default scheduler writes next-run state to a --schedule file. A corrupted or unwritable file makes Beat lose its place. Put it on durable, writable storage, or move to the DatabaseScheduler which keeps state in the database.

For one-shot future delivery rather than recurring cadences, this is the wrong mechanism — use a delay queue instead, as built in implementing delayed jobs with Redis sorted sets.

FAQ

What happens if I accidentally run two Beat processes? Every scheduled task fires twice (or more), because each Beat independently enqueues on its own ticks. Prevent it with a leader lock (single-beat), a DatabaseScheduler, or a strictly pinned singleton deployment — and make tasks idempotent as a backstop.

Does Celery Beat run the tasks itself? No. Beat only enqueues tasks when they come due; ordinary Celery workers execute them. You always need at least one worker running alongside Beat.

Can I change schedules without restarting Beat? With the default in-code beat_schedule, no — changes require a restart. With django-celery-beat's DatabaseScheduler, schedules are database rows you can edit at runtime and Beat picks them up automatically.

Related