Celery Beat Periodic Task Scheduling

This guide builds on the worker and broker setup from Celery Architecture & Configuration and the wider Backend Frameworks & Worker Scaling material, focusing on the scheduler that turns Celery from a fire-and-forget job runner into a cron replacement.

Celery Beat is the component that periodically dispatches tasks onto your queues — nightly reports, hourly cache warmers, a sweep every five minutes. The two failure modes engineers hit most are silent: a schedule that simply never fires because Beat isn't running, and the opposite — every task firing two or three times because two Beat processes are alive at once (a leftover from a rolling deploy, or one per replica in a scaled deployment). Both produce no error, just wrong behavior. This page walks through defining schedules with crontab, solar, and timedelta, persisting them in a database for runtime edits, pinning the timezone so "midnight" means what you expect, and guaranteeing exactly one Beat process drives the schedule.

Prerequisites

  • A running Celery app with broker and result backend (see Setting up Celery with Redis broker and RabbitMQ backend).
  • celery>=5.3; for the database-backed scheduler, django-celery-beat on a Django project, or redbeat for a non-Django stack.
  • At least one worker consuming the queues your scheduled tasks route to — Beat only dispatches, it does not execute.
  • A decision on which single host or pod will own the Beat process.

Step 1 — Define a static schedule with beat_schedule

The simplest schedule lives in your Celery config as the beat_schedule dict. Each entry names the task, its arguments, and a schedule object. Use a plain float for fixed intervals.

# celeryconfig.py
from celery.schedules import crontab

beat_schedule = {
    "warm-cache-every-30s": {
        "task": "myapp.tasks.warm_cache",
        "schedule": 30.0,                 # seconds; fires every 30s
    },
    "nightly-report": {
        "task": "myapp.tasks.build_report",
        "schedule": crontab(hour=2, minute=0),   # 02:00 every day
        "args": ("daily",),
        "options": {"queue": "reports"},  # route to a dedicated queue
    },
}

The options key accepts the same routing and execution options as apply_async, so you can pin scheduled work to a specific queue and keep it off the path of latency-sensitive jobs.

Step 2 — Use crontab, timedelta, and solar schedule types

Celery ships three schedule constructors that cover almost every recurrence pattern. crontab mirrors Unix cron fields, timedelta expresses fixed intervals readably, and solar fires relative to sunrise/sunset at a geographic point.

# celeryconfig.py
from datetime import timedelta
from celery.schedules import crontab, solar

beat_schedule = {
    "weekday-mornings": {
        "task": "myapp.tasks.send_digest",
        # 07:30 Monday-Friday
        "schedule": crontab(hour=7, minute=30, day_of_week="mon-fri"),
    },
    "every-15-minutes": {
        "task": "myapp.tasks.poll_inbox",
        "schedule": timedelta(minutes=15),   # clearer than 900.0
    },
    "quarter-past-each-hour": {
        "task": "myapp.tasks.reconcile",
        "schedule": crontab(minute=15),      # :15 of every hour
    },
    "at-dusk": {
        "task": "myapp.tasks.dim_lights",
        # sunset at Amsterdam; requires latitude/longitude
        "schedule": solar("sunset", 52.3676, 4.9041),
    },
}

crontab supports ranges and steps just like cron (minute="*/5", day_of_week="mon-fri"). For cron-style scheduling patterns shared across stacks, see cron-style scheduling with Celery Beat.

Step 3 — Pin the timezone

A crontab(hour=2) entry means 02:00 in Celery's configured timezone, which defaults to UTC. If your team thinks in local time, set timezone explicitly so the schedule fires when you intend and survives daylight-saving transitions correctly.

# celeryconfig.py
timezone = "Europe/Amsterdam"   # crontab times are interpreted in this zone
enable_utc = True               # store/transport timestamps in UTC internally

Keep enable_utc = True: timestamps on the wire stay UTC (avoiding ambiguity), while crontab interpretation uses the timezone value. Setting a local timezone without this combination is the usual cause of "the report ran an hour early after the clocks changed."

Step 4 — Start the Beat process

Beat runs as its own process, separate from workers. For development you can embed it in a worker with -B, but never do that in production with more than one worker — every worker would run its own scheduler.

# development only: embed beat in a single worker
celery -A myapp worker -B --loglevel=INFO

# production: run beat as a dedicated, single process
celery -A myapp beat --loglevel=INFO \
    --schedule=/var/run/celery/beat-schedule   # path to the persistent schedule DB

The --schedule file is a small shelve database where the default PersistentScheduler records the last run time of each task, so a Beat restart doesn't re-fire everything or skip a window.

Step 5 — Move schedules into a database with DatabaseScheduler

Editing beat_schedule requires a redeploy. For schedules that operators need to change at runtime — pausing a job, shifting a report time — use a database-backed scheduler. On Django, django-celery-beat stores schedules in tables editable from the admin.

pip install django-celery-beat
# settings.py
INSTALLED_APPS = [
    # ...
    "django_celery_beat",
]
# then: python manage.py migrate

Start Beat with the database scheduler so it reads schedules from the database instead of the config file:

celery -A myapp beat --loglevel=INFO \
    --scheduler=django_celery_beat.schedulers:DatabaseScheduler

Create and toggle schedules programmatically; changes take effect without restarting Beat:

# create a periodic task at runtime
from django_celery_beat.models import PeriodicTask, CrontabSchedule

schedule, _ = CrontabSchedule.objects.get_or_create(hour=3, minute=0)
PeriodicTask.objects.create(
    crontab=schedule,
    name="cleanup-tmp",
    task="myapp.tasks.cleanup",
    enabled=True,
)

For a non-Django stack, redbeat (RedBeatScheduler) stores the schedule in Redis and additionally provides a distributed lock, which directly helps with the duplicate-process problem below.

Step 6 — Guarantee exactly one Beat process

Beat has no built-in leader election with the default scheduler. Two Beat processes mean every task fires twice. Run exactly one. In Kubernetes, that means a Deployment with replicas: 1 and the Recreate strategy so a rolling update never briefly runs two pods.

# k8s/beat-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-beat
spec:
  replicas: 1                 # never more than one beat scheduler
  strategy:
    type: Recreate            # terminate old pod before starting new one (no overlap)
  template:
    spec:
      containers:
        - name: beat
          image: myapp:latest
          command: ["celery", "-A", "myapp", "beat", "--loglevel=INFO"]

If you cannot guarantee a single replica, switch to redbeat, whose RedBeatScheduler acquires a Redis lock so only the lock holder dispatches, making accidental duplicate Beat processes harmless.

Verification

Confirm the schedule is loaded and firing as expected.

On startup, Beat logs each registered entry and its next run time:

# look for "Scheduler: Sending due task ..." lines at the expected wall-clock time
celery -A myapp beat --loglevel=DEBUG

Check that a worker actually received and ran the dispatched task by watching the event stream:

celery -A myapp events --dump   # expect task-received / task-succeeded for the scheduled task

Assert the schedule registers correctly in a test:

# tests/test_schedule.py
from myapp.celery import app

def test_nightly_report_scheduled():
    entry = app.conf.beat_schedule["nightly-report"]
    assert entry["task"] == "myapp.tasks.build_report"
    assert entry["schedule"].hour == {2}       # crontab stores fields as sets
    assert entry["schedule"].minute == {0}

Gotchas & edge cases

  • Two Beat processes = double execution. This is the single most common Beat bug, usually from replicas: 2 or a worker started with -B alongside a standalone Beat. Enforce one scheduler or use redbeat's lock.
  • Timezone defaults to UTC. A crontab(hour=0) you assumed was local midnight fires at UTC midnight. Set timezone explicitly and verify after a daylight-saving change.
  • Beat dispatches, it does not execute. If no worker consumes the target queue, tasks pile up on the broker and "nothing runs." Confirm a worker is subscribed to the queue named in options.
  • Long-running tasks can overlap. Beat fires on schedule regardless of whether the previous run finished. For tasks that must not overlap, guard the body with a distributed lock rather than assuming serialization.
  • A missing or unwritable --schedule file. With the default PersistentScheduler, an unwritable path makes Beat unable to record last-run times, causing re-fires on restart. Ensure the directory exists and is writable.

Related