RQ vs Celery for Django scheduled tasks

This guide narrows RQ vs Celery for Python and the wider Backend Frameworks & Worker Scaling discussion to one decision: how should a Django app run recurring jobs — nightly reports, hourly syncs, periodic cleanups?

Django teams reach for scheduled tasks the moment a cron entry no longer fits — they want schedules versioned with the app, editable without a deploy, and observable in the admin. The two mainstream answers are django-rq + rq-scheduler and Celery Beat + django-celery-beat. They differ sharply on cron expressiveness, where the schedule lives, and what happens when the scheduler process dies. This guide sets up both, then gives a decision table.

Prerequisites

• A working Django 4.x/5.x project with settings.py you can edit.
• A reachable Redis instance (both stacks use Redis as broker here).
• Familiarity with running a separate long-lived process beside your web server (the scheduler), since neither approach piggybacks on the WSGI process.
• A short list of the jobs you need to schedule and their cadence (interval vs cron).

Step 1: Set up django-rq with rq-scheduler

Install and register the app, then point it at Redis.

pip install django-rq rq-scheduler

# settings.py
INSTALLED_APPS = [
    # ...
    "django_rq",
]

RQ_QUEUES = {
    "default": {
        "HOST": "redis-primary",
        "PORT": 6379,
        "DB": 0,
        "DEFAULT_TIMEOUT": 360,   # seconds before a job is considered failed
    },
}

# urls.py — exposes a queue dashboard under the Django admin auth
from django.urls import path, include

urlpatterns = [
    path("django-rq/", include("django_rq.urls")),  # /django-rq/ queue stats
]

django-rq ships a stats view (not native admin models) gated behind staff login. It shows queue depth, started/finished/failed counts, and lets you requeue failures — but it does not store schedules in the database.

Step 2: Define schedules in rq-scheduler

rq-scheduler keeps scheduled jobs in a Redis sorted set. Schedules are registered in code (or a startup management command), not in the database, so they are not editable from the admin without custom plumbing.

# jobs/schedule.py — run once at deploy (e.g. from a management command)
from datetime import datetime
import django_rq

scheduler = django_rq.get_scheduler("default")

# Clear prior registrations so redeploys don't stack duplicates
for job in scheduler.get_jobs():
    scheduler.cancel(job)

# Interval schedule: every 300s, forever
scheduler.schedule(
    scheduled_time=datetime.utcnow(),
    func="reports.tasks.refresh_cache",
    interval=300,
    repeat=None,         # None = run indefinitely
)

# Cron schedule (rq-scheduler supports cron strings)
scheduler.cron(
    "0 2 * * *",                       # 02:00 daily
    func="reports.tasks.nightly_report",
    queue_name="default",
)

# Run the scheduler process AND a worker (two separate processes)
python manage.py rqscheduler          # promotes due jobs into the queue
python manage.py rqworker default     # executes them

The critical architectural point: rqscheduler only moves due jobs into the queue; a separate rqworker runs them. If the scheduler process is down, due jobs are simply not enqueued — they fire late when it comes back (cron entries are recomputed; missed interval ticks are not back-filled).

Step 3: Set up Celery Beat with django-celery-beat

The Celery stack stores schedules in the database and surfaces them as editable Django admin models.

pip install "celery[redis]" django-celery-beat

# settings.py
INSTALLED_APPS = [
    # ...
    "django_celery_beat",     # adds PeriodicTask/CrontabSchedule admin models
]

CELERY_BROKER_URL = "redis://redis-primary:6379/1"
CELERY_RESULT_BACKEND = "redis://redis-primary:6379/2"
CELERY_TIMEZONE = "UTC"
CELERY_TASK_ACKS_LATE = True

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()   # picks up tasks.py in every installed app

python manage.py migrate django_celery_beat   # creates the schedule tables

After migrating, PeriodicTasks, CrontabSchedule, and IntervalSchedule appear in /admin/, so operators can add, pause, or retime jobs without a deploy.

Step 4: Define schedules for Celery Beat

You can declare schedules in code or create them in the admin. The database-backed scheduler reads whichever exists.

# proj/celery.py — code-declared schedule (read by the DB scheduler too)
from celery.schedules import crontab

app.conf.beat_schedule = {
    "nightly-report": {
        "task": "reports.tasks.nightly_report",
        "schedule": crontab(hour=2, minute=0),   # 02:00 daily
    },
    "refresh-cache": {
        "task": "reports.tasks.refresh_cache",
        "schedule": 300.0,                         # every 300s
    },
}

# Beat reads schedules from the DB (one replica only) and a worker runs them
celery -A proj beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
celery -A proj worker -l info -Q celery

Using DatabaseScheduler means admin edits take effect on Beat's next tick without restarting the process — the headline operational advantage over rq-scheduler.

Verification

Confirm a job actually fires under each stack.

# rq-scheduler: due jobs should appear in the queue, then drain
python manage.py shell -c "import django_rq; print(django_rq.get_scheduler('default').get_jobs())"
redis-cli -n 0 LLEN rq:queue:default

# Celery Beat: Beat logs the send, worker logs the receive
celery -A proj inspect scheduled
# Look for "Scheduler: Sending due task nightly-report" in beat logs
# and "Task reports.tasks.nightly_report ... succeeded" in worker logs

# Assertion you can run after a short wait to prove the periodic task ran
from django_celery_results.models import TaskResult  # if results stored
assert TaskResult.objects.filter(
    task_name="reports.tasks.nightly_report", status="SUCCESS"
).exists()

Decision table

Gotchas & edge cases

• Single scheduler, always. Both stacks assume exactly one scheduler process. Two rqscheduler or two celery beat replicas double-enqueue every due job. Use a single replica or a leader-election lock; this is the top cause of duplicate periodic runs.
• No missed-run back-fill. Neither tool replays jobs that were due while the scheduler was down. If a nightly report must run even after an outage, add a catch-up job that checks for the last successful run rather than trusting the schedule alone.
• Timezone surprises. django-celery-beat honors CELERY_TIMEZONE and Django's USE_TZ; mismatches make crontab(hour=2) fire at the wrong wall-clock time. rq-scheduler operates in UTC unless you convert explicitly. Pin both to UTC and convert at the edges.
• Schedules in code and DB drift. With DatabaseScheduler, beat_schedule entries are synced into the DB on first run, but later admin edits win. Editing the code dict afterward can be silently ignored — manage schedules in one place.

Step 5 — Make the Scheduler Singleton Real

Both options in this comparison share one requirement that is easy to state and easy to violate: exactly one scheduler process may be live at a time. Everything else is a preference; this is a correctness constraint.

The deployment strategy is the first line of defence. A rolling update deliberately overlaps old and new pods, so a scheduler deployed that way has a window of several seconds during which two are live — and any schedule that ticks in that window fires twice. A Recreate strategy terminates the old pod before starting the new one, accepting a short gap with no scheduler in exchange for never having two.

The second line is a lock. A short-lived lease in Redis or the database, acquired before emitting ticks and renewed periodically, means that even a misconfigured rollout or a network partition cannot produce two active schedulers for long. Combine it with a per-tick key so a duplicate tick from a stale holder becomes a no-op rather than a duplicate job.

The third is measurement. Count duplicate ticks and leadership changes, and alert on either being non-zero over a sustained period. Duplicate scheduled work is otherwise close to invisible: two identical jobs at the same second look exactly like one job in most dashboards, and the discrepancy surfaces days later in whatever the job wrote.

Whichever scheduler you adopt, add a check comparing fired slots against the expected sequence and alert on a gap. Both options fail silently: a missing run produces no error, no log line and no metric, only an absence that nothing detects unless something is looking for it.

Keep the scheduler out of the web deployment. Running it as a thread inside a web process is convenient in development and produces one scheduler per web replica in production, which is the duplicate-firing problem in its purest form and one of the easier mistakes to make when the framework makes it a one-line option.

Whichever you choose, keep the scheduled task itself thin: a function that enqueues real work rather than performing it. That keeps the scheduler fast, makes the actual work retryable on its own terms, and means a slow job cannot delay the next tick.

Related

• Migrating from RQ to Celery — the full framework move, including translating rq-scheduler jobs to Beat.
• Comparing RQ and Celery for lightweight Python tasks — the general trade-off beyond scheduling.
• Celery Beat periodic task scheduling — deeper Beat configuration and crontab patterns.
• Setting up Celery with Redis broker and RabbitMQ backend — broker wiring the Celery stack depends on.