RQ vs Celery for Django scheduled tasks

This guide narrows RQ vs Celery for Python and the wider Backend Frameworks & Worker Scaling discussion to one decision: how should a Django app run recurring jobs — nightly reports, hourly syncs, periodic cleanups?

Django teams reach for scheduled tasks the moment a cron entry no longer fits — they want schedules versioned with the app, editable without a deploy, and observable in the admin. The two mainstream answers are django-rq + rq-scheduler and Celery Beat + django-celery-beat. They differ sharply on cron expressiveness, where the schedule lives, and what happens when the scheduler process dies. This guide sets up both, then gives a decision table.

Prerequisites

• A working Django 4.x/5.x project with settings.py you can edit.
• A reachable Redis instance (both stacks use Redis as broker here).
• Familiarity with running a separate long-lived process beside your web server (the scheduler), since neither approach piggybacks on the WSGI process.
• A short list of the jobs you need to schedule and their cadence (interval vs cron).

Step 1: Set up django-rq with rq-scheduler

Install and register the app, then point it at Redis.

pip install django-rq rq-scheduler

# settings.py
INSTALLED_APPS = [
    # ...
    "django_rq",
]

RQ_QUEUES = {
    "default": {
        "HOST": "redis-primary",
        "PORT": 6379,
        "DB": 0,
        "DEFAULT_TIMEOUT": 360,   # seconds before a job is considered failed
    },
}

# urls.py — exposes a queue dashboard under the Django admin auth
from django.urls import path, include

urlpatterns = [
    path("django-rq/", include("django_rq.urls")),  # /django-rq/ queue stats
]

django-rq ships a stats view (not native admin models) gated behind staff login. It shows queue depth, started/finished/failed counts, and lets you requeue failures — but it does not store schedules in the database.

Step 2: Define schedules in rq-scheduler

rq-scheduler keeps scheduled jobs in a Redis sorted set. Schedules are registered in code (or a startup management command), not in the database, so they are not editable from the admin without custom plumbing.

# jobs/schedule.py — run once at deploy (e.g. from a management command)
from datetime import datetime
import django_rq

scheduler = django_rq.get_scheduler("default")

# Clear prior registrations so redeploys don't stack duplicates
for job in scheduler.get_jobs():
    scheduler.cancel(job)

# Interval schedule: every 300s, forever
scheduler.schedule(
    scheduled_time=datetime.utcnow(),
    func="reports.tasks.refresh_cache",
    interval=300,
    repeat=None,         # None = run indefinitely
)

# Cron schedule (rq-scheduler supports cron strings)
scheduler.cron(
    "0 2 * * *",                       # 02:00 daily
    func="reports.tasks.nightly_report",
    queue_name="default",
)

# Run the scheduler process AND a worker (two separate processes)
python manage.py rqscheduler          # promotes due jobs into the queue
python manage.py rqworker default     # executes them

The critical architectural point: rqscheduler only moves due jobs into the queue; a separate rqworker runs them. If the scheduler process is down, due jobs are simply not enqueued — they fire late when it comes back (cron entries are recomputed; missed interval ticks are not back-filled).

Step 3: Set up Celery Beat with django-celery-beat

The Celery stack stores schedules in the database and surfaces them as editable Django admin models.

pip install "celery[redis]" django-celery-beat

# settings.py
INSTALLED_APPS = [
    # ...
    "django_celery_beat",     # adds PeriodicTask/CrontabSchedule admin models
]

CELERY_BROKER_URL = "redis://redis-primary:6379/1"
CELERY_RESULT_BACKEND = "redis://redis-primary:6379/2"
CELERY_TIMEZONE = "UTC"
CELERY_TASK_ACKS_LATE = True

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()   # picks up tasks.py in every installed app

python manage.py migrate django_celery_beat   # creates the schedule tables

After migrating, PeriodicTasks, CrontabSchedule, and IntervalSchedule appear in /admin/, so operators can add, pause, or retime jobs without a deploy.

Step 4: Define schedules for Celery Beat

You can declare schedules in code or create them in the admin. The database-backed scheduler reads whichever exists.

# proj/celery.py — code-declared schedule (read by the DB scheduler too)
from celery.schedules import crontab

app.conf.beat_schedule = {
    "nightly-report": {
        "task": "reports.tasks.nightly_report",
        "schedule": crontab(hour=2, minute=0),   # 02:00 daily
    },
    "refresh-cache": {
        "task": "reports.tasks.refresh_cache",
        "schedule": 300.0,                         # every 300s
    },
}

# Beat reads schedules from the DB (one replica only) and a worker runs them
celery -A proj beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
celery -A proj worker -l info -Q celery

Using DatabaseScheduler means admin edits take effect on Beat's next tick without restarting the process — the headline operational advantage over rq-scheduler.

Verification

Confirm a job actually fires under each stack.

# rq-scheduler: due jobs should appear in the queue, then drain
python manage.py shell -c "import django_rq; print(django_rq.get_scheduler('default').get_jobs())"
redis-cli -n 0 LLEN rq:queue:default

# Celery Beat: Beat logs the send, worker logs the receive
celery -A proj inspect scheduled
# Look for "Scheduler: Sending due task nightly-report" in beat logs
# and "Task reports.tasks.nightly_report ... succeeded" in worker logs

# Assertion you can run after a short wait to prove the periodic task ran
from django_celery_results.models import TaskResult  # if results stored
assert TaskResult.objects.filter(
    task_name="reports.tasks.nightly_report", status="SUCCESS"
).exists()

Decision table

Gotchas & edge cases

• Single scheduler, always. Both stacks assume exactly one scheduler process. Two rqscheduler or two celery beat replicas double-enqueue every due job. Use a single replica or a leader-election lock; this is the top cause of duplicate periodic runs.
• No missed-run back-fill. Neither tool replays jobs that were due while the scheduler was down. If a nightly report must run even after an outage, add a catch-up job that checks for the last successful run rather than trusting the schedule alone.
• Timezone surprises. django-celery-beat honors CELERY_TIMEZONE and Django's USE_TZ; mismatches make crontab(hour=2) fire at the wrong wall-clock time. rq-scheduler operates in UTC unless you convert explicitly. Pin both to UTC and convert at the edges.
• Schedules in code and DB drift. With DatabaseScheduler, beat_schedule entries are synced into the DB on first run, but later admin edits win. Editing the code dict afterward can be silently ignored — manage schedules in one place.

Related

• Migrating from RQ to Celery — the full framework move, including translating rq-scheduler jobs to Beat.
• Comparing RQ and Celery for lightweight Python tasks — the general trade-off beyond scheduling.
• Celery Beat periodic task scheduling — deeper Beat configuration and crontab patterns.
• Setting up Celery with Redis broker and RabbitMQ backend — broker wiring the Celery stack depends on.