Flower for Real-Time Celery Monitoring

Flower is the web-based, real-time monitoring tool for Celery, and this guide covers it as the live-inspection layer of Observability & Monitoring for Job Queues. Where Prometheus answers "is the system healthy over the last hour", Flower answers the question you actually have during an incident: "which task is stuck right now, what arguments did it receive, and which worker is running it".

Flower attaches to Celery's event stream — the same task-sent, task-started, task-succeeded, task-failed, task-retried events that exporters consume — and renders them as a live dashboard. It shows every active, scheduled, reserved, and recently completed task; the state and concurrency of every worker; and it exposes a REST API to inspect and control the fleet programmatically. It is the fastest way to get visibility into a Celery deployment, and it ships as a single pip install.

Problem Framing: The Live-State Gap

Aggregated metrics deliberately discard per-task detail to stay cheap and bounded — that is exactly why they cannot tell you which of the 4,000 in-flight jobs is the one hanging on a deadlocked row. During an incident you need the opposite of aggregation: the specific job ID, its exact arguments, its current runtime, and the worker hosting it, so you can decide whether to revoke it, restart the worker, or let it finish. Flower fills that live-state gap. It is not a replacement for metrics — it has almost no history and no real alerting — but it is the right tool for the triage minute.

Architecture & Running Flower

Flower runs as a standalone process that connects to the same broker as your workers. It needs task events enabled — without them it sees worker presence but no task lifecycle detail. Enable events on the worker side and start Flower against the broker URL.

# celery_app.py — Flower depends on the event stream being on
app.conf.update(
    worker_send_task_events=True,   # required for the Tasks view to populate
    task_send_sent_event=True,      # adds enqueue events so you see queue wait
)

# Run Flower against the broker. --persistent keeps state across restarts.
celery -A celery_app flower \
  --broker=redis://redis:6379/0 \
  --port=5555 \
  --persistent=True \
  --db=/var/lib/flower/flower.db \
  --max_tasks=10000        # cap retained task history to bound memory

In containers, run Flower as its own service so it scales and restarts independently of the workers:

# docker-compose.yml
services:
  flower:
    image: mher/flower:latest
    command: ["celery", "--broker=redis://redis:6379/0", "flower", "--port=5555"]
    ports: ["5555:5555"]
    environment:
      - FLOWER_PERSISTENT=True
      - FLOWER_MAX_TASKS=10000

The Tasks and Workers Views

The Tasks view is the core of Flower. It lists every task Flower has observed, filterable by state (active, succeeded, failed, retried, revoked), by name, and by worker. Each row drills into the task UUID, the arguments and keyword arguments it was called with, its runtime, the result or exception traceback, and the retry count. This is the view you live in during an incident: filter to FAILURE, read the traceback, copy the args, reproduce locally.

The Workers view shows each worker's status, the pool implementation and concurrency, the number of active and processed tasks, and per-worker load average. From here you can issue control commands without touching a shell — adjust a worker's concurrency with pool grow/pool shrink, rate-limit a task, or revoke a runaway job. The Broker view (Redis/RabbitMQ) shows queue lengths, which is your live backlog read.

The REST API and Programmatic Control

Everything in the UI is also a JSON endpoint, which makes Flower a control plane, not just a viewer. This is useful for runbooks and automation — for example, revoking all instances of a misbehaving task during an incident, or scripting a concurrency change.

# List active tasks as JSON
curl -s http://flower:5555/api/tasks?state=STARTED | jq '.[] | {uuid, name, runtime}'

# Revoke and terminate a stuck task by UUID
curl -X POST http://flower:5555/api/task/revoke/<task-uuid>?terminate=true

# Grow a worker's pool by 4 processes during a backlog spike
curl -X POST http://flower:5555/api/worker/pool/grow/worker1@host \
  -d 'n=4'

# A runbook step: revoke every queued instance of a known-bad task
import requests

tasks = requests.get("http://flower:5555/api/tasks", params={"state": "RECEIVED"}).json()
for uuid, t in tasks.items():
    if t["name"] == "app.tasks.broken_report":
        requests.post(f"http://flower:5555/api/task/revoke/{uuid}", params={"terminate": "true"})

Persistence and Its Limits

By default Flower holds all state in memory — restart it and the task history is gone. The --persistent=True --db=... flags write a local shelve database so history survives restarts, and --max_tasks caps how many task records are retained to bound memory and disk. This is enough for short-lived operational history, but it is emphatically not a metrics store: there is no downsampling, no long retention, no efficient time-range query, and the local DB does not survive the pod being rescheduled to a new node unless you mount durable storage.

# Persistent Flower with a mounted volume so history survives container rescheduling
celery -A celery_app flower \
  --persistent=True \
  --db=/data/flower.db \      # mount /data on durable storage
  --max_tasks=50000           # tune against available memory

Trade-off Analysis: Flower vs. Prometheus

The two are complementary, not competing. Run Prometheus and Grafana for trends, percentiles, and paging, as covered in Prometheus Metrics for Workers; run Flower for the live, per-task triage that aggregated metrics structurally cannot provide.

Failure Modes & Recovery

Tasks view is empty. Events are not enabled. Recovery: set worker_send_task_events=True (and restart workers); confirm the workers, not just Flower, were redeployed with the setting.

Flower memory grows unbounded. No --max_tasks cap, so it retains every task forever. Recovery: set --max_tasks to a value sized to available memory, and enable persistence so the history is on disk rather than purely in RAM.

History lost on every deploy. Persistence off, or the DB on ephemeral container storage. Recovery: enable --persistent and mount the --db path on durable storage so a reschedule does not wipe it.

Flower exposed to the internet without auth. A wide-open Flower lets anyone revoke tasks and read job arguments — a serious exposure. Recovery: never run it unauthenticated in production; the full hardening procedure is in Securing the Flower dashboard in production.

Performance Tuning

Flower's cost is dominated by event volume and retained task count. On a very high-throughput fleet the event stream itself can become heavy; cap retention with --max_tasks, and consider running a dedicated Flower instance per broker rather than fanning one instance across many. Because the dashboard pushes live updates over WebSocket, keep the browser tab count modest on busy fleets. For anything beyond live triage — capacity planning, SLO tracking, alerting — push that load onto Prometheus rather than asking Flower to be a time-series database it was never designed to be.

Reading the Views Correctly

Two of Flower's views are routinely misread, and the misreadings lead to wrong conclusions during incidents.

The tasks view shows tasks Flower has seen since it started, which is not the same as tasks that exist. Messages sitting in the queue unclaimed produce no events at all, so a backlog of fifty thousand pending jobs appears nowhere in this view. An empty tasks list means "nothing has been dispatched recently", which during a stall is exactly the situation and exactly the opposite of what it appears to mean.

The workers view shows workers that have sent a heartbeat within the event window. A worker that has crashed disappears from the list rather than being marked failed, and a worker that is alive but blocked continues to appear healthy with its current task listed. Reading "four workers online" as "four workers processing" is therefore unsafe: check whether their current tasks are changing, not merely whether they are present.

Both misreadings share a cause — Flower reports events, not state — and both are avoided by the same habit: use Flower to inspect what is moving, and the broker or a metrics query to establish what is waiting. Once that division is clear, the tool becomes considerably more trustworthy, because you are asking it only the questions it can actually answer.

Where Flower Fits in a Monitoring Stack

Flower answers a question no metrics system answers well: what is happening in this fleet right now, at the level of individual tasks. That makes it genuinely useful and also frequently misused, because the same properties that make it good at live inspection make it unsuitable as a monitoring backend.

Its state is in memory and bounded, so a restart loses history and a busy fleet ages recent tasks out quickly. Its detail is per-task and unbounded in cardinality, which is exactly what you want when investigating one job and exactly what you cannot store for months. And its data comes from Celery's event stream, which is optional, adds broker traffic, and can be dropped under load — meaning Flower can be incomplete without saying so.

The complementary tool is a metrics pipeline: aggregated, bounded, durable and able to answer "is this normal for a Tuesday". Together they cover the two questions an operator actually asks, and neither substitutes for the other. Alerting belongs entirely on the metrics side, because an alert must fire on data that survives a restart and is evaluated by a system whose job is evaluation.

The third component worth naming is tracing, which answers "where did this specific job spend its time". Flower shows that a task took forty seconds; a trace shows that thirty-eight of them were a downstream call. For a fleet of any complexity all three are worth having, and the cost of running Flower alongside the other two is small — provided it is deployed behind authentication, since it exposes task arguments and the ability to revoke tasks and shut down workers.

Operating Flower Safely

Three practices keep Flower useful rather than hazardous. Bind it to loopback and put an authenticating reverse proxy in front, because its control endpoints can stop the fleet and it has no authorisation model of its own. Restrict network access to known sources, so a proxy misconfiguration is not immediately exploitable. And treat the proxy's access log as the audit trail, since Flower itself does not record who revoked a task or resized a pool.

Two operational details are worth knowing. Enabling worker events adds broker traffic proportional to task volume, which is negligible on most fleets and measurable on very high-rate ones — worth checking rather than assuming. And Flower's persistence option writes recent state to disk, which survives a restart but is not a substitute for a metrics store: it bounds the loss rather than removing it.

FAQ

Is Flower a replacement for Prometheus and Grafana? No, and trying to use it as one leads to pain. Flower excels at live, per-task inspection and worker control but has only ephemeral history, no percentile latency, and no real alerting. Prometheus and Grafana give you trends, p99 latency, long retention, and paging. Run both: Flower for the incident triage minute, Prometheus for everything time-based.

Does Flower add load to my Celery workers? Indirectly and usually negligibly. Flower itself runs as a separate process, but it requires worker_send_task_events=True, which makes workers publish lifecycle events to the broker. On most fleets that overhead is tiny; on extreme-throughput fleets the event volume is worth measuring, and you can cap Flower's retained history with --max_tasks to keep its own footprint bounded.

How do I keep task history across restarts? Run Flower with --persistent=True and point --db at a path on durable storage, then bound it with --max_tasks. Without persistence all state is in memory and vanishes on restart; with persistence on ephemeral container storage it vanishes when the pod is rescheduled, so the volume must be durable.

Can I control workers from Flower or only watch them? You can control them. The Workers view and the REST API let you revoke and terminate tasks, grow or shrink a worker's pool, set rate limits, and shut workers down. That power is exactly why Flower must be authenticated in production — an open instance is a remote control for your entire job fleet.

A useful habit when onboarding someone to a fleet is to open Flower alongside the metrics dashboard and walk through the same incident on both. It makes the division of labour concrete — the dashboard shows that something changed, Flower shows which tasks are involved — and it prevents the common mistake of treating a live view as evidence about a period it never observed.

What to Use Flower For

Reduced to specifics, Flower earns its place in four situations.

Confirming a fleet is alive during an incident. The workers view shows which workers have checked in, their pool size and their current tasks. When queue depth is rising and you need to know whether consumers exist at all, that is a faster answer than a metrics query.

Inspecting a specific task's arguments and outcome. When a customer reports a job that behaved oddly, finding it by identifier and reading its arguments, runtime and traceback is exactly what Flower is for, and no aggregated metric substitutes.

Revoking a runaway task. Occasionally the correct action is to stop a specific job or drain a specific worker. Flower exposes that through a UI rather than requiring a control command, which matters when the person who needs to act is not the person who wrote the deployment.

Checking that a change took effect. After adjusting concurrency, prefetch or routing, the workers view shows the new configuration in place across the fleet — a direct confirmation rather than an inference from a graph.

What it should not be used for is equally short. Not alerting, because its state is in memory and its availability is not guaranteed. Not capacity planning, because it has no history. Not as a general dashboard, because per-task detail is the wrong resolution for spotting a trend. Keeping those boundaries clear is what stops a useful tool from becoming a monitoring system nobody can rely on.

For teams running more than one Celery application, it is worth resisting the temptation to point a single Flower at several brokers. The event streams interleave, the workers view mixes fleets, and the control endpoints then reach further than anyone intends. One Flower per application, each behind the same proxy, keeps both the data and the blast radius interpretable.

Running Flower in Development

Flower is at its most valuable locally, where the event volume is small and the feedback loop is short. Running it against a development broker while writing a new task shows routing, retries and failures immediately, and catches misconfigured queues before they reach an environment where the symptom is a silent backlog.

Related

• Securing the Flower dashboard in production — authentication, TLS, and network controls for a tool that can revoke jobs.
• Observability & Monitoring for Job Queues — where live inspection fits among metrics, logs, and traces.
• Prometheus Metrics for Workers — the metrics backbone that complements Flower's live view.
• Grafana Dashboards for Queues — the trend and alerting layer Flower deliberately lacks.
• Celery Architecture & Configuration — the framework Flower observes.