Scaling Workers with KEDA on Queue Length

CPU is a poor scaling signal for a worker fleet, because a worker blocked on a slow API is idle by every CPU measure while the backlog grows behind it. This guide implements queue-driven scaling from Horizontal Worker Scaling, part of Backend Frameworks & Worker Scaling.

Problem Statement

Your workers scale on CPU with a 70% target. Most jobs wait on HTTP calls, so CPU sits at 15% no matter how deep the queue gets, and the HPA never adds a replica. During the morning burst the backlog reaches 40,000 messages and takes ninety minutes to clear, while Kubernetes has ample idle capacity. You need the backlog itself to drive replica count, with sensible behaviour when the burst ends.

Prerequisites

  • Kubernetes 1.27+ with KEDA 2.12+ installed (keda-operator in the keda namespace).
  • A worker Deployment that drains cleanly on SIGTERM, per zero-downtime worker deploys on Kubernetes — scaling down is a shutdown.
  • Broker credentials available as a Kubernetes secret.
  • Measured p50 job duration; the target value is derived from it.

Step 1 — Pick the Metric That Reflects Backlog

Broker KEDA scaler Metric Caveat
Redis (list) redis listLength One list per queue; Celery's Redis keys are plain lists
Redis (stream) redis-streams pending entries Counts unacknowledged, not waiting
RabbitMQ rabbitmq queueLength or messageRate Length excludes unacked by default
AWS SQS aws-sqs-queue ApproximateNumberOfMessages Add NotVisible to include in-flight
Prometheus prometheus any query The escape hatch: use for oldest-age scaling

Depth is the usual choice, but it is a lagging signal for slow jobs. If a job takes two minutes, a queue of ten messages is a twenty-minute backlog, and depth-based scaling will under-react. For those workloads scale on oldest-message age via the Prometheus scaler instead.

Step 2 — Write the ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-media
spec:
  scaleTargetRef:
    name: worker-media
  minReplicaCount: 2          # never zero for a latency-sensitive queue
  maxReplicaCount: 40         # bounded by the downstream, not by the cluster
  pollingInterval: 15         # how often KEDA reads the metric
  cooldownPeriod: 300         # wait before scaling to minReplicaCount
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0     # react to bursts immediately
          policies:
            - type: Percent
              value: 100                    # at most double per step
              periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300   # ignore dips for 5 minutes
          policies:
            - type: Pods
              value: 2                      # remove at most 2 pods per minute
              periodSeconds: 60
  triggers:
    - type: redis
      metadata:
        addressFromEnv: REDIS_URL
        listName: media
        listLength: "30"      # target messages PER REPLICA — see step 3
        enableTLS: "false"

The asymmetry between scaleUp and scaleDown is deliberate and is most of what makes autoscaling behave. Scaling up fast costs a few extra pods for a few minutes; scaling down fast interrupts in-flight jobs and then immediately needs the capacity back.

Fast up, slow down Queue depth spikes at the start of a morning burst and the replica count rises within a minute, doubling per step until the backlog drains. When depth falls, replicas stay elevated through a five-minute stabilisation window before stepping down two pods at a time, avoiding a second scale-up when the next burst arrives. Scale up on sight, scale down on evidence queue depth burst replicas 2 → 4 → 8 → 16 within ~90s depth clears stabilisation ends → step down replicas held for 5 min

Step 3 — Derive the Target from Job Duration

listLength is the target backlog per replica, and KEDA divides total depth by it to get the desired replica count. Setting it by intuition is how fleets end up oscillating.

def target_per_replica(p50_seconds: float, concurrency: int,
                       drain_target_seconds: float = 60) -> int:
    """Messages per replica such that a full backlog drains in ~drain_target."""
    throughput_per_replica = concurrency / p50_seconds     # jobs/sec
    return max(1, int(throughput_per_replica * drain_target_seconds))

target_per_replica(0.4, concurrency=8)     # 1200 — fast jobs, big target
target_per_replica(30, concurrency=4)      # 8    — slow jobs, small target

A slow-job queue needs a small target: with 30-second jobs, a target of 100 means each replica has a 50-minute backlog before another is added. Getting this backwards — a large target on slow jobs — is the most common KEDA misconfiguration.

Step 4 — Bound maxReplicaCount by the Real Constraint

Kubernetes is rarely the limit. A downstream database, a connection pool, or a third-party quota usually is, and an autoscaler that ignores them converts a queue backlog into a downstream outage.

def max_replicas(db_max_connections: int, conns_per_worker: int,
                 api_quota_rps: float, job_rps_per_worker: float) -> int:
    by_db = (db_max_connections * 0.7) // conns_per_worker   # leave 30% for the app
    by_api = api_quota_rps / job_rps_per_worker
    return int(min(by_db, by_api))

max_replicas(200, conns_per_worker=10, api_quota_rps=100, job_rps_per_worker=5)
# → 14 (API-bound, not database-bound)

Record which constraint bound the number. When someone later asks to raise maxReplicaCount, the answer is either "raise the quota first" or "yes", and the difference matters.

Step 5 — Handle Scale-to-Zero Carefully

minReplicaCount: 0 is attractive for rare batch queues and dangerous for anything latency-sensitive: KEDA's polling interval plus pod scheduling plus application start-up is typically 30–90 seconds of cold start on the first message.

  minReplicaCount: 0
  cooldownPeriod: 600          # keep the last pod for 10 minutes after idle
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/1234/nightly-reports
        queueLength: "5"
        awsRegion: eu-west-1
        scaleOnInFlight: "false"   # do not count in-flight toward scale-up

Use it for nightly batch, never for a queue with an interactive SLO. scaleOnInFlight: false matters when scaling to zero: counting in-flight messages keeps a replica alive that has nothing waiting for it.

Step 6 — Make Scale-Down a Clean Drain

Scale-down is a shutdown, and inherits its problems Without a sufficient grace period, a pod removed by the autoscaler is killed mid-job and its messages are redelivered. With a grace period longer than the p99 job duration, the pod stops fetching, finishes its in-flight jobs, acknowledges them, and exits cleanly with no redelivery. Every scale-down event is a deploy in miniature grace 30s, p99 job 90s draining SIGKILL — jobs redelivered elsewhere the backlog you scaled down from grows slightly back grace 120s stop fetching → finish in-flight → ack → exit clean exit no redelivery, no duplicate side effects Scale-down happens far more often than deploys — this setting matters more here. 
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 120   # > p99 job duration
      containers:
        - name: worker
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "kill -TSTP 1 && sleep 5"]

Step 7 — Stop Oscillation Before It Starts

Oscillation — the fleet growing and shrinking every few minutes — wastes capacity, multiplies redeliveries, and makes every latency graph unreadable. It has three usual causes and each has a specific fix.

The first is a target value close to steady-state depth, so normal jitter crosses the threshold repeatedly. Set the target so that steady-state depth sits well below one replica's worth, and let the stabilisation window absorb the rest.

The second is scaling down faster than the work drains. A pod removed mid-job releases its messages back to the queue, depth rises, and the autoscaler adds a replica — a loop that is entirely self-inflicted. The 300-second scaleDown.stabilizationWindowSeconds plus a generous grace period is the fix, and it is why those two settings must be tuned together rather than separately.

The third is competing autoscalers. A ScaledObject and a hand-written HPA on the same Deployment fight, each undoing the other's decision every polling interval. KEDA creates and owns its own HPA; delete any pre-existing one.

# Watch a burst end-to-end: the desired replica count is what KEDA computed
kubectl get scaledobject worker-media -w
kubectl get hpa keda-hpa-worker-media -w
kubectl describe scaledobject worker-media | tail -20   # last scaling decisions
Target per replica falls as job duration rises Three workloads. Fast four-hundred-millisecond jobs support a target of twelve hundred messages per replica. Five-second jobs support around one hundred. Thirty-second jobs support only eight, because a larger target would let each replica accumulate a backlog measured in tens of minutes before another is added. listLength is per replica — derive it from duration p50 = 0.4s target 1,200 — a full backlog drains in ~60s p50 = 5s target ~100 p50 = 30s target 8 — a target of 100 here means a 50-minute backlog per replica Slow jobs need small targets. Getting this backwards is the most common KEDA misconfiguration.

Verification

  1. A burst scales up within a minute. Inject 20,000 messages; replicas should begin rising within one polling interval and reach the computed target within a couple of minutes.
  2. Depth drains at the expected rate. Time-to-drain should approximate depth ÷ (replicas × concurrency ÷ p50); a large discrepancy means the target value is wrong.
  3. Scale-down is clean. During the wind-down, the redelivery counter must not move.
  4. No oscillation. Over an hour of steady traffic, replica count should change no more than a few times.

Gotchas & Edge Cases

RabbitMQ queueLength excludes unacked messages by default. A fleet with a large prefetch holds much of the backlog invisibly, so KEDA under-scales. Use mode: QueueLength with value sized accordingly, or scale on messageRate.

Celery's Redis keys are not always plain lists. Priority emulation appends a suffix to the key name, so listName: celery misses messages on celery\x06\x163. Verify with redis-cli --scan --pattern 'celery*'.

KEDA scaling to zero hides the queue from your alerts. With no consumer, a stuck message produces no worker logs at all. Keep the oldest-age alert running independently of replica count.

Multiple triggers take the maximum, not the sum. A ScaledObject with both a depth trigger and an age trigger scales to whichever demands more replicas, which is usually what you want — but it also means a misconfigured trigger can hold the fleet high on its own. Check each trigger's computed target separately when a fleet refuses to scale down.

Cluster autoscaling adds another minute. If scaling replicas requires new nodes, real scale-up latency is KEDA's interval plus node provisioning. Keep headroom or use overprovisioning pods.

Scaling on age needs the Prometheus scaler and a different target. Depth-based targets scale with throughput; age-based targets encode a latency objective directly, which is usually what the SLO actually says. A trigger of max(queue_oldest_message_age_seconds) > 30 with a target of 30 adds replicas whenever the head of the queue is older than the objective allows, and it behaves correctly for slow jobs where depth is a poor proxy. The cost is a dependency on Prometheus being available for scaling decisions, so keep a depth-based trigger alongside it as a fallback — KEDA takes the maximum across triggers.

The metric is polled, not streamed. With pollingInterval: 30 the autoscaler is up to thirty seconds behind reality. For bursty interactive queues, poll every 10–15 seconds and accept the extra broker load.

Related