Alerting on Dead-Letter Queue Growth

A dead-letter queue is the highest-signal metric a job system has, and the easiest to let rot into background noise. This guide adds the alerting layer to the quarantine mechanics in Dead-Letter Queues & Poison-Message Handling, part of Queue Fundamentals & Architecture.

Problem Statement

Your DLQ has 3,412 messages. Nobody knows when most of them arrived, which are still relevant, or whether the 40 that landed this morning are a new problem or more of the old one. There is an alert — "DLQ depth > 100" — which fired eight months ago, was acknowledged, and has been firing ever since. You want alerts that distinguish a new failure from an old backlog, that name an owner, and that stay quiet when nothing is wrong.

Prerequisites

• A DLQ per source queue rather than one shared quarantine; a mixed DLQ cannot be alerted on usefully.
• Metrics for depth, arrival rate, and oldest-message age. Depth alone is insufficient.
• Failure metadata stamped at quarantine time: exception class, source queue, attempt count.
• An owner per queue, so the alert can route somewhere with context.

Step 1 — Alert on Three Signals, Not One

Depth, rate, and age answer different questions, and every mature DLQ alerting setup uses all three.

Step 2 — Page on Arrival Rate

Any message arriving in a DLQ is work the system gave up on. In steady state the rate should be zero, which makes it one of the few metrics where "greater than zero" is a legitimate threshold.

- alert: DeadLetterArrivals
  expr: sum by (queue) (rate(dlq_messages_received_total[5m])) > 0
  for: 10m                    # ride out a single bad payload; catch a pattern
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.queue }} is dead-lettering {{ $value | humanize }} msg/s"
    runbook: "https://runbooks.internal/queues/dlq-triage"

- alert: DeadLetterStorm
  expr: sum by (queue) (increase(dlq_messages_received_total[5m])) > 50
  for: 0m                     # a storm needs no confirmation window
  labels:
    severity: page

The two tiers matter. A trickle deserves a ten-minute confirmation so a single malformed payload does not page anyone; fifty in five minutes is a regression or an outage and should page immediately.

Step 3 — Escalate on Age

Age catches the failure mode alerting usually misses: messages that arrived quietly and were never looked at.

- alert: DeadLetterMessagesStale
  expr: max by (queue) (dlq_oldest_message_age_seconds) > 14400   # 4 hours
  for: 30m
  labels: { severity: ticket }

- alert: DeadLetterMessagesAbandoned
  expr: max by (queue) (dlq_oldest_message_age_seconds) > 172800  # 48 hours
  for: 1h
  labels: { severity: page }
  annotations:
    summary: "{{ $labels.queue }} DLQ has messages older than 48h — approaching retention"

The second alert is a data-loss warning: on SQS with 14-day retention, a message ignored long enough is deleted by the broker, and nobody finds out. Tie the threshold to your retention so the page arrives with time to act.

Step 4 — Label by Cause So the Alert Is Diagnosable

An alert that says "12 messages dead-lettered" sends someone to a dashboard. One that says "12 messages, all SchemaValidationError, all from orders" sends them to a code change.

DLQ_RECEIVED = Counter(
    "dlq_messages_received_total", "Messages quarantined",
    ["queue", "exception", "source"],       # keep cardinality bounded
)

def on_dead_letter(msg, exc: Exception) -> None:
    DLQ_RECEIVED.labels(
        queue=msg.queue,
        exception=type(exc).__name__,       # class name only — never the message
        source=msg.headers.get("producer", "unknown"),
    ).inc()
    dlq.publish(msg.body, headers={
        **msg.headers,
        "dlq_reason": type(exc).__name__,
        "dlq_at": time.time(),
        "dlq_attempts": msg.attempts,
        "dlq_last_error": str(exc)[:500],   # in the payload, not the label
    })

Exception messages must stay out of labels — they frequently embed IDs and would explode cardinality. Class names are bounded and are what you group by anyway.

# Which failure dominates right now?
topk(3, sum by (exception) (increase(dlq_messages_received_total[1h])))

Step 5 — Route to the Queue's Owner

route:
  group_by: [queue, exception]
  routes:
    - matchers: [alertname=~"DeadLetter.*", queue=~"payments|checkout"]
      receiver: payments-oncall
    - matchers: [alertname=~"DeadLetter.*", queue=~"exports|reports"]
      receiver: data-platform
    - matchers: [alertname=~"DeadLetter.*"]
      receiver: platform-fallback      # unowned queues are a defect, not a default

Grouping by exception as well as queue means one notification per distinct failure rather than one per message, which is the difference between an actionable alert and a pager storm during an incident.

Step 6 — Keep the DLQ at Zero

The discipline is simple and frequently skipped: every message either gets fixed and replayed, or gets explicitly recorded and discarded. Leaving "we'll look at those later" residue in the queue is what turned the original depth alert into noise, and no threshold tuning recovers from it. Replay mechanics are covered in replaying dead-letter messages in RabbitMQ and the SQS redrive path.

Step 7 — Distinguish Expected Failures from Real Ones

Some failures are permanent by design — a webhook endpoint that has been deleted, a tenant that cancelled mid-job. These will dead-letter forever, and if they share a DLQ with genuine defects the alert becomes unusable within a month.

Give them a separate destination. Classify at the point of failure: known-terminal business outcomes go to a rejected queue that is monitored on a weekly report rather than paged on, while everything else goes to the DLQ that pages. The test for which is which is whether an engineer would do anything differently on seeing it — if the answer is "nothing, that account is gone", it does not belong in the alerting path.

This is worth revisiting quarterly, because the boundary moves. A failure that was a genuine defect gets fixed and becomes impossible; a business outcome that was rare becomes common after a product change. Reviewing the top exception classes by volume and asking "did anyone act on these?" is a ten-minute exercise that keeps the page meaningful, and it is the same reasoning that makes an error budget useful rather than decorative — see SLOs & alerting for job queues.

Step 8 — Make Triage Cheap Enough to Actually Happen

Alerts only work if responding to them is quick. Most DLQ neglect traces back to triage requiring a laptop, a VPN, an AWS console, and twenty minutes — so it gets deferred, and deferral becomes permanent. Two small investments change that.

First, a read-only summary command that answers "what is in there" without touching the messages:

# Group the DLQ by reason and age band — the whole triage decision in one view
dlq-summary --queue orders-dlq
# reason                    count   oldest    newest    sample_id
# SchemaValidationError       312   2h14m     4m        01J9F2...
# PaymentProviderGone          18   3d02m     3d01m     01J8Q7...
# TenantDeletedError            7   6d11m     1d04m     01J7M1...

Second, a replay command that is safe by construction: it takes a reason filter and a batch size, refuses to run without both, and reports what it moved. Batching by reason rather than by arrival order is what makes replay safe after a fix — you replay exactly the class of message the fix addressed, verify it drains, and leave the rest quarantined.

With those two commands, triage is a five-minute task that fits inside an on-call shift, and the alert stops being something to acknowledge and start being something to resolve. Without them, no amount of alert tuning helps, because the bottleneck was never detection.

Verification

1. A single poison message pages appropriately. Inject one; the trickle alert should fire after its confirmation window, not instantly.
2. A storm pages immediately. Inject sixty in a minute; the storm alert must fire without waiting.
3. Age escalates. Leave a message for the stale threshold and confirm the ticket alert appears.
4. Cause is in the notification. The alert text must name the dominant exception class without anyone opening a dashboard.
5. Zero is reachable. After triage, depth must return to zero — if it cannot, the residue needs an explicit decision.

Gotchas & Edge Cases

A shared DLQ across queues. Mixed causes and mixed owners make every alert ambiguous. One DLQ per source queue is the baseline.

Depth alerts with no age condition. They fire during triage — while someone is actively working the queue — and go silent only when the backlog is cleared, which is precisely backwards.

Exception messages as label values. "user 8891 not found" creates one series per user. Class names only.

No alert on the DLQ's own consumer. If a redrive process is running, it can fail silently. Alert on DLQ depth falling to zero and the replay counter incrementing, not on depth alone.

Retention shorter than triage latency. Messages deleted by the broker before anyone looked at them are silent data loss. Set retention to the maximum and alert well before it.

Related

• Dead-Letter Queues & Poison-Message Handling — quarantine mechanics and redrive policy.
• Configuring an SQS redrive policy — the settings that decide what arrives here.
• Replaying dead-letter messages in RabbitMQ — the fix-and-replay half of the triage loop.
• SLOs & Alerting for Job Queues — where DLQ alerts sit in the wider alerting model.