Debugging Duplicate Deliveries in SQS

Duplicate delivery is a symptom, and the visibility timeout is nearly always the cause. This guide, part of Queue Fundamentals & Architecture, narrows a real duplicate down to one of five mechanisms and fixes each.

Problem Statement

Your order-processing consumer sends confirmation emails, and roughly 0.3% of customers receive two. CloudWatch shows NumberOfMessagesReceived about 15% higher than NumberOfMessagesDeleted, which means SQS is handing out messages that never get acknowledged. The consumer logs show no errors. You need to determine whether the duplicates come from timeout expiry, a crashed worker, a misconfigured poll, or an at-least-once delivery you simply have to tolerate — and then make the handler safe regardless.

Prerequisites

  • CloudWatch metrics for the queue, and consumer logs that record the message ID and receipt handle.
  • Permission to call GetQueueAttributes, ReceiveMessage with ApproximateReceiveCount, and ChangeMessageVisibility.
  • Known p50/p99 handler durations — every diagnosis below compares them against the timeout.

Step 1 — Capture the Receive Count

ApproximateReceiveCount is the single most informative field, and it is not returned unless you ask for it.

import boto3

sqs = boto3.client("sqs")
resp = sqs.receive_message(
    QueueUrl=QUEUE_URL,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,                      # long polling — see step 4
    AttributeNames=["ApproximateReceiveCount", "SentTimestamp",
                    "ApproximateFirstReceiveTimestamp"],
)
for msg in resp.get("Messages", []):
    attrs = msg["Attributes"]
    receive_count = int(attrs["ApproximateReceiveCount"])
    first_seen = int(attrs["ApproximateFirstReceiveTimestamp"]) / 1000
    if receive_count > 1:
        log.warning("redelivery", extra={
            "message_id": msg["MessageId"],
            "receive_count": receive_count,
            "seconds_since_first_receive": time.time() - first_seen,
        })

The gap between receives is the diagnosis. If it equals the visibility timeout almost exactly, the timeout expired while the handler was still running. If it is much shorter, the consumer crashed or the message was explicitly released. If it is much longer, something changed the visibility mid-flight.

Reading the cause from the gap between receives Three timelines. In the first, the second receive happens exactly one visibility timeout after the first, indicating the handler outran the timeout. In the second, the gap is much shorter than the timeout, indicating a worker crash or an explicit release. In the third, the gap is longer and irregular, indicating something extended or reset the visibility mid-flight. The gap between receives names the cause gap ≈ visibility timeout handler still running at 30s → timeout too short gap ≪ timeout crash → worker died, or visibility set to 0 on error gap ≫ timeout heartbeat extended it, then stopped two receives, no gap → genuine at-least-once duplicate first receive +60s Log receive count and first-receive timestamp on every message — this table is free after that.

Step 2 — Compare Timeout Against Real Runtime

The most common cause is boring: p99 handler duration exceeds the visibility timeout, so a slow-but-successful job is redelivered while it is still working.

# What the queue is actually configured with
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names VisibilityTimeout ReceiveMessageWaitTimeSeconds \
                    RedrivePolicy ApproximateNumberOfMessagesNotVisible
# The rule: timeout > p99 duration × safety factor, and > any batch's total time
def required_visibility_timeout(p99_seconds: float, batch_size: int = 1) -> int:
    # A batch is processed serially: the LAST message in the batch has been
    # invisible for the whole batch, not just for its own handler run.
    return int(p99_seconds * batch_size * 1.5) + 10

required_visibility_timeout(18)        # 37s  — single-message receive
required_visibility_timeout(18, 10)    # 280s — MaxNumberOfMessages=10

The batch multiplier catches many teams by surprise. With MaxNumberOfMessages=10 and an 18-second handler, the tenth message has been invisible for roughly three minutes before it is even touched, so a 60-second timeout guarantees it is redelivered.

Step 3 — Extend Visibility from a Heartbeat

For handlers whose duration varies widely, extend rather than over-provision. A long fixed timeout delays legitimate redelivery after a real crash.

import threading

def with_heartbeat(queue_url: str, receipt_handle: str, interval: int = 20,
                   extend_to: int = 60):
    """Keep a message invisible while the handler runs; stop on completion."""
    stop = threading.Event()

    def beat():
        while not stop.wait(interval):
            try:
                sqs.change_message_visibility(
                    QueueUrl=queue_url, ReceiptHandle=receipt_handle,
                    VisibilityTimeout=extend_to)
            except sqs.exceptions.ReceiptHandleIsInvalid:
                # Already deleted or expired — nothing left to extend.
                return
    t = threading.Thread(target=beat, daemon=True)
    t.start()
    return stop      # caller sets it in a finally block

Extending beyond the queue's 12-hour maximum silently fails, and a heartbeat thread that outlives its handler keeps a deleted message invisible — so always stop it in a finally.

Step 4 — Rule Out Poll and Client Misconfiguration

Three settings produce duplicates that look like timeout problems but are not.

Setting Bad value Effect
WaitTimeSeconds 0 (short poll) Samples a subset of servers; empty responses and uneven delivery
Client socket timeout < WaitTimeSeconds The SDK abandons a request whose messages SQS already marked received
VisibilityTimeout on ReceiveMessage overrides the queue default A per-call value silently shortens the window
Multiple consumer groups same queue Every consumer receives every message — you need one queue per consumer

Short polling deserves particular attention: with WaitTimeSeconds=0, SQS samples a subset of its servers, so a message can appear absent to one poll and present to the next. Always use long polling (20 seconds) unless you have a specific reason not to.

A client socket timeout below the long-poll duration is subtler and worse: the SDK gives up while SQS has already counted the receive and started the visibility clock. Nothing processes those messages until the timeout expires, and the retry looks exactly like a duplicate.

from botocore.config import Config
sqs = boto3.client("sqs", config=Config(
    read_timeout=25,        # MUST exceed WaitTimeSeconds (20)
    connect_timeout=5,
    retries={"max_attempts": 3, "mode": "standard"},
))

Step 5 — Make Duplicates Harmless

SQS standard queues are at-least-once by design. Even with a perfect timeout, the service can deliver a message twice — so the only complete fix is an idempotent handler.

import hashlib

def dedup_key(msg: dict) -> str:
    """Derive identity from the business payload, not the message ID: a genuine
    SQS duplicate has a new MessageId but the same content."""
    body = json.loads(msg["Body"])
    return hashlib.sha256(f"{body['order_id']}:{body['event']}".encode()).hexdigest()

def handle(msg: dict) -> None:
    key = dedup_key(msg)
    # SET NX is the whole mechanism: first writer wins, everyone else no-ops.
    if not r.set(f"dedup:{key}", "1", nx=True, ex=86_400):
        log.info("duplicate suppressed", extra={"key": key})
        return
    try:
        process(json.loads(msg["Body"]))
    except Exception:
        r.delete(f"dedup:{key}")     # allow a genuine retry after a real failure
        raise

FIFO queues offer server-side deduplication within a five-minute window via MessageDeduplicationId, which handles producer-side duplicates but not consumer-side redelivery. The application-level key covers both, and the patterns are developed further in preventing duplicate job execution with idempotency.

Step 6 — Watch the Right CloudWatch Metrics

SQS metric signatures for redelivery Two paired series. Number of messages received consistently exceeds number of messages deleted, and the difference is the redelivery rate. Separately, approximate number of messages not visible grows while the delete rate stays flat, which indicates handlers are holding messages longer than the visibility timeout allows. Received minus deleted is your redelivery rate NumberOfMessagesReceived NumberOfMessagesDeleted gap = redeliveries A healthy queue has these two lines nearly on top of each other. ApproximateAgeOfOldestMessage rising with a flat delete rate = handlers outrunning the timeout. 
# Redelivery ratio over the last hour
aws cloudwatch get-metric-statistics --namespace AWS/SQS \
  --metric-name NumberOfMessagesReceived --statistics Sum --period 3600 \
  --dimensions Name=QueueName,Value=orders --start-time "$(date -u -d '1 hour ago' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)"

Step 7 — Choose Between a Long Timeout and a Heartbeat

Both fixes stop the redelivery; they fail differently, and the difference matters once a worker actually crashes.

Long fixed timeout Heartbeat extension
Slow-but-healthy handler Safe Safe
Worker crash Message stuck invisible for the full timeout Message returns within one heartbeat interval
Configuration surface One queue attribute Code in every consumer
Failure mode Recovery delayed by minutes or hours Thread outlives handler, holds a deleted message
Best for Uniform, predictable durations Wide duration spread, or slow outliers

The decisive question is how quickly you need a crashed worker's message to be retried. With a 15-minute timeout sized for a p99 job, a worker that dies one second into a job leaves that message invisible for fifteen minutes — during which a customer is waiting and no alert fires, because depth looks normal and nothing is failing. A heartbeat with a 60-second base and a 20-second interval bounds that recovery at a minute regardless of how long the job would have taken.

The practical rule: if p50 and p99 are within a factor of two or three, set a fixed timeout from p99 and keep the consumers simple. If p99 is ten times p50 — common when the same handler processes both a two-row and a two-million-row account — heartbeat, and set the base timeout from something close to p50 so crash recovery stays fast. Mixing the two, a generous fixed timeout and a heartbeat, is the worst of both: you pay the code complexity and still wait the full timeout after a crash.

Batch receive multiplies the visibility requirement Ten messages are received in one call and processed serially at eighteen seconds each. The visibility clock starts for all ten at the moment of receipt, so the tenth message has been invisible for nearly three minutes before processing begins. With a sixty-second visibility timeout it is redelivered while the batch is still being worked through. MaxNumberOfMessages=10 starts ten clocks at once msg 1 processed at 0–18s — safe msg 5 waits 72s, processed at 72–90s msg 10 waits 162s — long gone 60s visibility timeout expires here receive +180s Size the timeout for batch_size x p99 duration, not for one message.

Verification

  1. Receive counts drop. After raising the timeout, messages with ApproximateReceiveCount > 1 should fall to near zero outside genuine failures.
  2. Received ≈ deleted. The CloudWatch gap should close to within a percent.
  3. Duplicates are suppressed anyway. Deliberately re-send an identical payload; the dedup key must short-circuit it and log a suppression.
  4. The heartbeat stops. After a handler completes, no further ChangeMessageVisibility calls for that receipt handle should appear in CloudTrail.

Gotchas & Edge Cases

Receipt handles change on every receive. A handle captured on receive one cannot delete the message after receive two. Always delete with the handle from the current receive.

Visibility timeout is capped at 12 hours. A heartbeat cannot extend past it. Jobs longer than that must checkpoint, as in configuring visibility timeouts for long-running workers.

Deleting after a partial failure creates silent data loss. If the handler half-succeeded, deleting hides the problem. Either make the whole handler idempotent and re-runnable, or let the message return and rely on the dedup key.

FIFO deduplication only covers five minutes and only the producer side. It is not a substitute for an idempotent consumer.

Lambda triggers manage visibility themselves. With an SQS event source, the function timeout must be below the queue's visibility timeout — AWS recommends at least six times the function timeout — or Lambda's own retries manufacture the duplicates you are chasing.

Related