Claim-Check Pattern for Large Job Payloads

When a job's data exceeds what a broker should carry, the fix is to stop carrying it — a direct application of the limits described in Message Size Limits & Serialization under Queue Fundamentals & Architecture.

Problem Statement

Your document-processing jobs embed the uploaded file as base64 in the message body. Most are under a megabyte, but a slow tail reaches forty, and SQS rejects anything over 256KB while RabbitMQ accepts the large ones and then suffers: memory pressure on the broker, slow redelivery, and a queue whose depth in bytes bears no relation to its depth in messages. You want messages to stay small and uniform regardless of how large the underlying data is.

Prerequisites

  • Object storage the producer can write and every worker can read — S3, GCS, or MinIO.
  • A lifecycle policy capability on that bucket, for orphan cleanup.
  • An idempotent consumer, because a claim check is read at least once.
  • Agreement on a retention window: how long a payload must remain readable after enqueue.

Step 1 — Decide What Belongs in the Message

The message should carry everything needed for routing and scheduling, and nothing else. Anything the consumer needs only after deciding to run belongs behind the reference.

Field In the message Why
Job type, tenant, priority Yes Needed to route and schedule before any work starts
Idempotency key Yes Needed to deduplicate without a storage round trip
Storage URI + version/etag Yes The claim check itself
Content hash, size Yes Cheap integrity check before download
The document, image, or dataset No Belongs in object storage
Rendered output No Written back to storage; the message carries the key
Claim-check flow from producer to worker A producer writes a large payload to object storage, receives a key, and enqueues a small message containing only that key, a content hash and routing metadata. A worker receives the small message, fetches the payload from object storage using the key, verifies the hash, processes it, and writes any result back to storage. The broker carries a key, not the cargo producer 40 MB upload 1 · PUT object object storage s3://uploads/2026/07/ab12… 2 · enqueue 400 bytes queue { key, sha256, tenant } 3 · receive worker verifies hash 4 · GET object by key Message size becomes uniform, so queue depth means the same thing for every job. The cost is one extra round trip and a lifecycle policy to clean up orphans.

Step 2 — Write the Payload, Then Enqueue

Order matters: the object must exist before any consumer can see the reference.

import hashlib, json, uuid, boto3

s3 = boto3.client("s3")
BUCKET = "job-payloads"

def enqueue_with_claim_check(task: str, blob: bytes, tenant: str) -> str:
    digest = hashlib.sha256(blob).hexdigest()
    # Content-addressed key: identical payloads dedupe for free, and a retry
    # that re-uploads writes the same bytes to the same place.
    key = f"{tenant}/{digest[:2]}/{digest}"

    put = s3.put_object(Bucket=BUCKET, Key=key, Body=blob,
                        ServerSideEncryption="aws:kms",
                        Metadata={"task": task, "tenant": tenant})

    # Only now is the reference safe to publish.
    return queue.publish(task=task, body=json.dumps({
        "bucket": BUCKET, "key": key, "etag": put["ETag"],
        "sha256": digest, "size": len(blob), "tenant": tenant,
        "idempotency_key": f"{task}:{digest}",
    }))

Content addressing pays off twice: the same upload submitted twice costs one object, and a producer retry after an ambiguous timeout is harmless because it rewrites identical bytes to an identical key.

Step 3 — Fetch, Verify, Process

def handle(msg: dict) -> None:
    ref = json.loads(msg["body"])
    try:
        obj = s3.get_object(Bucket=ref["bucket"], Key=ref["key"])
    except s3.exceptions.NoSuchKey:
        # The payload is gone: lifecycle expiry, or a producer that never
        # completed the upload. This is permanent — do not retry it.
        raise PermanentFailure(f"claim check missing: {ref['key']}")

    blob = obj["Body"].read()
    if hashlib.sha256(blob).hexdigest() != ref["sha256"]:
        raise PermanentFailure("payload hash mismatch — refusing to process")

    process(blob, tenant=ref["tenant"])

Classifying a missing object as permanent matters. Retrying it burns the attempt budget on something that will never appear; dead-lettering it immediately puts a real, diagnosable message in front of a human, which is exactly the behaviour argued for in retrying only transient errors by exception type.

Step 4 — Set Retention Longer Than the Worst Case

The object must outlive every possible delay between enqueue and final processing: the queue backlog, the full retry curve, the dead-letter retention, and any replay you might perform.

def payload_retention_days(max_backlog_h: float, retry_window_h: float,
                           dlq_retention_d: int) -> int:
    """Objects must survive the longest path from enqueue to the last replay."""
    hours = max_backlog_h + retry_window_h + dlq_retention_d * 24
    return int(hours / 24) + 7        # a week of slack for manual replay

payload_retention_days(4, 1, 14)      # 21 days
{
  "Rules": [{
    "ID": "expire-job-payloads",
    "Filter": {"Prefix": ""},
    "Status": "Enabled",
    "Expiration": {"Days": 21},
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}
  }]
}

Setting retention to a day because "jobs run in seconds" is the classic failure: the one message that dead-letters, sits for a week, and is replayed finds nothing behind its claim check.

Step 5 — Clean Up Orphans

An orphan is an object whose message never arrived — a producer that crashed between the upload and the publish. Lifecycle expiry handles it eventually, but for large or sensitive payloads a shorter reconciliation is worth running.

def sweep_orphans(older_than_h: int = 24) -> int:
    """Objects with no corresponding job record are orphans. Requires the
    producer to write a job row transactionally — see the outbox pattern."""
    removed = 0
    for obj in iter_objects(BUCKET, older_than_h=older_than_h):
        digest = obj["Key"].rsplit("/", 1)[-1]
        if not db.job_exists(payload_sha=digest):
            s3.delete_object(Bucket=BUCKET, Key=obj["Key"])
            removed += 1
    return removed

If the producer writes a job row and the object reference in the same transaction — the transactional outbox pattern — orphans become detectable rather than guessed at.

Step 6 — Understand the Costs You Just Moved

What the claim check trades away Four paired bars comparing inline payloads with the claim-check pattern. Message size and broker memory fall dramatically with the claim check. Per-job latency rises slightly because of the extra storage round trip, and storage cost and operational surface rise because objects must be retained and cleaned up. Not free — the cost moves rather than disappearing message size 40 MB → 400 B broker memory large win per-job latency +1 round trip (~20–60 ms) inline payload claim check plus storage cost and a lifecycle policy to own

For jobs under roughly 64KB the round trip is not worth it — keep those inline. Between 64KB and the broker limit it is a judgement call driven by volume; above the limit there is no choice.

Step 7 — Secure the Reference

A claim check is a pointer to data, and the pointer travels through a broker that may have a wider audience than the storage bucket. Three practices keep that from becoming an exposure.

Grant workers read access scoped to the prefix they process, not to the whole bucket — a per-tenant prefix with a role condition means a compromised worker cannot enumerate other tenants' payloads. Prefer bucket policies and IAM over pre-signed URLs in the message: a pre-signed URL is a bearer token that lives as long as the message, survives in dead-letter queues, and is copied into logs by anyone debugging. If you must use one, keep the expiry short and re-sign inside the worker.

Encrypt at rest with a key the queue operator does not hold, so broker access alone does not yield the data. And exclude the payload from log lines entirely: it is now easy to log ref in full, which is small and looks harmless, but the object key frequently encodes tenant and content identity. Log the digest prefix rather than the whole reference, and the payload never appears in a log aggregator that has different retention rules than the bucket does.

Object retention must cover the longest path to processing A cumulative timeline. The queue backlog accounts for four hours, the retry window one hour, dead-letter retention fourteen days, and a manual replay margin adds a further week. The object lifecycle policy must exceed the sum, or a replayed message finds an expired claim check. Retention = backlog + retries + DLQ + replay margin 4h backlog 1h retries 14 days of dead-letter retention DLQ 7-day replay margin lifecycle expiry: 21 days minimum 1 day the mistake A one-day expiry looks generous next to a two-second job and loses every replayed message.

Verification

  1. Message size is uniform. Sample published messages; every one should be within a few hundred bytes regardless of payload size.
  2. A missing object dead-letters immediately. Delete an object, replay its message, and confirm it lands in the DLQ on attempt one.
  3. Retention outlives the DLQ. Take the oldest message in the dead-letter queue and confirm its object is still readable.
  4. Orphans are collected. Upload without enqueueing, run the sweep, and confirm the object disappears.

Gotchas & Edge Cases

Enqueueing before the upload finishes. A worker can receive the reference before the object is durable, producing a NoSuchKey for a payload that appears seconds later. Always upload first and publish second.

Lifecycle expiry shorter than DLQ retention. The most common production failure: replayed messages point at expired objects, and the replay is unrecoverable.

Object storage rate limits are per prefix. S3 scales by key prefix, so a date-based prefix concentrates every write of the day into one partition and starts returning SlowDown under load. A hash prefix — the first two characters of the digest, as used above — spreads writes across partitions automatically.

Cross-region reads. A worker in another region pays latency and egress on every fetch. Replicate the bucket or pin workers to the payload's region.

Large payloads still cost memory in the worker. The broker is protected; the worker is not. Stream the object rather than reading it whole where the format allows.

Results need the same treatment as inputs. A job that returns a 30MB rendered PDF through its result backend recreates the problem on the way out. Write the output to storage and return its key, and give result objects their own retention policy — usually longer than the input's, because a user may fetch the result days later.

Content-addressed keys leak equality. Two tenants uploading the same file produce the same digest and therefore the same key, which is efficient but means one tenant can test whether another has a given file. Where that matters, salt the key with the tenant identifier so identical content still lands in separate objects.

Encryption context must be available to the worker. A KMS key the worker's role cannot use turns every job into an access-denied failure that looks like a storage outage.

Related