Exactly-Once vs At-Least-Once Delivery

Network partitions and process crashes make true exactly-once delivery across distributed systems impossible in the general case. Modern backend architectures achieve practical exactly-once semantics through application-level idempotency, transactional outboxes, and precise broker configurations.

Key architectural considerations include:

  • The CAP theorem's impact on message routing and network partitions
  • Why at-least-once serves as the default for high-throughput distributed systems
  • Designing consumers that safely handle duplicate payloads
  • Operational trade-offs between broker-side guarantees and application-level deduplication
Delivery semantics compared on consumer crash At-most-once may drop the message, at-least-once redelivers and may duplicate side effects, and practical exactly-once redelivers but an idempotency check suppresses the duplicate. At-most-once At-least-once Exactly-once* deliver once crash = lost deliver + retry duplicate effect deliver + retry idempotency drops dup one effect * practical exactly-once = at-least-once delivery + application-level idempotency

Theoretical Foundations & Distributed Systems Reality

Network partitions, packet loss, and clock drift make absolute exactly-once delivery mathematically unattainable in the general case. The CAP theorem dictates that distributed systems must sacrifice consistency or availability during partitions. Message brokers prioritize availability and partition tolerance.

Understanding baseline routing and acknowledgment flows requires a solid grasp of Queue Fundamentals & Architecture. Brokers rely on persistent storage and consumer heartbeat mechanisms to track message state. When a consumer crashes mid-processing, the broker cannot distinguish between a slow worker and a failed node.

Consequently, the broker requeues the message to ensure delivery. This safety mechanism inherently produces at-least-once semantics. Engineers must design systems assuming duplicate delivery is a baseline operational reality, not an edge case.

At-Least-Once Delivery: The Industry Standard

At-least-once delivery relies on explicit acknowledgment workflows. Producers publish messages to durable storage. Consumers pull payloads, process them, and send explicit ACK or NACK signals. Unacknowledged messages trigger automatic retries.

Properly configuring the Visibility Timeout Deep Dive prevents duplicate processing spikes during consumer scaling events. Visibility timeouts must exceed the p99 processing latency of your workload. Setting timeouts too low causes premature redelivery. Setting them too high delays failure recovery.

Design stateless consumers that tolerate duplicate payloads. Implement exponential backoff with jitter for retry policies. Configure dead-letter queue routing to isolate poison messages after max retry thresholds. This approach maximizes throughput while maintaining fault tolerance.

What each contract does when the acknowledgement is lost A worker completes a job but its acknowledgement is lost. Under at-most-once the message was already deleted, so the work is lost. Under at-least-once the broker redelivers and the side effect happens twice. Under effectively-once the broker still redelivers, but a deduplication key makes the second execution a no-op. The ack is lost. Now what? at-most-once work is lost, silently at-least-once redelivered, effect repeats effectively-once customer never gets the receipt customer gets two receipts dedup key: second run is a no-op exactly one receipt unrecoverable visible to the user

Achieving Practical Exactly-Once Semantics

Practical exactly-once behavior is engineered at the application layer. It combines at-least-once broker delivery with deterministic idempotency controls. Consumers generate or extract unique keys from incoming payloads.

Store these keys in a distributed, persistent datastore. Use database-level unique constraints or Redis SET NX operations to block duplicate execution. Upsert operations ensure subsequent payloads return cached results without side effects. A comprehensive implementation guide is detailed in Preventing duplicate job execution with idempotency.

Deploy the transactional outbox pattern for cross-system consistency. Write business state and outbox records in a single local database transaction. A background poller publishes outbox entries to the queue. This guarantees message publication matches committed state.

Broker-Level Exactly-Once: Kafka Streams & Pulsar

Modern streaming brokers offer native exactly-once semantics (EOS). Kafka EOS v2 utilizes transactional producers and consumer offset commits. Producers batch writes into atomic transactions. Consumers read only committed data via isolation.level=read_committed.

Apache Pulsar implements native transactional messaging with coordinated cursor management. Both systems guarantee exactly-once processing within their internal ecosystems. However, broker-side deduplication introduces measurable latency and throughput penalties.

Evaluating native versus application-level guarantees is essential when reviewing the Message Broker Comparison. Streaming brokers require careful partition tuning and transaction timeout configuration. Misconfigured transactional IDs cause producer fencing and throughput degradation.

Operational Workflows & Scaling Strategies

Platform teams must monitor duplicate detection rates and idempotency cache hit ratios. High duplicate rates indicate visibility timeout misalignment or consumer scaling thrash. Track DLQ ingestion velocity to identify systemic payload corruption.

Horizontal scaling requires careful partition affinity management. Consumer group rebalancing triggers temporary message stalls. Deploy sticky partition assignments to minimize state migration during scaling events. Share idempotency state across pods using Redis Cluster or a distributed relational database.

Balance cost against reliability using a service-tier matrix. Fire-and-forget analytics pipelines tolerate at-least-once delivery. Financial reconciliation and inventory updates require strict idempotency controls. Align broker selection and consumer architecture with your SLA requirements.

Code Examples

Idempotency Middleware (Node.js)

const redis = require('redis');
const client = redis.createClient({ url: process.env.REDIS_URL });

// Production middleware: intercepts payloads, checks idempotency store
async function idempotencyMiddleware(req, res, next) {
    const idempotencyKey = req.headers['x-idempotency-key'] || req.body.idempotency_key;
    if (!idempotencyKey) return next(new Error('Missing idempotency key'));

    try {
        // TTL prevents unbounded cache growth; adjust based on SLA
        const cached = await client.get(`idemp:${idempotencyKey}`);
        if (cached) {
            // Short-circuit duplicate execution, return cached response
            return res.status(200).json(JSON.parse(cached));
        }

        // Attach key to request context for downstream handlers
        req.idempotencyKey = idempotencyKey;
        next();
    } catch (err) {
        // Fail-open strategy: allow processing if Redis is unavailable
        console.error('Idempotency check failed:', err);
        next();
    }
}

Kafka EOS Producer Configuration

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import java.util.Properties;

// Producer EOS configuration
Properties producerProps = new Properties();
// Enable idempotence to prevent duplicate sends during retries
producerProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Unique transactional ID per producer instance; prevents fencing collisions
producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "tx-payment-processor-1");
// With enable.idempotence=true, max.in.flight defaults to 5 — leave it at that
producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
producerProps.put(ProducerConfig.ACKS_CONFIG, "all");

// Consumer configuration — isolation.level must be set on the consumer
Properties consumerProps = new Properties();
// Ensure consumers only read committed transactions
consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

Transactional Outbox Poller

import psycopg2
import time
from queue_publisher import publish_batch

def poll_outbox(db_conn, limit=100, backoff_sec=0.5):
    """Background worker reads unsent outbox records, publishes, and marks sent."""
    while True:
        try:
            with db_conn.cursor() as cur:
                # Lock rows to prevent concurrent poller collisions
                cur.execute("""
                    SELECT id, payload FROM outbox
                    WHERE status = 'PENDING'
                    ORDER BY created_at ASC
                    LIMIT %s FOR UPDATE SKIP LOCKED
                """, (limit,))
                records = cur.fetchall()

                if not records:
                    time.sleep(backoff_sec)
                    continue

                # Publish to broker outside DB transaction to avoid holding locks
                publish_batch(records)

                # Mark as sent in a separate atomic update
                ids = [r[0] for r in records]
                cur.execute("UPDATE outbox SET status = 'SENT' WHERE id = ANY(%s)", (ids,))
                db_conn.commit()
        except Exception:
            db_conn.rollback()
            time.sleep(backoff_sec * 2)
            backoff_sec = min(backoff_sec * 2, 30)

Common Pitfalls

  • Assuming broker-level exactly-once guarantees eliminate the need for application-level idempotency
  • Over-engineering deduplication for low-risk, fire-and-forget async notifications
  • Failing to tune visibility timeouts, causing duplicate processing spikes during consumer scaling events
  • Ignoring idempotency key collisions during high-concurrency retries without exponential backoff
  • Storing idempotency state in ephemeral memory instead of a persistent, distributed datastore
Deduplication key lifetime versus the retry window A timeline. The deduplication key is written in the same transaction as the side effect and given a time to live. The maximum retry span, including dead-letter retention and replay, must fit inside that lifetime, or a replayed message finds no key and repeats the effect. The dedup key must outlive every possible replay dedup key TTL — 30 days retry window dead-letter retention 14 days replay key expired here a replay now duplicates Write the key transactionally with the side effect, or a crash between the two reopens the gap. Derive the key from business identity, not from the broker message id, which changes on redelivery.

The Cost of Each Guarantee

Delivery guarantees are usually discussed as correctness properties, which hides the fact that they are primarily cost decisions. Each step up the ladder buys certainty with latency, throughput, and operational surface, and the increments are not small.

At-most-once is nearly free: acknowledge on receipt, process afterwards, and accept that a crash loses the message. It is the right choice more often than its reputation suggests — a metrics sample, a cache invalidation that the next write will fix, a best-effort notification that the user can trigger again. Choosing it deliberately for that class of work removes machinery you would otherwise carry for no benefit.

At-least-once costs an acknowledgement round trip after processing and a redelivery whenever that acknowledgement is lost. In exchange it guarantees the work happens. This is the default in every mainstream broker because the cost is modest and the guarantee is the one most systems need.

Effectively-once — at-least-once plus deduplication — adds a durable write per job for the key, and that write must be transactional with the side effect. On a busy queue that is a real cost: an extra row, an extra index, and a storage-retention obligation measured in weeks. It is worth paying wherever a duplicate is visible to a user or moves money, and worth skipping where it is not.

True exactly-once, in the sense of a distributed transaction spanning broker and application state, costs coordination on every message: two-phase commit or its equivalent, a coordinator that becomes a new failure domain, and throughput typically an order of magnitude below the same system running at-least-once. Kafka's transactional producer and Pulsar's transactions implement a restricted version of this within their own boundaries, and even there the guarantee stops at the edge of the system — the moment your handler calls an external API, you are back to at-least-once with an idempotency key.

The practical framing is that most systems should be at-least-once everywhere, with deduplication applied selectively to the handlers whose side effects are user-visible or financial. That gives correctness where it matters and avoids paying for it where it does not, and it is a decision you can make per handler rather than per system.

Where Duplicates Actually Come From

Before choosing a delivery contract it helps to know which failure modes produce duplicates in practice, because they are not evenly distributed and the mitigations differ.

Acknowledgement loss is the largest source in most systems. The work completed, the acknowledgement did not arrive, and the broker redelivers. No configuration prevents this; only an idempotent handler makes it harmless. It happens on every network blip and, at scale, constantly.

Lease expiry is the second: a job outruns its visibility timeout and is redelivered while still running, so two workers process it concurrently. Unlike acknowledgement loss, this one is directly controllable — it is a sizing problem, and a correctly sized window plus a heartbeat removes nearly all of it.

Deploys and scale-downs are the third and are self-inflicted. Every worker terminated without a drain releases everything it held, and a fleet that deploys several times a day generates a steady background rate of duplicates that teams often mistake for broker behaviour. The fix is the drain window described in graceful shutdown & worker deployments, and the effect is usually dramatic because this source is entirely eliminable.

Producer retries are the fourth and the most easily overlooked, because they happen before the queue is involved at all. A publish that times out ambiguously — the broker received it, the response was lost — leads a well-behaved client to republish, and now two genuinely distinct messages carry the same work. Broker-side deduplication windows help within a few minutes; beyond that, only a business-level key does.

Replay is the fifth: dead-letter redrive, a manual re-enqueue after an incident, a backfill run twice. These are deliberate duplicates, which makes them easy to forget and expensive when the deduplication key has already expired.

Quantifying the mix is worth an afternoon. Stamp each execution with the message identity and attempt number, count executions where the attempt is greater than one, and group by whether a deploy, a lease expiry, or neither preceded them. Most teams discover that one source dominates — usually deploys or lease expiry — and that it is the cheap one to fix. Doing that first reduces duplicate volume by an order of magnitude and leaves acknowledgement loss as the irreducible remainder, which is exactly the part idempotency exists to handle.

The measurement also settles the recurring architectural argument. A team that can say "we see four duplicates per million, all from acknowledgement loss, all absorbed by dedup keys" is in a position to reject the complexity of a transactional exactly-once system with evidence. A team that cannot measure it tends either to over-engineer against a phantom or to under-engineer against a real and growing rate.

Designing Idempotent Handlers in Practice

"Make it idempotent" is easy to say and has three distinct implementations, each appropriate to a different kind of side effect.

Natural idempotency is the cheapest and should be preferred wherever the operation allows it. An upsert keyed on a business identifier, a SET rather than an INCR, a state transition guarded by its current state — these are safe to repeat by construction, with no extra bookkeeping. Rewriting an operation to be naturally idempotent is often a small change: balance = balance - 10 is not repeatable, while balance = 90 WHERE balance = 100 is.

A deduplication key covers operations that cannot be made naturally idempotent. The key must be derived from the business identity of the work rather than from the message — a broker redelivery frequently carries a new message ID, so keying on it deduplicates nothing. The key must be written in the same transaction as the side effect it guards; a key written afterwards leaves a window in which a crash produces the effect with no record of it, and a key written beforehand leaves a window in which a crash blocks a legitimate retry.

A conditional external call handles the case where the side effect lives in another system entirely. Most payment providers, email services, and messaging APIs accept an idempotency key on the request and will return the original result rather than performing the action twice. Passing your own deduplication key through to them extends the guarantee across the boundary, which is the only way to make a remote side effect safe under redelivery.

The failure mode to watch for is partial idempotency: a handler that performs three side effects and guards only the expensive one. On the second execution the guarded step is skipped and the other two repeat, which produces a state nobody designed. Either guard the whole handler with one key at the start, or make each step independently safe — the middle ground is where the confusing bugs live.

Finally, be explicit about the deduplication window. A key retained for an hour cannot protect against a replay from a dead-letter queue two weeks later. Retention should exceed the longest path from enqueue to final processing, including dead-letter retention and any manual replay, which usually means weeks rather than hours.

FAQ

Is true exactly-once delivery possible in distributed systems? Theoretically, no in the general case. Network partitions, node failures, and clock drift make absolute exactly-once delivery impossible under the CAP theorem. Practically, engineers achieve effectively exactly-once semantics by combining at-least-once broker delivery with application-level idempotency and transactional outboxes.

How do I implement idempotency for database writes in async jobs? Generate a deterministic idempotency key from the job payload or request context. Before executing the write, check a persistent store (Redis or relational DB) for the key. If absent, execute the write and store the key with the result. If present, return the cached result without re-executing.

When should I use at-least-once over exactly-once? Use at-least-once when throughput, low latency, and cost efficiency are prioritized over strict data consistency. Examples include logging, analytics ingestion, or non-critical notifications. Use exactly-once (or practical idempotency) for financial transactions, inventory updates, or operations where duplicate execution causes data corruption.

Does Kafka's exactly-once guarantee eliminate the need for application-level deduplication? Kafka EOS guarantees exactly-once processing within the Kafka ecosystem (producer to broker to consumer offsets). However, if your consumer writes to external systems (PostgreSQL, S3, third-party APIs), you still need application-level idempotency or transactional outboxes to guarantee exactly-once end-to-end.

How long should an idempotency key live before it can be safely expired? Set the TTL to at least the maximum possible redelivery window — visibility timeout times max retries, plus any DLQ replay delay. For most workloads 24-72 hours is safe. If a key expires before a late duplicate arrives, the duplicate executes again, so align the TTL with your retry policy rather than a round number. The idempotency implementation guide covers TTL tuning in depth.

What happens to messages that keep failing even with at-least-once retries? After the configured max receive count they should route to a dead-letter queue for inspection rather than retrying forever. Without a DLQ, a single poison message can saturate a partition and stall otherwise healthy work. Monitor DLQ ingestion velocity to detect systemic payload corruption early.

Does enabling exactly-once on the broker hurt throughput? Yes — transactional producers and read-committed consumers add coordination and a transaction-timeout window, typically reducing throughput and adding latency versus plain at-least-once. For high-volume, non-critical streams, plain at-least-once with idempotent consumers is usually cheaper and faster than broker-native exactly-once.

Related