Skip to content

Delivery guarantees and idempotency

A delivery guarantee says how often a message can arrive when things fail: at most once, at least once, or exactly once. Idempotency is the property that processing the same message twice has the same effect as processing it once.

Before reading

Kafka and Kafka Connect

Why it matters

Networks drop connections, processes crash and retries happen. A pipeline must decide what it does when it cannot tell whether a message got through. Most pipelines choose to retry, which means duplicates are normal, and every later step has to be safe to run on duplicates.

How it works

Every delivery step has the same problem. A sender sends a message, and the acknowledgement does not come back. Either the message was lost, or it arrived and the acknowledgement was lost. The sender cannot know which.

At-most-once delivery does not retry. No duplicates are created, but messages can be lost. It is acceptable only where losing some data does not matter, such as some metrics.

At-least-once delivery retries until it gets an acknowledgement. Nothing is lost, but a message can arrive two or more times. This is the default in most pipelines. A consumer has the same choice: if it processes a batch and crashes before committing its offset, it reads the batch again after restarting.

Exactly-once means each message affects the result once. It cannot be achieved by delivery alone; it comes from combining at-least-once delivery with deduplication or transactions. Kafka offers two building blocks. An idempotent producer, the default since Kafka 3.0, attaches a sequence number to each record so that a broker drops retried duplicates from the same producer session. Transactions let an application read from topics, write to topics and commit its offsets atomically, so a crash never leaves partial results visible to consumers that read only committed data. The idempotent producer removes only the client's own internal retries; an application that sends the same event again, after a timeout or a restart, creates a real duplicate. These guarantees hold inside Kafka. As soon as data leaves Kafka, for a database, an API or a table, the receiving side must cooperate.

sequenceDiagram
  participant P as Payments service
  participant K as Kafka
  participant S as Sink to lake
  participant T as raw_payments
  P->>K: payment p-77 captured
  K--xP: acknowledgement lost
  P->>K: after a timeout, the service sends p-77 again<br/>(a new send, not a producer retry)
  K->>S: record 1 and record 2, same payment_id
  S->>T: both written
  Note over T: Duplicate rows. Safe only if later steps<br/>use payment_id to keep one.

Duplicates come from many places: producer retries, consumer restarts before an offset commit, connector restarts, sources that send the same event twice, and reruns of batch jobs. The defence is idempotency at the point where data is written.

An idempotent write uses a key that identifies the event or the row. Merging on that key, instead of appending, means the second copy updates the first instead of adding a row. Writing a batch to a fixed location, such as replacing one partition, means a rerun overwrites rather than duplicates. For this to work, each event needs a stable ID set by its producer, not by the pipeline, so that a retried event carries the same ID as the original.

A common lakehouse approach is to accept duplicates in the raw layer, which keeps ingestion simple and complete, and remove them when building the clean layer, using the event ID and the source's ordering.

Key terms

  • At-most-once: never retried; may lose messages.
  • At-least-once: retried until acknowledged; may duplicate messages.
  • Exactly-once: each message affects the result once, through deduplication or transactions.
  • Idempotent operation: one that gives the same result when repeated.
  • Idempotent producer: a Kafka producer whose retries are deduplicated by the broker.
  • Idempotency key: a stable ID used to recognise repeats of the same event.

Example

The payments service retries an event, and raw_payments receives two rows for p-77. Appending would count the payment twice. Merging on the event's own ID keeps one:

MERGE INTO shop.clean_payments AS t
USING (
  SELECT * FROM shop.raw_payments
  WHERE _ingested_at >= TIMESTAMP '2026-10-28 00:00:00 UTC'
) AS s
ON t.payment_id = s.payment_id
WHEN MATCHED THEN UPDATE SET status = s.status, amount = s.amount
WHEN NOT MATCHED THEN INSERT (payment_id, order_id, status, amount)
  VALUES (s.payment_id, s.order_id, s.status, s.amount);

If the source holds the same payment_id twice, MERGE fails when the payment already exists in the target, and inserts both copies when it does not. So in practice the source is first reduced to one row per payment, as shown in Deduplication and merge. Running the job again gives the same table: it is idempotent.

Pitfalls

  • Believing a product's "exactly-once" covers the whole pipeline. It covers its own boundary; every write outside it must be idempotent too.
  • IDs generated by the pipeline. A retried event gets a new ID, and duplicates can no longer be recognised.
  • Appending in batch jobs. A rerun after a partial failure doubles the data.
  • Side effects outside the data. A retried step that sends an email or calls an API repeats that effect unless it is guarded by a key.

Learn more

  • Apache Kafka documentation (message delivery semantics section): https://kafka.apache.org/documentation/
  • Martin Kleppmann, Designing Data-Intensive Applications (1st edition, 2017), chapter 11, fault tolerance and exactly-once.

Deduplication and merge · Late and out-of-order data · Writing to Iceberg from Kafka · The outbox pattern