Skip to content

The outbox pattern

Optional

An application publishes events reliably by writing them to an outbox table in the same database transaction as its business change; change data capture then delivers the outbox rows to an event stream.

Before reading

Change data capture

Why it matters

Services often need to update their database and tell other systems what happened, for example save an order and publish OrderPlaced. Doing both directly can leave them out of step: an order saved with no event, or an event for an order that was never saved. The outbox pattern removes that gap, and gives the data platform deliberate business events instead of copies of internal tables.

How it works

The naive approach is a dual write: the service commits to its database, then sends a message to Kafka. If the service crashes between the two, the order exists but the event is lost. If it sends first and the database commit fails, the event describes something that did not happen. Retrying does not help, because the service cannot make two separate systems commit together.

With an outbox, the service writes only to its own database. In the same transaction as the business change, it inserts a row into an outbox table describing the event: an event ID, the type, the ID of the business object it concerns (the aggregate), and a payload. Either both rows are committed or neither is.

A CDC connector reads the outbox table's inserts from the database log and publishes each one to a topic, routed by event type or aggregate, keyed by aggregate ID so that events about one order stay in order. Debezium, for example, has a transform built for this, with the outbox column names mapped in its configuration. Because the log contains only committed transactions, every published event matches a committed change, and every committed change produces an event.

flowchart LR
  subgraph TX[One database transaction]
    direction TB
    A[INSERT INTO shop.orders] ~~~ B[INSERT INTO shop.outbox<br/>OrderPlaced]
  end
  S[Orders service] --> TX
  TX --> LOG[Transaction log]
  LOG --> C[CDC connector<br/>with outbox routing]
  C -->|key = order_id| T[Topic: shop.order-events]
  T --> D[Other services<br/>and the data platform]

Delivery is at least once: after a connector restart, an event can be published again. Consumers deduplicate by the event ID, which the service generated and which therefore never changes.

The outbox table does not need to keep rows. Since CDC reads the log rather than the table, rows can be deleted right after insertion, or by a regular cleanup job, without losing events.

For the data platform, the outbox has a second benefit. CDC on internal tables exposes the service's private schema, which changes whenever the service is refactored. Outbox events are a published interface: their structure is designed for consumers, can be covered by a contract, and can stay stable while the internal tables change.

Key terms

  • Dual write: writing to two systems separately, with no way to commit both together.
  • Outbox table: a table in the service's database holding events to publish.
  • Aggregate: the business object an event is about, such as one order.
  • Event ID: a unique, producer-generated ID used by consumers to deduplicate.
  • Outbox routing: sending outbox rows to topics by event type or aggregate.

Example

The orders service places an order:

BEGIN;
INSERT INTO shop.orders (order_id, customer_id, amount, status)
VALUES ('o-1002', 'c-88', 129.00, 'placed');

INSERT INTO shop.outbox (event_id, aggregate_type, aggregate_id, event_type, payload)
VALUES ('evt-5f1c', 'order', 'o-1002', 'shop.order.placed.v1',
        '{"order_id":"o-1002","customer_id":"c-88","amount":129.00}');
COMMIT;

The CDC connector publishes the outbox row to shop.order-events with key o-1002. The data platform lands it in raw_order_events and deduplicates on event_id. If the transaction is rolled back, neither the order nor the event exists anywhere.

Pitfalls

  • Publishing the internal row as the payload. The event is then as unstable as the table it was meant to hide.
  • Missing event IDs. Without them, consumers cannot recognise repeated deliveries.
  • Large payloads in the outbox. They grow the transaction log; send references to large data instead.
  • Forgetting cleanup. An outbox table that is never emptied grows forever, even though no one reads it.

Learn more

  • Chris Richardson, Pattern: Transactional outbox: https://microservices.io/patterns/data/transactional-outbox.html
  • Debezium documentation, outbox event router: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html

Delivery guarantees and idempotency · Data contracts · Kafka and Kafka Connect