Writing to Iceberg from Kafka¶

Moving events from Kafka topics into Iceberg tables, usually with a sink connector that buffers records, writes data files and commits them to the table at a fixed interval.

Before reading

Change data capture · Apache Iceberg: table format basics

Why it matters¶

Kafka keeps events for days; the lakehouse keeps them for years and makes them queryable with SQL. The step between the two decides how fresh the tables are, how many small files they get, whether changes from databases arrive as history or as current state, and whether a restart creates duplicates.

How it works¶

A sink reads records from one or more topics, converts each to a row, and writes rows to Parquet files in the table's location. Files are not visible until they are committed. At each commit interval, the sink commits all files written since the last commit as one new snapshot, together with the Kafka offsets they cover.

In the Apache Iceberg sink for Kafka Connect, worker tasks write files and report them on a control topic, together with their source offsets, in one Kafka transaction. A single coordinator collects the reports, commits the files to the table and records in the snapshot which reports it applied, so a restart neither loses nor repeats a commit. Files written but never committed are left as orphan files for maintenance to remove. This gives exactly-once delivery into the table, as long as only this sink writes those offsets. The default commit interval is five minutes.

flowchart TB
  T[Topic:<br/>orders-db.shop.orders] --> W1[Task 1]
  T --> W2[Tasks 2 to 4]
  W1 -->|write Parquet files| S[(Object storage)]
  W2 -->|write Parquet files| S
  W1 -->|files written| CT[Control topic]
  W2 -->|files written| CT
  CT --> CO[Coordinator]
  CO -->|commit every 5 min:<br/>files + offsets| CAT[Catalog:<br/>shop.raw_orders]

The commit interval is a trade-off. A shorter interval makes data fresher but produces more snapshots and smaller files, which compaction must later merge. A few minutes is typical for raw tables; seconds are rarely worth the maintenance cost. With four tasks committing every five minutes, raw_orders gets about 1,150 new files a day, averaging about 600 KB, which compaction later merges (see Iceberg maintenance).

There are two ways to write changes. Append mode adds every record as a new row. For CDC topics this produces a change log: every insert, update and delete of an order is a row, with the operation and source position kept as metadata columns. Upsert mode applies changes to the table directly: an update replaces the row with the same key and a delete removes it, using equality delete files. The table then holds only current state.

Append is the safer default for raw tables. It keeps full history, never conflicts, and is cheap to write and read. Current state is built in the clean layer with a MERGE that applies changes in log order. Upsert is attractive because the table is immediately current, but it relies on equality deletes, which slow reads until compaction (see Apache Iceberg: table format basics). The Apache Iceberg sink for Kafka Connect only appends (checked October 2026); upsert needs another writer, such as Flink's Iceberg sink with upsert enabled.

One sink can write to several tables. Static routing sends each topic to a configured table. Dynamic routing reads the target table name from a field of each record, which suits CDC connectors that capture many tables into many topics.

Stream processors such as Flink have Iceberg sinks with similar commit behaviour, and some streaming platforms expose topics as Iceberg tables directly.

Key terms¶

Sink connector: a Kafka Connect component that writes topic records to an outside system.
Commit interval: how often written files are committed to the table.
Control topic: the topic on which sink tasks and the coordinator exchange commit information.
Append mode: every record becomes a new row.
Upsert mode: records replace or delete existing rows by key.
CDC metadata columns: operation, source position and commit time kept alongside each change row.

Example¶

Order change events from CDC appended to raw_orders, in Kafka Connect configuration (abbreviated):

{
  "name": "orders-to-iceberg",
  "config": {
    "connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
    "topics": "orders-db.shop.orders",
    "tasks.max": "4",
    "iceberg.tables": "shop.raw_orders",
    "iceberg.control.commit.interval-ms": "300000",
    "iceberg.tables.evolve-schema-enabled": "true",
    "iceberg.catalog.type": "rest",
    "iceberg.catalog.uri": "https://catalog.example.com/api/catalog",
    "transforms": "cdc,source",
    "transforms.cdc.type": "org.apache.iceberg.connect.transforms.DebeziumTransform",
    "transforms.source.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.source.static.field": "_source",
    "transforms.source.static.value": "orders-db",
    "transforms.source.timestamp.field": "_ingested_at"
  }
}

The DebeziumTransform, marked experimental, flattens each change event into the row's columns plus a _cdc structure holding the operation, the Debezium event time, the Kafka offset, the source table, the target table and the record key. The operation is written as I, U or D. Within one key, Kafka offset order matches log order, because every change to a row goes to the same partition. The second transform adds _source, and _ingested_at from the Kafka record's timestamp: the time the event entered Kafka, up to one commit interval before the row is visible, which is why incremental readers overlap their window. With evolve-schema-enabled, new event fields become new columns. raw_orders therefore holds three rows for order o-1001 after it is created, updated and deleted. The tombstone after the delete has no value and is not written as a row; set Debezium's tombstones.on.delete to false if no compacted topic needs it. clean_orders is built from them, keeping the latest change per order and dropping orders whose latest change is a delete.

Pitfalls¶

A commit interval of seconds. It creates thousands of tiny files and snapshots per day; compaction then has to run constantly.
Two writers for the same offsets. Running the same sink twice, or another job writing the same topic to the same table, breaks the exactly-once guarantee.
Expecting upsert from an append-only sink. Check what the chosen writer supports; with the Apache Iceberg sink, current state is built downstream.
Losing CDC metadata. Without the operation and source position, deletes and ordering cannot be applied later.
Schema changes that the sink cannot apply. Decide whether the sink may evolve the table, and monitor for failed tasks.

Learn more¶

Apache Iceberg, Kafka Connect: https://iceberg.apache.org/docs/latest/kafka-connect/
Apache Iceberg, Flink writes: https://iceberg.apache.org/docs/latest/flink-writes/

Iceberg maintenance · Deduplication and merge · Delivery guarantees and idempotency · Stream processing