Table formats compared: Iceberg, Delta Lake, Hudi and Paimon¶

Optional

The main open table formats solve the same problem, turning files into tables, with different designs and ecosystems; this page compares them and explains how they interoperate.

Before reading

Apache Iceberg: table format basics · Catalogs

Why it matters¶

This wiki uses Iceberg in its examples, but many platforms use Delta Lake, and some use Hudi or Paimon. The choice affects which engines and catalogs can use the tables, how updates and streams perform, and how hard it is to change later. Knowing the differences helps readers apply the rest of the wiki to whichever format they meet.

How it works¶

All four formats share the core idea: data in open columnar files, usually Parquet, plus metadata that records which files make up each version of a table, with atomic commits, schema evolution and time travel. They differ in how the metadata is organised, which workloads they optimise for, and who governs them.

Apache Iceberg keeps a tree of metadata files and manifests, tracks columns by ID, and hides partitioning behind transforms. It was designed for many engines reading and writing the same tables, and has the broadest support across engines and catalogs, including the REST catalog specification. Format version 3, ratified in 2025, added deletion vectors, a variant type for semi-structured data, and row lineage. It is an Apache project with contributors from many vendors.

Delta Lake keeps a transaction log: a folder of ordered JSON commit files, periodically summarised into checkpoints. It originated at Databricks and is integrated most deeply with Spark and Databricks, where its catalog is Unity Catalog. It introduced deletion vectors, which Iceberg later adopted. It is a Linux Foundation project.

Apache Hudi was built for fast upserts and incremental processing on data lakes. It keeps a timeline of actions and offers record-level indexes, so updates by key are cheap, and two table types: copy-on-write and merge-on-read. It is strongest in Spark and Flink ingestion, with more operational settings to tune.

Apache Paimon was built for streaming. It stores data in a log-structured merge tree, so continuous inserts and updates from Flink or CDC are efficient, and offers changelog output for downstream streams. It is Flink-first, with growing support elsewhere.

flowchart TB
  subgraph Core[Shared core]
    direction LR
    F[(Parquet files)] ~~~ M[Metadata:<br/>versions, schema,<br/>atomic commits]
  end
  Core ~~~ D
  subgraph D[Designs]
    direction LR
    I[Iceberg:<br/>metadata tree,<br/>many engines] ~~~ DL[Delta Lake:<br/>transaction log,<br/>Spark and Databricks]
    DL ~~~ H[Hudi:<br/>timeline and indexes,<br/>fast upserts]
    H ~~~ P[Paimon:<br/>LSM tree,<br/>streaming]
  end

Interoperability has improved because the data files are the same and only the metadata differs. Delta Lake's UniForm writes Iceberg metadata alongside Delta's own, so Iceberg clients can read Delta tables. Apache XTable, incubating, translates metadata between Iceberg, Delta and Hudi. Paimon has an Iceberg compatibility mode. In practice, Iceberg has become the common format that other formats expose themselves as, because most engines and catalogs read it.

Choosing comes down to a few questions. Which engines must read and write the tables, and which catalog will govern them? If many engines and a neutral catalog matter, Iceberg is the usual choice; if the platform is centred on Databricks, Delta Lake is the natural one (Databricks also supports managed Iceberg tables, so check both). Is the main workload heavy keyed upserts or continuous streaming updates? Hudi and Paimon are built for those. And which format does the team already know: operating a format well matters more than small feature differences.

Key terms¶

Open table format: a specification that turns files into tables with versions and atomic commits.
Transaction log: Delta Lake's ordered record of commits.
Timeline: Hudi's record of actions on a table.
LSM tree: a log-structured merge tree, a storage design optimised for frequent writes.
UniForm: Delta Lake's feature that also writes Iceberg-compatible metadata.
Apache XTable: a project that translates metadata between table formats.
Deletion vector: a per-file record of deleted rows, used by Delta Lake and Iceberg v3.

Example¶

The shop runs Databricks for data science and Trino for reporting. Its data scientists create shop.ml_churn_scores as a Delta table with UniForm enabled:

CREATE TABLE shop.ml_churn_scores (customer_id STRING, churn_score DOUBLE, scored_at TIMESTAMP)
TBLPROPERTIES (
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Trino reads the same table through the Iceberg REST interface of the catalog, without a copy. Writes still go through Delta: Iceberg clients read it, but do not write to it.

Pitfalls¶

Choosing by benchmark. Engine and catalog support decide more than raw speed.
Assuming interoperability means two-way writes. Translated metadata is usually read-only for the other format.
Mixed formats without a plan. Each additional format adds maintenance jobs, tooling and knowledge to keep.
Feature differences between versions. Engines support different format versions and features; check deletion vectors and new types in particular.
UniForm with IcebergCompatV2 requires deletion vectors to be off on the Delta table; existing tables with deletion vectors need REORG before enabling it.

Learn more¶

Apache Iceberg, table spec: https://iceberg.apache.org/spec/
Delta Lake documentation, UniForm: https://docs.delta.io/delta-uniform/
Apache XTable: https://xtable.apache.org/

Data lake, warehouse and lakehouse · SQL query engines · Stream processing · Writing to Iceberg from Kafka