# Apache Iceberg: table format basics

Iceberg is an open table format: a set of rules for storing a table as plain files in object storage so that many engines can read and write it safely, as if it were a database table.

> [!NOTE]
>
> **Before reading**
>
> [Object storage and the S3 API](<https://wiki.4tm0.net/data/object-storage.html>) · [Columnar files and Parquet](<https://wiki.4tm0.net/data/columnar-files-parquet.html>) · [Partitioning](<https://wiki.4tm0.net/data/partitioning.html>)

## Why it matters

Files in a bucket are not a table. Without a format on top, two writers can corrupt each other, a reader can see half of a write, and renaming a column means rewriting everything. Iceberg adds what a database normally provides: transactions, schema changes and history. The data stays in open files that Trino, Spark, Snowflake, DuckDB and others can all use, so no single engine owns it.

## How it works

An Iceberg table is a tree of files. Data sits in Parquet files at the bottom. Everything above them is metadata that says which files make up the table at each point in time. The diagram shows a table after two commits.

```
flowchart LR
  subgraph CAT[Catalog]
    P["orders<br/>current: v2.metadata.json"]
  end
  subgraph META[Metadata layer]
    V1["v1.metadata.json<br/>after commit 1"]
    V2["v2.metadata.json<br/>after commit 2"]
    ML1["manifest list<br/>snapshot 1"]
    ML2["manifest list<br/>snapshot 2"]
    MA["manifest A<br/>files 1, 2 + stats"]
    MB["manifest B<br/>file 3 + stats"]
  end
  subgraph DATA[Data layer: Parquet]
    D1[file 1]
    D2[file 2]
    D3[file 3]
  end
  P --> V2
  V1 --> ML1
  V2 --> ML2
  ML1 --> MA
  ML2 --> MA
  ML2 --> MB
  MA --> D1
  MA --> D2
  MB --> D3
  style V1 stroke-dasharray: 4 4
  style ML1 stroke-dasharray: 4 4
```

Commit 1 wrote files 1 and 2, manifest A, a manifest list and v1.metadata.json. Commit 2 added file 3. It wrote only what was new, manifest B, a new manifest list and v2.metadata.json, and reused manifest A unchanged. Snapshot 2 is files 1, 2 and 3; snapshot 1 is still files 1 and 2.

A write becomes visible in one step, when the catalog swaps its pointer from v1.metadata.json to v2.metadata.json. Readers always follow a complete snapshot, so they never see a half-finished write. If two writers commit at the same time, one swap wins. The other writer must re-apply its change on top of the new state, and fails if the two changes conflict.

The dashed files belong to the previous snapshot. Its manifest list stays until snapshot 1 is expired, which is what lets you query the table as it was at an earlier point in time.

Here is v2.metadata.json for the orders table in the example below, simplified (the real file has more fields):

```
{
  "format-version": 2,
  "location": "s3://lake/shop/orders",
  "current-schema-id": 0,
  "schemas": [{
    "schema-id": 0,
    "fields": [
      { "id": 1, "name": "order_id",    "type": "string" },
      { "id": 2, "name": "customer_id", "type": "string" },
      { "id": 3, "name": "amount",      "type": "decimal(12, 2)" },
      { "id": 4, "name": "status",      "type": "string" },
      { "id": 5, "name": "ordered_at",  "type": "timestamptz" }
    ]
  }],
  "partition-specs": [{
    "spec-id": 0,
    "fields": [{ "name": "ordered_at_day", "transform": "day", "source-id": 5 }]
  }],
  "current-snapshot-id": 2,
  "snapshots": [
    { "snapshot-id": 1, "manifest-list": "s3://lake/shop/orders/metadata/snap-1.avro",
      "summary": { "operation": "append" } },
    { "snapshot-id": 2, "parent-snapshot-id": 1,
      "manifest-list": "s3://lake/shop/orders/metadata/snap-2.avro",
      "summary": { "operation": "append" } }
  ]
}
```

Three things to notice. The partition rule refers to the column by its ID (source-id 5), not its name. Both snapshots are listed, so both can be queried. And the file holds no data or file list itself; it only points to the manifest lists.

Columns are tracked by ID, not by name. Renaming or reordering a column changes only metadata, and old files are still read correctly.

Partitioning is hidden. You declare a rule such as "partition by day of ordered\_at", and Iceberg derives the partition values itself. Queries filter on ordered\_at as usual, and Iceberg skips the files that cannot match. You can change the rule later; old files keep their old layout and new files follow the new one.

Updates and deletes can be applied in two ways. Copy-on-write rewrites every data file that contains a changed row: slow writes, fast reads. Merge-on-read writes small delete files instead, and readers combine them with the data at query time: fast writes, reads that slow down until the table is compacted. The engine and table settings decide which one is used.

## Key terms

- Catalog The service that knows where each table's current metadata.json is. Every commit goes through it.
- Snapshot The state of the table after one commit.
- Metadata file (metadata.json) The root of the tree, holding the schema, partitioning and snapshot list.
- Manifest A list of files with per-column statistics, used to skip files that cannot match a query.
- Partition transform The rule that derives partition values, such as day(ordered\_at) or bucket(16, customer\_id).
- Delete file Records which rows a later commit removed. Position deletes point at row positions; equality deletes match on column values.
- Format version V2 added row-level deletes; v3 adds deletion vectors and new data types. Engines differ in which versions they support.

## Example

An orders table partitioned by day, queried through Trino:

```
CREATE TABLE shop.orders (
  order_id     varchar,
  customer_id  varchar,
  amount       decimal(12, 2),
  status       varchar,
  ordered_at   timestamp(6) with time zone
)
WITH (partitioning = ARRAY['day(ordered_at)']);

-- normal query: only one day's files are read
SELECT sum(amount) FROM shop.orders
WHERE ordered_at >= TIMESTAMP '2026-10-28 00:00:00 UTC'
  AND ordered_at <  TIMESTAMP '2026-10-29 00:00:00 UTC';

-- the table as it was yesterday morning
SELECT count(*) FROM shop.orders
FOR TIMESTAMP AS OF TIMESTAMP '2026-10-28 06:00:00 UTC';

-- its commit history
SELECT snapshot_id, committed_at, operation
FROM shop."orders$snapshots";
```

## Sample setup

One set of tables shared by several engines. Each engine asks the catalog where a table's current state is, then reads or writes the files in the bucket directly. No data is copied between systems.

```
flowchart TD
  subgraph Engines
    W1[Streaming writer<br/>appends new rows]
    W2[Trino<br/>transforms and queries]
    R1[Python notebook<br/>reads for analysis]
    R2[Cloud warehouse<br/>reads for reporting]
  end
  CAT[Catalog<br/>pointer to each table's<br/>current metadata]
  subgraph Bucket[Object storage bucket]
    T1[(orders)]
    T2[(customers)]
  end
  Engines -->|1. where is the table now?| CAT
  Engines ==>|2. read and write files directly| Bucket
```

The writers commit through the catalog, so a reader that starts a query sees either the state before a commit or after it, never in between.

## Pitfalls

- Small files and lingering deletes. Frequent commits need compaction, and deleted rows stay in storage until compaction rewrites their files and old snapshots expire; see [Iceberg maintenance](<https://wiki.4tm0.net/data/iceberg-maintenance.html>).
- Equality deletes are cheap to write and expensive to read, and not every engine handles them well. A common pattern is to append changes to a raw table and merge them later in batch.
- Changing files directly. Deleting or overwriting files outside Iceberg breaks the table; files added outside a commit are ignored and become orphans.

## Learn more

- [Iceberg table spec](<https://iceberg.apache.org/spec/>)
- [Trino Iceberg connector](<https://trino.io/docs/current/connector/iceberg.html>)
- [Iceberg maintenance](<https://iceberg.apache.org/docs/latest/maintenance/>)

## Related

[Iceberg maintenance](<https://wiki.4tm0.net/data/iceberg-maintenance.html>) [Catalogs](<https://wiki.4tm0.net/data/catalogs.html>) [Medallion layers](<https://wiki.4tm0.net/data/medallion-layers.html>) [Writing to Iceberg from Kafka](<https://wiki.4tm0.net/data/kafka-to-iceberg.html>) [SQL query engines](<https://wiki.4tm0.net/data/sql-query-engines.html>)