# Iceberg internals: manifests and planning Optional How Iceberg's metadata files are structured, and how an engine uses them to find the files a query needs without listing storage. > [!NOTE] > > **Before reading** > > [Iceberg maintenance]() ## Why it matters When planning is slow, or a query reads far more files than expected, the explanation is in the metadata. Iceberg exposes its metadata as tables that can be queried with SQL, so these questions can be answered directly instead of guessed. ## How it works Planning a query is a series of filters, each working on a smaller set. The engine reads the current metadata.json, finds the current snapshot and opens its manifest list. The manifest list has one entry per manifest, and each entry carries a partition summary: for every partition field, the lowest and highest value of the files in that manifest. The engine drops every manifest whose range cannot match the query. Each remaining manifest has one entry per data file. An entry records the file's status (added, existing or deleted in that snapshot), its path, its partition value, its row count and size, and per-column metrics: value counts, null counts, and lower and upper bounds. The engine drops every file whose partition or bounds cannot match. Only then does the engine open data files, where Parquet row group statistics can skip more. ``` flowchart TB Q["Query: orders on 28 Oct
with amount > 500"] --> M[metadata.json:
current snapshot] M --> L[Manifest list:
partition range per manifest] L -->|drop manifests whose
day range excludes 28 Oct| MF[Manifests:
bounds per file and column] MF -->|drop files whose amount
upper bound is 500 or less| F[Parquet files:
row group statistics] F -->|drop row groups| R[Rows read] ``` In format versions 1 to 3, manifests and manifest lists are Avro files. A manifest written by one commit is reused by later commits as long as its files are still current, which is why a commit writes little metadata. It is also why many small commits produce many small manifests that each cover a wide range of partitions, and why rewriting manifests, grouped by partition, speeds up planning. Delete files are tracked in separate delete manifests. Each commit has a sequence number. An equality delete applies only to data files with a lower sequence number; a position delete also applies to files written in the same commit. Either way, a delete never removes rows added after it. Column metrics are configurable. By default Iceberg keeps metrics for the first 100 columns only, top-level fields first, and truncates string bounds to 16 characters. On a wide table, a column beyond the first hundred has no bounds, and filters on it cannot skip files. The write.metadata.metrics.\* table properties change this per column. Optional statistics files, in the Puffin format, hold extra statistics such as estimates of distinct values per column. Cost-based optimizers use them to choose join order. Format version 3 also stores deletion vectors in Puffin files. ## Key terms - Manifest list The per-snapshot list of manifests, with a partition summary for each. - Manifest An Avro file listing data or delete files with their partition and column metrics. - Partition summary The lowest and highest partition values in one manifest. - Column bounds The lowest and highest value of a column in one data file. - Sequence number The order of commits, used to decide which deletes apply to which files. - Metadata table A read-only table exposing metadata, such as $files or $manifests. - Puffin file A file of extra table statistics, such as distinct value estimates, or deletion vectors. ## Example Inspecting shop.orders in Trino. The manifests of the current snapshot: ``` SELECT path, added_data_files_count, existing_data_files_count, partition_summaries FROM shop."orders$manifests"; ``` If one manifest's partition summary spans every day of the year, filtering by day cannot skip it, and rewriting manifests will help. The files of one day, with their column bounds: ``` SELECT file_path, record_count, file_size_in_bytes, readable_metrics FROM shop."orders$files" WHERE partition.ordered_at_day = DATE '2026-10-28'; ``` readable\_metrics shows, for each column, its lower and upper bound in the file. If every file's amount bounds run from near zero to the maximum, a filter on amount cannot skip any of them, and sorting during compaction would help. ## Pitfalls - Manifests not grouped by partition. Planning reads all of them even for a one-day query. - Filters on wide tables. Columns beyond the default metrics limit have no bounds and cannot be used to skip files. - Long string keys. Truncated bounds are less precise, so fewer files are skipped. - Querying the $all\_ metadata tables on large tables. They cover every snapshot and can themselves be slow. - Editing metadata files by hand. Always change a table through an engine or the catalog. ## Learn more - [Iceberg table spec, manifests and scan planning]() - [Apache Iceberg, table configuration (metrics)]() - [Trino Iceberg connector, metadata tables]() ## Related [Query performance on lake tables]() [Apache Iceberg: table format basics]() [Columnar files and Parquet]()