02 Storage#

1 Object storage and the S3 API

Object storage keeps data as whole objects, each a sequence of bytes plus a little metadata, addressed by a bucket and a key and read or written over HTTP.

2 Columnar files and Parquet

Parquet is an open file format that stores a table's values column by column, encoded and compressed, with statistics that let readers skip the parts they do not need.

3 Partitioning

Partitioning splits a table's files into groups by the value of a key, such as the day, so that a query filtering on that key reads only the matching groups.

4 Apache Iceberg: table format basics

Iceberg is an open table format: a set of rules for storing a table as plain files in object storage so that many engines can read and write it safely, as if it were a database table.

5 Iceberg maintenance

Regular jobs that keep an Iceberg table fast and its storage under control: compacting small files, expiring old snapshots and removing files that nothing references.

6 Iceberg internals: manifests and planning Optional

How Iceberg's metadata files are structured, and how an engine uses them to find the files a query needs without listing storage.

7 Query performance on lake tables

Query performance on lake tables comes mostly from reading less: fewer files, fewer row groups and fewer columns.

8 Catalogs

A technical catalog is the service that maps table names to each table's current metadata, coordinates commits, and controls who may read or write each table.

9 Table formats compared: Iceberg, Delta Lake, Hudi and Paimon Optional

The main open table formats solve the same problem, turning files into tables, with different designs and ecosystems; this page compares them and explains how they interoperate.