# Columnar files and Parquet Parquet is an open file format that stores a table's values column by column, encoded and compressed, with statistics that let readers skip the parts they do not need. > [!NOTE] > > **Before reading** > > [OLTP vs OLAP]() ยท [Object storage and the S3 API]() ## Why it matters Analytical queries read a few columns of many rows. In a row format such as CSV or JSON, a reader has to read every byte of every row to get those columns. Parquet is the usual data file format in a lakehouse, including under Iceberg tables, so its layout decides how much data a query actually reads. ## How it works A Parquet file is divided into row groups, horizontal slices of the table such as a million rows each. Inside a row group, the values of each column are stored together as a column chunk, and each chunk is split into pages. At the end of the file sits the footer. It holds the schema, the position of every column chunk, and statistics for each chunk: minimum, maximum and number of nulls. ``` flowchart TB subgraph F[orders-0001.parquet] direction TB subgraph RG1[Row group 1: 28 Oct, 00:00 to 11:59] direction LR A1[order_id] ~~~ E1[customer_id] ~~~ B1[status] ~~~ C1[amount] ~~~ D1[ordered_at] end subgraph RG2[Row group 2: 28 Oct, 12:00 to 23:59] direction LR A2[order_id] ~~~ E2[customer_id] ~~~ B2[status] ~~~ C2[amount] ~~~ D2[ordered_at] end FT[Footer: schema, offset of every column chunk,
min, max and null count per chunk] RG1 ~~~ RG2 ~~~ FT end ``` A reader works backwards. It fetches the footer with one range read from the end of the file. It compares the query's filter with each row group's statistics and drops the row groups that cannot match. Then it fetches only the column chunks the query uses, from the row groups that remain. Reading only the needed columns is projection; skipping row groups by statistics is predicate pushdown. Newer files can also carry a page index, with statistics per page, and Bloom filters, which answer "is this value possibly here?" for columns with many distinct values such as order\_id. A Bloom filter can give false positives but never false negatives. Storing a column's values together makes them compress well, because they are similar. Parquet first encodes them. Dictionary encoding stores each distinct value once and replaces the values with small numbers, which suits columns like status. Run-length encoding stores runs of repeated dictionary indexes as a value and a count. Delta encoding stores the differences between consecutive numbers or timestamps, which are small when values are sorted. A compression codec then runs over each page: Snappy and LZ4 are fast, ZSTD and GZIP produce smaller files. Statistics skip data only when similar values sit together, for example when the file is sorted by the filtered column; see [Query performance on lake tables](). Parquet files are written once and never changed, which fits object storage. Changing a row means writing a new file; a table format then records which files are current. Two other formats come up often. ORC is also columnar, with a similar design, and is common in the Hive ecosystem. Avro is row-based and carries its schema with the data, which makes it good for writing records one at a time: it is common for Kafka messages and is used for Iceberg's manifest files. ## Key terms - Row group A horizontal slice of a file, holding one column chunk per column. - Column chunk The values of one column within one row group. - Page The unit of encoding and compression inside a column chunk. - Footer Metadata at the end of the file: schema, chunk positions and statistics. - Projection Reading only the columns a query needs. - Predicate pushdown Using a filter to skip data before reading it. - Encoding A compact representation of values, such as dictionary or run-length. ## Example One orders file sorted by ordered\_at, inspected with PyArrow: ``` import pyarrow.parquet as pq meta = pq.ParquetFile("orders-0001.parquet").metadata names = [meta.schema.column(i).name for i in range(meta.num_columns)] col = names.index("ordered_at") for i in range(meta.num_row_groups): stats = meta.row_group(i).column(col).statistics print(i, stats.min, stats.max, stats.null_count) ``` | Row group | ordered\_at min | ordered\_at max | Nulls | | --- | --- | --- | --- | | 0 | 2026-10-28 00:00:04 | 2026-10-28 11:59:58 | 0 | | 1 | 2026-10-28 12:00:01 | 2026-10-28 23:59:59 | 0 | A query for the total amount of orders placed after 18:00 reads the footer, skips row group 0, and from row group 1 reads only the amount and ordered\_at chunks: two of five columns in one of two row groups. ## Pitfalls - Unsorted data. Statistics cannot skip anything when every row group spans every value. - Many small files. Each has its own footer to read and compresses poorly; tiny row groups have the same problem inside a file. - SELECT \*. Reading every column throws away the main advantage of the format. - Numbers and timestamps stored as text. Statistics and encodings then work on strings and help much less. - Files with different schemas in one folder. Without a table format, readers must reconcile them. ## Learn more - [Apache Parquet, file format documentation]() - [Apache Parquet format specification]() ## Related [Partitioning]() [Query performance on lake tables]() [Apache Iceberg: table format basics]() [Schema evolution]()