Python on the lakehouse¶

Working with lakehouse tables from Python: notebooks for exploration, dataframe libraries for analysis and feature work, and the choices that decide where the computation runs.

Before reading

The data science workflow · SQL query engines

Why it matters¶

Data scientists and ML engineers work mostly in Python. Because lakehouse tables are open files with an open catalog, Python tools can read them directly, without exporting copies from a warehouse. Choosing the right library and compute for the size of the data decides whether an analysis takes seconds or hours, and whether it can later run as a pipeline.

How it works¶

A notebook, such as Jupyter, mixes code, results and notes in one document. It is the usual place for exploration. Notebooks run on a laptop, on a shared notebook server, or on a managed platform close to the data.

Dataframe libraries hold tables in memory and transform them. They differ mainly in how much data they handle and where they run.

pandas is the long-standing standard. It loads everything into memory on one machine and is simple and widely known, but slow and memory-hungry on large data.
Polars is a newer single-machine library, written in Rust, columnar and multi-threaded. Its lazy API plans a whole query before running it, so it reads only the needed columns and rows. It needs much less memory than pandas for the same query, because the lazy API skips unneeded columns and rows.
DuckDB is an in-process SQL engine. It runs SQL on files and Iceberg tables from Python, returns dataframes, and works out of core, spilling to disk when data does not fit in memory.
Spark, through PySpark, runs dataframe operations across a cluster. It is the choice when data does not fit one large machine, or when the work is already part of Spark pipelines.

Many workloads need far less than a cluster. A single machine with enough memory and Polars or DuckDB covers tens or hundreds of gigabytes, especially after pruning to the columns and partitions needed.

All of these read Iceberg tables through the catalog. PyIceberg, the Python implementation of Iceberg, connects to a REST catalog, plans a scan with filters and column selection, and returns Arrow data, which Polars and DuckDB read without copying; converting to pandas usually copies. It can also write: append, overwrite, and upsert by key. DuckDB's Iceberg extension can attach a REST catalog and read and write tables with SQL; write support is newer and narrower than read support: as of DuckDB 2.0 (October 2026), updates, deletes and merges are merge-on-read only, and fail on tables set to copy-on-write. Polars can scan Iceberg tables lazily and write to them with sink_iceberg. Access goes through the same catalog permissions and credential vending as any other engine, so row and column rules still apply.

flowchart LR
  NB[Notebook] --> PY[PyIceberg:<br/>plan scan with filters]
  PY -->|metadata and<br/>vended credentials| CAT[Catalog]
  PY -->|read only needed<br/>files and columns| S[(Object storage)]
  PY --> AR[Arrow table]
  AR --> PD[pandas]
  AR --> PL[Polars]
  AR --> DD[DuckDB SQL]

Moving work out of the notebook is part of the workflow. Exploration code becomes production code when it is moved into a module or a dbt Python model, tested, given parameters instead of hard-coded dates, and run by the orchestrator. Notebooks can be scheduled directly, but that keeps untested code and hidden state in production. A useful rule: notebooks call functions from tested modules, rather than containing the only copy of the logic.

Environments matter as in any software: pinned library versions, so that results do not change when a package updates, and the same versions in development and production.

Key terms¶

Notebook: an interactive document of code, results and notes.
Dataframe: an in-memory table manipulated with code.
Lazy evaluation: building a query plan first and running it once, so only needed data is read.
Out-of-core processing: handling data larger than memory by spilling to disk.
Apache Arrow: a columnar in-memory format shared by many Python data tools.
PyIceberg: the Python library for reading and writing Iceberg tables.

Example¶

Loading last quarter's orders into a notebook:

from pyiceberg.catalog import load_catalog
import polars as pl

catalog = load_catalog("lake", uri="https://catalog.example.com/api/catalog",
                       token=token, warehouse="lake")
orders = catalog.load_table("shop.clean_orders")

arrow = orders.scan(
    row_filter="ordered_at >= '2026-07-01T00:00:00+00:00' and ordered_at < '2026-10-01T00:00:00+00:00'",
    selected_fields=("order_id", "customer_id", "amount", "ordered_at"),
).to_arrow()

df = pl.from_arrow(arrow)
multi_order_share = (df.group_by("customer_id").len()
            .filter(pl.col("len") > 1).height / df["customer_id"].n_unique())

The scan reads only the July to September partitions and four columns; multi_order_share is the share of customers with more than one order in the quarter. The same filter could be pushed down with DuckDB, attaching the catalog and querying with SQL. Once the logic is settled, it moves into a tested function used by both the notebook and the pipeline.

Pitfalls¶

pandas for everything. Loading whole tables into one process is slow and fails on large data.
A cluster by default. Many analyses run faster and cheaper on one machine with Polars or DuckDB.
Exported copies. CSV extracts of lake tables go stale and escape access rules; read through the catalog.
Hidden notebook state. Cells run out of order give results nobody can reproduce; restart and run all before sharing.
Unpinned libraries. Results change silently when a package updates.

Learn more¶

PyIceberg documentation: https://py.iceberg.apache.org/
DuckDB documentation, Iceberg extension: https://duckdb.org/docs/stable/core_extensions/iceberg/overview
Polars user guide: https://docs.pola.rs/

Catalogs · Feature engineering and feature stores · Environments and CI · Query performance on lake tables