01 Foundations#
The ideas every other section builds on: how analytical systems differ from operational ones, where analytical data is kept, how it is moved and processed, and how a platform is organised into layers and owned by teams.
Before starting
nothing. This is the first section.
Pages, in reading order#
-
OLTP (online transaction processing) systems run the business one record at a time; OLAP (online analytical processing) systems answer questions across many records at once.
-
2 Data lake, warehouse and lakehouse
Three ways to build an analytical store: a warehouse keeps data in a database that manages its own storage, a lake keeps files in cheap object storage, and a lakehouse adds table semantics to the lake so it can work like a warehouse.
-
Three orderings of the same steps, extract, transform and load, that differ in where data is reshaped: before it reaches the analytical store, inside it, or lightly before loading and fully inside.
-
4 Processing patterns: batch, micro-batch and streaming
Data can be processed in large scheduled chunks, in small frequent chunks, or continuously as each event arrives; the choice trades freshness against cost and complexity.
-
5 Schema-on-write and schema-on-read
Schema-on-write checks data against a fixed structure when it is stored; schema-on-read stores data as it comes and applies a structure only when it is queried.
-
6 Medallion layers: bronze, silver and gold
A way to organise a data platform into three layers of increasing quality: bronze holds data as it arrived, silver holds it cleaned and conformed, and gold holds it shaped for specific uses.
-
A data product is a dataset treated like a product, with an owner, a promise to its consumers and a clear interface; data mesh is an approach that lets the teams who know the data own and publish such products themselves.
-
8 Data team roles Optional
The jobs people do on a data platform, such as data engineer, analytics engineer, analyst, data scientist, ML engineer, platform engineer and data product owner, what each is responsible for, and where they overlap.