# 01 Foundations

The ideas every other section builds on: how analytical systems differ from operational ones, where analytical data is kept, how it is moved and processed, and how a platform is organised into layers and owned by teams.

> [!NOTE]
>
> **Before starting**
>
> nothing. This is the first section.

## Pages, in reading order

- 1 [OLTP vs OLAP](<https://wiki.4tm0.net/data/oltp-vs-olap.html>)

  OLTP (online transaction processing) systems run the business one record at a time; OLAP (online analytical processing) systems answer questions across many records at once.
- 2 [Data lake, warehouse and lakehouse](<https://wiki.4tm0.net/data/lake-warehouse-lakehouse.html>)

  Three ways to build an analytical store: a warehouse keeps data in a database that manages its own storage, a lake keeps files in cheap object storage, and a lakehouse adds table semantics to the lake so it can work like a warehouse.
- 3 [ETL, ELT and EtLT](<https://wiki.4tm0.net/data/etl-elt-etlt.html>)

  Three orderings of the same steps, extract, transform and load, that differ in where data is reshaped: before it reaches the analytical store, inside it, or lightly before loading and fully inside.
- 4 [Processing patterns: batch, micro-batch and streaming](<https://wiki.4tm0.net/data/processing-patterns.html>)

  Data can be processed in large scheduled chunks, in small frequent chunks, or continuously as each event arrives; the choice trades freshness against cost and complexity.
- 5 [Schema-on-write and schema-on-read](<https://wiki.4tm0.net/data/schema-on-write-vs-read.html>)

  Schema-on-write checks data against a fixed structure when it is stored; schema-on-read stores data as it comes and applies a structure only when it is queried.
- 6 [Medallion layers: bronze, silver and gold](<https://wiki.4tm0.net/data/medallion-layers.html>)

  A way to organise a data platform into three layers of increasing quality: bronze holds data as it arrived, silver holds it cleaned and conformed, and gold holds it shaped for specific uses.
- 7 [Data products and data mesh](<https://wiki.4tm0.net/data/data-products-and-mesh.html>)

  A data product is a dataset treated like a product, with an owner, a promise to its consumers and a clear interface; data mesh is an approach that lets the teams who know the data own and publish such products themselves.
- 8 [Data team roles](<https://wiki.4tm0.net/data/data-team-roles.html>) Optional

  The jobs people do on a data platform, such as data engineer, analytics engineer, analyst, data scientist, ML engineer, platform engineer and data product owner, what each is responsible for, and where they overlap.

## Next section

[02 Storage](<https://wiki.4tm0.net/data/section-storage.html>)