Skip to content

Data platform learning wiki

Short pages that explain the concepts behind a modern data platform: how data is stored, brought in, modelled, governed, served and operated, and how it is used for data science, machine learning and generative AI. Each page explains one concept, how it works, and where it goes wrong, using one shared example: a small online shop with orders, payments, customers, products and a clickstream.

The pages describe concepts and common patterns, not any particular company's implementation. Products are named as examples; facts about products and standards were checked in October 2026 and are marked where they may change.

How to use it

Every page has the same structure: a one-line definition, the pages to read first, why the concept matters, how it works with one diagram, key terms, an example from the shop, pitfalls, and links to authoritative sources. Most pages take five to ten minutes to read.

Pages build on each other. The "Before reading" line at the top of each page lists what it assumes; the section pages list their pages in reading order. Pages marked optional go deeper or cover narrower needs and can be skipped on a first pass.

The glossary defines every key term in one line with a link to its page, and further reading collects the books, specifications and articles cited across the wiki.

Sections

  1. 01 Foundations: analytical vs operational systems, lakes and lakehouses, ETL and ELT, batch and streaming, layers, data products.
  2. 02 Storage: object storage, Parquet, partitioning, Iceberg, maintenance, performance, catalogs.
  3. 03 Ingestion: loads, Kafka, change data capture, duplicates, late data, schema changes, stream processing.
  4. 04 Modelling and transformation: query engines, dimensional models, merging, history, dbt, testing, publishing, backfills.
  5. 05 Governance and security: roles, contracts, quality, catalogs and lineage, identity and access, personal data, residency.
  6. 06 Serving: semantic layer, data APIs, reverse ETL, sharing, serving stores, AI agents, BI and dashboards.
  7. 07 Operations: orchestration, environments and CI, observability, cost, disaster recovery.
  8. 08 Data science and ML: the data science workflow, Python, experiments, features, training data, MLOps, serving and monitoring.
  9. 09 Generative AI: unstructured data, embeddings and retrieval, LLM applications and evaluation, fine-tuning, responsible AI.

Learning paths

Core path, for anyone joining a data platform team. 27 pages:

  1. Foundations, all pages except the optional Data team roles.
  2. Object storage and the S3 API, Columnar files and Parquet, Partitioning, Apache Iceberg: table format basics, Catalogs, SQL query engines.
  3. Ingestion patterns, Kafka and Kafka Connect, Change data capture, Delivery guarantees and idempotency.
  4. Dimensional modelling, Deduplication and merge, dbt, Testing data pipelines.
  5. Governance roles, Data contracts, Catalog and lineage, PII and data classification.
  6. Semantic layer, Orchestration.

Data engineer path, after the core path: the rest of Storage, Ingestion and Modelling in order, then Authentication and authorization: OIDC and JWT, RBAC and ABAC, Row-level security and masking, Privacy lifecycle, Encryption and key management, Audit logging, Data quality dimensions, the rest of Operations, and Unstructured and multimodal data.

Analyst and consumer path, for people who use the platform's data rather than build it. It skips mechanics on purpose: when a page's Before reading list names a page that is not on this path, that page's definition and Why it matters section are enough.

  1. OLTP vs OLAP, Medallion layers, Data products and data mesh.
  2. Dimensional modelling, History and slowly changing dimensions.
  3. Data contracts, Data quality dimensions, Catalog and lineage, Row-level security and masking.
  4. Semantic layer, BI and dashboards, The data science workflow, Experimentation and A/B testing.
  5. Agents and MCP, Unstructured and multimodal data, Embeddings, vector search and RAG.

Data scientist and ML path, after the core path: History and slowly changing dimensions, Query performance on lake tables and Low-latency serving stores, then all of 08 Data science and ML in order, then 09 Generative AI in order (Fine-tuning or RAG is optional).

For contributors

GUIDELINES.md describes the page template, writing rules, the shared example and the review process. PAGE-SPEC.md lists every page with its scope. Three scripts keep the wiki consistent: check_wiki.py validates frontmatter and links, build_sections.py fills the page lists on section pages, and build_reference.py generates the glossary and further reading.