Page spec#

One row per page; the Page column is the page id and filename. Order is the row position: section number, dot, position with two digits (the third row of Storage is 2.03). Scope says what the page covers; Not here says what it leaves to another page (named in brackets). Before reading lists prerequisite pages. Example uses the shared example world from GUIDELINES.md. Research: S = stable concept, light check; P = check claims against spec or official docs; C = current research, products change.

Pages are listed in learning order within each section.

01 Foundations#

Page	Scope	Not here	Before reading	Example	Research
oltp-vs-olap	Operational vs analytical workloads: access patterns, row vs column orientation, why analytics needs a separate copy	Columnar file internals [columnar-files-parquet]	—	The same order stored for checkout vs counted for a monthly sales report	S
lake-warehouse-lakehouse	The three storage approaches, what each optimises for, how open table formats created the lakehouse	Table format mechanics [iceberg-table-format-basics]	oltp-vs-olap	Where shop data lives in each approach	S
etl-elt-etlt	Where transformation happens relative to loading; the light-transform-on-load variant; trade-offs (cost, privacy, flexibility)	Tools [dbt]; streaming [processing-patterns]	lake-warehouse-lakehouse	Masking card numbers before load (EtLT) vs modelling after load (ELT)	S
processing-patterns	Batch, micro-batch, streaming, real-time; latency vs cost vs complexity; how to choose by use case	Stream processing engines [stream-processing]; CDC mechanics [change-data-capture]	etl-elt-etlt	Daily sales report vs fraud check vs stock alert	S
schema-on-write-vs-read	Validate at load vs interpret at query; where each fits; how layers combine both	Schema changes over time [schema-evolution]	lake-warehouse-lakehouse	Raw clickstream JSON vs typed clean_orders	S
medallion-layers	Bronze, silver, gold: purpose, rules and consumers of each layer; what goes wrong without layers	Modelling inside gold [dimensional-modeling]; naming [naming-conventions]	schema-on-write-vs-read	raw_orders → clean_orders → fct_daily_sales	S
data-products-and-mesh	Data as a product (owner, contract, consumers); data mesh principles; central vs federated ownership	Contract mechanics [data-contracts]; roles [governance-roles]	medallion-layers	Orders team owns clean_orders and publishes fct_daily_sales	S
data-team-roles (opt)	Data engineer, analytics engineer, analyst, data scientist, ML engineer, platform engineer, data product owner; how responsibilities split and overlap	Governance decision rights [governance-roles]	data-products-and-mesh	Who builds what for the daily sales report	S

02 Storage#

Page	Scope	Not here	Before reading	Example	Research
object-storage	Buckets and keys, the S3 API as a standard, consistency, no in-place edits, cost model, S3-compatible stores	Table semantics on top [iceberg-table-format-basics]	lake-warehouse-lakehouse	Layout of shop files in a bucket	P
columnar-files-parquet	Row groups, column chunks, encoding, compression, footer statistics, why engines skip data; brief mention of ORC and Avro	Partitioning [partitioning]; tuning [query-performance]	oltp-vs-olap, object-storage	One orders Parquet file, its footer stats	P
partitioning	Splitting data by a key, partition pruning, choosing keys, over- and under-partitioning, bucketing	Iceberg hidden partitioning [iceberg-table-format-basics]	columnar-files-parquet	Orders by order date vs by customer	S
iceberg-table-format-basics	Snapshots, metadata tree, atomic commits, schema evolution by ID, hidden partitioning, copy-on-write vs merge-on-read, time travel	Maintenance [iceberg-maintenance]; catalog role in depth [catalogs]	object-storage, columnar-files-parquet, partitioning	shop.orders (draft done)	P
iceberg-maintenance	Small files, compaction, snapshot expiry, orphan file removal, delete files, ordering and frequency of jobs	Erasure policy [privacy-lifecycle]	iceberg-table-format-basics	Compacting a streamed orders table	P
iceberg-internals (opt)	Manifest lists and manifests in detail, inspecting them with metadata tables, how planning uses statistics	—	iceberg-maintenance	Inspecting shop.orders$files and $manifests	P
query-performance	Pruning, predicate and projection pushdown, file sizing, sorting and clustering, statistics, reading a query plan	Engine choice [sql-query-engines]	partitioning, iceberg-table-format-basics	Making a daily sales query read 1% of files	P
catalogs	What a catalog does, Iceberg REST protocol, authentication and credential vending, examples of implementations, backup and recovery of the catalog	Business catalog and lineage [catalog-and-lineage]	iceberg-table-format-basics	One table registered and read by two engines	C
table-format-comparison (opt)	Iceberg, Delta Lake, Hudi and Paimon: design differences, engine and catalog support, interoperability (UniForm, XTable), how to choose	Iceberg mechanics [iceberg-table-format-basics]	iceberg-table-format-basics, catalogs	One table read through Iceberg and Delta clients	C

03 Ingestion#

Page	Scope	Not here	Before reading	Example	Research
ingestion-patterns	Full vs incremental loads, watermarks and cursors, API and file ingestion, push vs pull, landing raw	Log-based CDC [change-data-capture]	processing-patterns, medallion-layers	Loading products daily (full) and orders hourly (incremental by updated_at)	S
kafka-and-connect	Topics, partitions, offsets, consumer groups, retention, compaction; Kafka Connect sources, sinks and transforms	Delivery semantics [delivery-guarantees-idempotency]	processing-patterns	Clickstream topic partitioned by session	P
change-data-capture	Log-based vs query-based CDC, the database log, snapshot then stream, change event shape, tombstones, deletes, risks (log retention, replication slots)	Writing changes to tables [kafka-to-iceberg]; merging [deduplication-and-merge]	ingestion-patterns, kafka-and-connect	An order update and delete as change events	P
delivery-guarantees-idempotency	At-most-once, at-least-once, exactly-once; where duplicates come from; idempotent writes and keys	Deduplication in models [deduplication-and-merge]	kafka-and-connect	A payment event delivered twice	P
late-and-out-of-order-data	Event time vs processing time, lateness, watermarks in streams, reprocessing windows	Stream engines [stream-processing]	delivery-guarantees-idempotency	A mobile click arriving three hours late	S
schema-evolution	Compatible and breaking changes, schema registries, compatibility modes, handling drift in raw and clean layers	Contracts [data-contracts]	schema-on-write-vs-read, kafka-and-connect	Adding discount_code to orders; renaming a column	P
event-design	What makes a good event: facts not commands, naming, envelope, IDs, timestamps, granularity, versioning, CloudEvents	Schema mechanics [schema-evolution]	kafka-and-connect, schema-evolution	OrderPlaced and OrderCancelled events	P
kafka-to-iceberg	Sink connectors writing to table formats, append vs upsert, commit intervals, routing, CDC metadata columns	Compaction [iceberg-maintenance]	change-data-capture, iceberg-table-format-basics	Orders change events appended to raw_orders	C
outbox-pattern (opt)	Publishing events reliably from an application via a table and CDC; why dual writes fail	—	change-data-capture	OrderPlaced event via an outbox table	S
iot-and-device-data (opt)	Device telemetry characteristics, MQTT basics, topic design, time-series storage concerns	Stream engines [stream-processing]	kafka-and-connect	Warehouse scanners sending stock counts	P
stream-processing (opt)	Stateful processing, windows, joins on streams, exactly-once in engines, when it is worth it	Basics of streaming [processing-patterns]	late-and-out-of-order-data	Live basket abandonment alerts	C

04 Modelling and transformation#

Page	Scope	Not here	Before reading	Example	Research
sql-query-engines	What a distributed SQL engine does, separation of storage and compute, connectors, federation, examples of engines and how they differ	Tuning [query-performance]	iceberg-table-format-basics	Joining orders in the lake with customers in Postgres	C
dimensional-modeling	Facts, dimensions, grain, star schema, conformed dimensions, wide tables as an alternative	History [history-and-scd]	medallion-layers	fct_order_line and dim_product	S
deduplication-and-merge	Finding duplicates, choosing the winning record, MERGE from a change log, ordering by sequence	Delivery semantics [delivery-guarantees-idempotency]	change-data-capture, delivery-guarantees-idempotency	clean_orders built from raw change events	S
history-and-scd	Keeping history: SCD type 1 and 2, snapshots, effective dates, point-in-time joins	—	dimensional-modeling, deduplication-and-merge	Customer address changes and past orders	S
multi-source-conformance	Combining the same entity from several sources: mapping, matching, identity resolution, survivorship, master data	—	history-and-scd	Customers from the shop and an acquired marketplace	S
dbt	Models, materializations, ref and lineage, tests, incremental models, packages, documentation	Testing strategy [testing-data-pipelines]; contracts [data-contracts]	medallion-layers, sql-query-engines	stg_orders → clean_orders → fct_daily_sales	P
testing-data-pipelines	Kinds of tests (schema, data, unit, reconciliation), where each runs, test data, what to test first	Quality monitoring in production [data-quality-dimensions]	dbt	Testing that daily sales equals the sum of orders	S
write-audit-publish	Writing to a staging branch or table, checking, then publishing atomically; branches and tags in table formats	—	testing-data-pipelines, iceberg-table-format-basics	Publishing fct_daily_sales only after checks pass	P
backfill-and-replay	Re-running history, idempotent and partition-scoped models, replaying from raw or from a log, cost control	Orchestration mechanics [orchestration]	deduplication-and-merge, dbt	Rebuilding three months after a bug fix	S
naming-conventions (opt)	Naming layers, schemas, tables, columns, keys, timestamps; names carry contracts not deployments; enforcing in CI	—	medallion-layers, dbt	The shop naming rules	S
other-modelling-approaches (opt)	Data Vault (hubs, links, satellites), one big table, activity schema; when each fits next to star schemas	—	dimensional-modeling, history-and-scd	Orders as a Data Vault and as one big table	S

05 Governance and security#

Page	Scope	Not here	Before reading	Example	Research
governance-roles	Owners, stewards, custodians, consumers, platform team; who decides what	Contract mechanics [data-contracts]	data-products-and-mesh	Who owns clean_orders	S
data-contracts	What a contract states (schema, semantics, quality, SLAs, ownership), contract standards, enforcement in CI	Schema mechanics [schema-evolution]	governance-roles, dbt	A contract for fct_daily_sales	C
data-quality-dimensions	Accuracy, completeness, consistency, timeliness, validity, uniqueness; measuring and alerting; freshness	Test types [testing-data-pipelines]	testing-data-pipelines	Freshness and completeness checks on clean_orders	S
catalog-and-lineage	Business catalog, search, glossary, ownership, column-level lineage, how metadata is collected	Technical table catalog [catalogs]	dbt, governance-roles	Tracing a dashboard number back to raw_orders	C
governance-as-code (opt)	Tags, owners, glossary and policies kept in version control and applied by CI	—	catalog-and-lineage, data-contracts	Tagging PII columns from contract files	C
authn-vs-authz-oidc-jwt	Authentication vs authorization, OIDC and OAuth2 flows, JWT structure and claims, service accounts	—	—	A token for a shop analyst and one for a pipeline	P
rbac-vs-abac	Role-based vs attribute-based access, policy evaluation, where policies live	Row filters and masks [row-level-security-masking]	authn-vs-authz-oidc-jwt	Analysts vs regional managers	S
row-level-security-masking	Row filters, column masks, where to enforce (engine, semantic layer, catalog), one place of enforcement	—	rbac-vs-abac	Managers see only their region; card numbers masked	S
multi-tenancy	Tenant isolation models (shared tables, schema per tenant, stack per tenant), tenant keys, noisy neighbours	—	row-level-security-masking	Marketplace sellers seeing only their sales	S
pii-and-classification	Identifying and tagging personal data, classification levels, pseudonymisation vs anonymisation	Retention and erasure [privacy-lifecycle]	governance-roles, data-contracts	Email, address and card data in customers	S
privacy-lifecycle	Retention, archiving, deletion requests, erasure across layers and snapshots, proving deletion	Iceberg mechanics [iceberg-maintenance]	pii-and-classification, iceberg-maintenance	Erasing one customer everywhere	S
residency-and-regions	Keeping data in a jurisdiction, regional stacks, global aggregates, routing by home region	—	pii-and-classification	EU and US shop customers	S
encryption-and-key-management	Encryption in transit and at rest, server-side and client-side, envelope encryption and KMS, Parquet and Iceberg encryption, key rotation, crypto-shredding, secrets	—	object-storage, privacy-lifecycle	Customer tables encrypted with a per-region key	P
audit-logging	What to log (access, queries, changes, admin actions), sources (engine, catalog, storage, identity provider), central tamper-resistant storage, retention, using logs	Operational monitoring [observability-and-slos]	authn-vs-authz-oidc-jwt, row-level-security-masking	Who read customer emails last month	S

06 Serving#

Page	Scope	Not here	Before reading	Example	Research
semantic-layer	What it does (metrics, dimensions and joins defined once, in business terms); three placements: inside a BI tool (e.g. Power BI semantic model, LookML), standalone/headless with its own SQL, REST or GraphQL APIs (e.g. Cube, dbt Semantic Layer), inside the warehouse or engine (e.g. semantic views, metric views); who can consume each (dashboards, apps, agents); caching; access rules in the layer; metrics API vs data API; pitfall: the same metric defined in two places	Row-level security in general [row-level-security-masking]; serving records to apps [data-api-and-bff]; agent patterns [agents-and-mcp]	dimensional-modeling	Revenue and order count defined once, used by a dashboard, an app and an agent	C
data-api-and-bff	Serving data to applications, metrics API vs data API (records for an app vs aggregates), the backend-for-frontend pattern, tokens, pagination, caching, why apps should not query the lake directly	—	semantic-layer, authn-vs-authz-oidc-jwt	A customer's order history page	S
reverse-etl-activation	Pushing platform results into operational tools, push vs pull (activation by the target), risks	—	data-api-and-bff	Sending a churn score to the CRM	C
data-sharing (opt)	Sharing tables with other teams or companies without copying, open sharing protocols, shares vs exports	—	catalogs, row-level-security-masking	Sharing a supplier's sales with that supplier	C
low-latency-serving-stores (opt)	When the lake is too slow, real-time OLAP and key-value stores, keeping them in sync	—	query-performance, processing-patterns	Live product view counts on the shop homepage	C
agents-and-mcp	AI agents querying governed data, tools over a semantic layer, MCP, acting as the user, guardrails	Embeddings and retrieval [embeddings-vector-search-rag]	semantic-layer, authn-vs-authz-oidc-jwt	An assistant answering "how did sales do last week"	C
bi-and-dashboards	BI tools, live queries and extracts, curated and self-service dashboards, designing for decisions, performance, governance of reports	Metric definitions [semantic-layer]	semantic-layer	The daily sales dashboard	C

07 Operations#

Page	Scope	Not here	Before reading	Example	Research
orchestration	Scheduling, dependencies, task vs asset-based orchestration, retries, sensors, examples of tools	Backfill strategy [backfill-and-replay]	dbt	A nightly build triggered after orders land	C
environments-and-ci	Dev, test, prod for data; CI checks, deploying models, isolated test data, promoting changes	Testing types [testing-data-pipelines]	dbt, testing-data-pipelines	A pull request changing fct_daily_sales	S
observability-and-slos	Pipeline and data observability, metrics, logs, SLOs for freshness and availability, alerting	Quality dimensions [data-quality-dimensions]	orchestration, data-quality-dimensions	An SLO: daily sales ready by 07:00	S
cost-finops (opt)	Where cost comes from (storage, compute, egress), measuring per team or product, cutting cost safely	—	query-performance	Finding the query that doubled the bill	S
disaster-recovery (opt)	Backup of data and metadata, recovery objectives, region failure, rebuilding from raw	—	catalogs, iceberg-maintenance	Restoring after an accidental table drop	S
infrastructure-as-code (opt)	Declaring buckets, catalogs, clusters, grants and jobs as code; Terraform and OpenTofu, Helm, GitOps; state, drift, environments	Governance rules as code [governance-as-code]	environments-and-ci	The shop's bucket, catalog and roles as code	C
platform-migration (opt)	Moving from a legacy warehouse to a lakehouse: inventory, parallel running, reconciliation, consumer cutover, decommissioning	—	environments-and-ci, testing-data-pipelines	Moving daily sales off a legacy warehouse	S

08 Data science and ML#

Page	Scope	Not here	Before reading	Example	Research
data-science-workflow	From a business question to an answer: framing, exploration, hypotheses, sandbox schemas, reproducibility, when analysis becomes a pipeline or a product	Python tooling [python-on-the-lakehouse]	dimensional-modeling, semantic-layer	Why repeat purchases dropped in Q3	S
python-on-the-lakehouse	Notebooks, dataframes (pandas, Polars, DuckDB, Spark), reading and writing Iceberg from Python, choosing compute, moving notebook code into pipelines	Engine internals [sql-query-engines]	data-science-workflow, sql-query-engines	Loading clean_orders into a notebook	C
experimentation-ab-testing	Randomisation units, assignment logging, metrics and guardrail metrics, sample size, peeking, sample ratio mismatch, what the platform must provide	Metric definitions [semantic-layer]	data-science-workflow	Testing a new checkout page	S
feature-engineering-feature-stores	Features, point-in-time joins, leakage, offline and online stores, training-serving skew, feature reuse	Low-latency stores in general [low-latency-serving-stores]	history-and-scd, python-on-the-lakehouse	Customer features for a churn model	C
training-data-management	Dataset versions with table snapshots and tags, splits, labelling, class balance, synthetic data, rights and consent for training use	Feature computation [feature-engineering-feature-stores]	feature-engineering-feature-stores, pii-and-classification	The training set for the demand forecast	S
ml-lifecycle-mlops	Training runs, experiment tracking, model registry, versioning and approval, batch scoring, predictions returning as data	Features [feature-engineering-feature-stores]; serving and monitoring [model-serving-monitoring]	training-data-management, data-contracts	A demand forecast per product	C
model-serving-monitoring	Batch and online serving, shadow and canary releases, rollback, drift, joining predictions with outcomes, retraining triggers	Stream engines [stream-processing]	ml-lifecycle-mlops, low-latency-serving-stores	A fraud score at checkout	C

09 Generative AI#

Page	Scope	Not here	Before reading	Example	Research
unstructured-multimodal-data	Documents, images, audio and video in the lake, metadata tables, parsing, OCR and transcription pipelines, multimodal formats such as Lance, governance of files	—	object-storage, catalog-and-lineage	Product images and PDF manuals	C
embeddings-vector-search-rag	Embeddings, vector indexes, similarity search, retrieval-augmented generation, keeping access rules on documents	Agents over metrics [agents-and-mcp]	unstructured-multimodal-data	Searching product manuals and support articles	C
llm-applications-evaluation	Hosted vs self-hosted models, prompts as versioned code, evaluation sets, model-graded evaluation, tracing, cost and latency, guardrails, LLMOps	Agents and tools [agents-and-mcp]	embeddings-vector-search-rag	Evaluating the support assistant	C
fine-tuning-vs-rag (opt)	Prompting, retrieval, fine-tuning and distillation: what each changes, the data each needs, how to choose	—	llm-applications-evaluation, training-data-management	Product descriptions in the shop's tone	C
responsible-ai-governance	AI inventory, model and data cards, bias and fairness checks, explainability, human oversight, rights to training data, EU AI Act risk classes and timeline	Privacy mechanics [privacy-lifecycle]	ml-lifecycle-mlops, pii-and-classification	Reviewing the churn model before launch	C

Index, glossary, further reading#

Page	Scope
README	What the wiki is, how to use it, four learning paths (core, data engineer, analyst or consumer, data scientist and ML)
glossary	Every key term from every page, one line each, linking to its page
further-reading	Books, specs and articles worth reading in full, grouped by section

Totals#

79 topic pages (16 optional) plus 3 reference pages. Research: S 38, P 16, C 25.