Data platform learning wiki: guidelines#

How this wiki is written and maintained. Read this before writing or reviewing a page.

Purpose and audience#

A set of short pages that teach the concepts behind a modern data platform, for people learning the field (architects, engineers, analysts). It explains concepts and patterns. It does not describe any company's implementation.

Scope: data platform topics only. Business domains (for example eSIM provisioning) and tooling basics (Docker, Python, shell) are out of scope.

Page template#

Every page follows this order. Sections marked optional can be left out when they add nothing.

Writing rules#

Content - One concept per page. If a page needs "and also…", split it or link to another page. - Explain the mechanism, not only the definition. The reader should be able to reason about a new situation. - No references to any real implementation, project, decision record or company. Use generic samples. - Every sentence teaches something. Cut anything that doesn't.

Style - Plain prose. Short sentences, active voice, concrete nouns. - No bold or italics for emphasis. Headings, lists, tables and code formatting only where they help. - Lists only for parallel items; explanations are prose. - Code formatting only for real identifiers, commands and code. Identifiers that start with an underscore, such as _ingested_at, are always in backticks, because Markdown and Confluence can read the underscore as italics. - No intros ("In today's data-driven world…"), no summaries, no "key takeaways". - No filler adjectives (robust, seamless, powerful, comprehensive, cutting-edge). - No hedging piles. State it, or mark it uncertain once. - Neutral product names: explain the concept first; products appear as examples.

Length - 500–700 words of prose. Code and diagrams come on top. Going a little over is fine when cutting would lose important information. - A page that needs more is probably two pages.

Diagrams - Mermaid only, so diagrams stay editable as text. - One diagram per idea. The text and the diagram must describe the same thing. - Render every diagram once before publishing to check it draws and reads clearly. - Side-by-side comparisons: use flowchart TB, put direction LR inside each subgraph, and fix the order with invisible links (A ~~~ B ~~~ C). Without this Mermaid may reverse or stack the subgraphs. - Edges into a subgraph that sets its own direction attach to the subgraph border, not the node. - Edges to nodes inside a subgraph often attach to the subgraph border; when the target node matters, leave the nodes outside subgraphs. - No semicolons in sequence diagram messages; Mermaid treats them as line ends.

Shared example world#

All pages use one neutral domain, a small online shop, so readers see the same tables evolve across pages.

• Sources: an orders database (orders, order_items, customers, products), a supplier database, a payments service, a clickstream of web events. The shop operates in the EU and the US.
• Layers: raw_ (bronze), stg_ (dbt staging views), clean_ (silver), and fct_, dim_, agg_, obt_ tables (gold). Feature tables feat_, model outputs ml_*.
• Names: schema shop, columns in snake_case, timestamps ending in _at, booleans starting with is_ or has_. Code always qualifies tables, as in shop.orders.
• Load metadata on raw tables: _source (source system) and _ingested_at (when the record entered the platform; for streamed data, the Kafka record timestamp, minutes before the lake commit).
• Feeds: raw_orders comes only from CDC through Kafka and the Iceberg sink (one coordinator, four tasks, a commit every 5 minutes). clean_orders and clean_order_items (from raw_order_items, also CDC) are built hourly; fct_daily_sales nightly. Clickstream tables are raw_clickstream and clean_clickstream.
• Volume: about 250 GB of orders a year (about 700 MB a day), 200,000 customers.
• Lineage: fct_daily_sales (one row per day and product; revenue = order line amounts, tax included, cancelled orders excluded) is built from clean_orders and clean_order_items by the orders team; payments feed fct_payments. Marketing and finance read fct_daily_sales, not clean_orders.
• Promises: fct_daily_sales ready by 07:00 Europe/Berlin, SLO 98% of days over 90 days. Retention: Iceberg snapshots 7 days, Kafka topics 7 days, noncurrent bucket versions 30 days.
• Keys and personal data: customer_key is the dim_customer surrogate key; customer_pseudo_id is the keyed hash used for pseudonymous analysis. Email is visible to support, privacy and the CRM sync, masked for others. Card data is a provider token, card_token.
• Recurring examples: customer c-88 (EU, moved Munich to Lyon on 15 September 2026), order o-1001 (discounted, then cancelled), order o-1002 (placed and cancelled on 28 October 2026), payment p-77, erasure example customer c-91. All example dates are in 2026.

New pages reuse these names where possible and add new tables only when the concept needs them.

Research and fact checking#

Research is scaled to the page type.

• Stable concept pages (OLTP vs OLAP, dimensional modelling, medallion layers): no new research. One or two authoritative sources are read to confirm the explanation and to choose the Learn more links.
• Spec or protocol pages (Iceberg, Parquet, OAuth2 and OIDC, CDC, Kafka semantics): check specific claims against the spec or the official docs.
• Product or ecosystem pages (catalogs, orchestration, semantic layer, observability tools): current research, because products, licences and features change. Name the product versions or dates the page relies on.

Rules - Anything version-specific is checked and linked, or removed. - When something cannot be verified, the page says so once, or drops the claim. - Learn more links are opened and checked, never taken from search snippets.

Structure#

All pages sit in one flat directory. Sections are metadata in each page's frontmatter, not folders, so a page's path never changes and links never break. The page tree (for example in Confluence) is built from the section and order fields.

wiki/
├── README.md            index and learning paths
├── glossary.md
├── further-reading.md
├── GUIDELINES.md
├── PAGE-SPEC.md
├── check_wiki.py        validates frontmatter and links
├── build_sections.py    fills the page lists in section pages
├── build_reference.py   generates glossary.md and further-reading.md
├── lint_wiki.py         style lint: markup, filler words, unqualified tables, length
├── section-foundations.md   one overview page per section
├── ...
├── oltp-vs-olap.md
├── ...
└── embeddings-vector-search-rag.md

Sections#

The pages of each section, in order, are listed in PAGE-SPEC.md.

Section pages#

Each section has an overview page, section-

---
id: iceberg-table-format-basics
title: "Apache Iceberg: table format basics"
section: storage
order: 2.04
optional: false
prerequisites: [object-storage, columnar-files-parquet, partitioning]
related: [iceberg-maintenance, catalogs, sql-query-engines, kafka-to-iceberg]
research: P
status: draft
verified: 2026-10-29
---