Data platform learning wiki: guidelines¶

How this wiki is written and maintained. Read this before writing or reviewing a page.

Purpose and audience¶

A set of short pages that teach the concepts behind a modern data platform, for people learning the field (architects, engineers, analysts). It explains concepts and patterns. It does not describe any company's implementation.

Scope: data platform topics only. Business domains (for example eSIM provisioning) and tooling basics (Docker, Python, shell) are out of scope.

Page template¶

Every page follows this order. Sections marked optional can be left out when they add nothing.

Writing rules¶

Content - One concept per page. If a page needs "and also…", split it or link to another page. - Explain the mechanism, not only the definition. The reader should be able to reason about a new situation. - No references to any real implementation, project, decision record or company. Use generic samples. - Every sentence teaches something. Cut anything that doesn't.

Style - Plain prose. Short sentences, active voice, concrete nouns. - No bold or italics for emphasis. Headings, lists, tables and code formatting only where they help. - Lists only for parallel items; explanations are prose. - Code formatting only for real identifiers, commands and code. Identifiers that start with an underscore, such as _ingested_at, are always in backticks, because Markdown and Confluence can read the underscore as italics. - No intros ("In today's data-driven world…"), no summaries, no "key takeaways". - No filler adjectives (robust, seamless, powerful, comprehensive, cutting-edge). - No hedging piles. State it, or mark it uncertain once. - Neutral product names: explain the concept first; products appear as examples.

Length - 500–700 words of prose. Code and diagrams come on top. Going a little over is fine when cutting would lose important information. - A page that needs more is probably two pages.

Diagrams - Mermaid only, so diagrams stay editable as text. - One diagram per idea. The text and the diagram must describe the same thing. - Render every diagram once before publishing to check it draws and reads clearly. - Side-by-side comparisons: use flowchart TB, put direction LR inside each subgraph, and fix the order with invisible links (A ~~~ B ~~~ C). Without this Mermaid may reverse or stack the subgraphs. - Edges into a subgraph that sets its own direction attach to the subgraph border, not the node. - Edges to nodes inside a subgraph often attach to the subgraph border; when the target node matters, leave the nodes outside subgraphs. - No semicolons in sequence diagram messages; Mermaid treats them as line ends.

Shared example world¶

All pages use one neutral domain, a small online shop, so readers see the same tables evolve across pages.

• Sources: an orders database (orders, order_items, customers, products), a supplier database, a payments service, a clickstream of web events. The shop operates in the EU and the US.
• Layers: raw_ (bronze), stg_ (dbt staging views), clean_ (silver), and fct_, dim_, agg_, obt_ tables (gold). Feature tables feat_, model outputs ml_*.
• Names: schema shop, columns in snake_case, timestamps ending in _at, booleans starting with is_ or has_. Code always qualifies tables, as in shop.orders.
• Load metadata on raw tables: _source (source system) and _ingested_at (when the record entered the platform; for streamed data, the Kafka record timestamp, minutes before the lake commit).
• Feeds: raw_orders comes only from CDC through Kafka and the Iceberg sink (one coordinator, four tasks, a commit every 5 minutes). clean_orders and clean_order_items (from raw_order_items, also CDC) are built hourly; fct_daily_sales nightly. Clickstream tables are raw_clickstream and clean_clickstream.
• Volume: about 250 GB of orders a year (about 700 MB a day), 200,000 customers.
• Lineage: fct_daily_sales (one row per day and product; revenue = order line amounts, tax included, cancelled orders excluded) is built from clean_orders and clean_order_items by the orders team; payments feed fct_payments. Marketing and finance read fct_daily_sales, not clean_orders.
• Promises: fct_daily_sales ready by 07:00 Europe/Berlin, SLO 98% of days over 90 days. Retention: Iceberg snapshots 7 days, Kafka topics 7 days, noncurrent bucket versions 30 days.
• Keys and personal data: customer_key is the dim_customer surrogate key; customer_pseudo_id is the keyed hash used for pseudonymous analysis. Email is visible to support, privacy and the CRM sync, masked for others. Card data is a provider token, card_token.
• Recurring examples: customer c-88 (EU, moved Munich to Lyon on 15 September 2026), order o-1001 (discounted, then cancelled), order o-1002 (placed and cancelled on 28 October 2026), payment p-77, erasure example customer c-91. All example dates are in 2026.

New pages reuse these names where possible and add new tables only when the concept needs them.

Research and fact checking¶

Research is scaled to the page type.

• Stable concept pages (OLTP vs OLAP, dimensional modelling, medallion layers): no new research. One or two authoritative sources are read to confirm the explanation and to choose the Learn more links.
• Spec or protocol pages (Iceberg, Parquet, OAuth2 and OIDC, CDC, Kafka semantics): check specific claims against the spec or the official docs.
• Product or ecosystem pages (catalogs, orchestration, semantic layer, observability tools): current research, because products, licences and features change. Name the product versions or dates the page relies on.

Rules - Anything version-specific is checked and linked, or removed. - When something cannot be verified, the page says so once, or drops the claim. - Learn more links are opened and checked, never taken from search snippets.

Structure¶

All pages sit in one flat directory. Sections are metadata in each page's frontmatter, not folders, so a page's path never changes and links never break. The page tree (for example in Confluence) is built from the section and order fields.

wiki/
├── README.md            index and learning paths
├── glossary.md
├── further-reading.md
├── GUIDELINES.md
├── PAGE-SPEC.md
├── check_wiki.py        validates frontmatter and links
├── build_sections.py    fills the page lists in section pages
├── build_reference.py   generates glossary.md and further-reading.md
├── lint_wiki.py         style lint: markup, filler words, unqualified tables, length
├── section-foundations.md   one overview page per section
├── ...
├── oltp-vs-olap.md
├── ...
└── embeddings-vector-search-rag.md

Sections¶

The pages of each section, in order, are listed in PAGE-SPEC.md.

Section pages¶

Each section has an overview page, section-

---
id: iceberg-table-format-basics
title: "Apache Iceberg: table format basics"
section: storage
order: 2.04
optional: false
prerequisites: [object-storage, columnar-files-parquet, partitioning]
related: [iceberg-maintenance, catalogs, sql-query-engines, kafka-to-iceberg]
research: P
status: draft
verified: 2026-10-29
---