# Data platform wiki > Learning wiki on data platforms, from lakehouse basics to generative AI. A learning wiki on data platforms: storage, ingestion, modelling, governance, serving, operations, data science and generative AI. Pages are short and follow one template (definition, prerequisites, how it works, key terms, example, pitfalls). Examples use a fictional online shop, not a real company. Read pages in the listed order; each page names its prerequisites. ## Start here - [Home](): Short pages that explain the concepts behind a modern data platform: how data is stored, brought in, modelled, governed, served and operated, and how it is used for data science, machine learning and generative AI. Each page explains one concept, how it works, and where it goes wrong, using one shared example: a small online shop with orders, payments, customers, products and a clickstream. ## 01 Foundations - [Overview](): The ideas every other section builds on: how analytical systems differ from operational ones, where analytical data is kept, how it is moved and processed, and how a platform is organised into layers and owned by teams. - [OLTP vs OLAP](): OLTP (online transaction processing) systems run the business one record at a time; OLAP (online analytical processing) systems answer questions across many records at once. - [Data lake, warehouse and lakehouse](): Three ways to build an analytical store: a warehouse keeps data in a database that manages its own storage, a lake keeps files in cheap object storage, and a lakehouse adds table semantics to the lake so it can work like a warehouse. - [ETL, ELT and EtLT](): Three orderings of the same steps, extract, transform and load, that differ in where data is reshaped: before it reaches the analytical store, inside it, or lightly before loading and fully inside. - [Processing patterns: batch, micro-batch and streaming](): Data can be processed in large scheduled chunks, in small frequent chunks, or continuously as each event arrives; the choice trades freshness against cost and complexity. - [Schema-on-write and schema-on-read](): Schema-on-write checks data against a fixed structure when it is stored; schema-on-read stores data as it comes and applies a structure only when it is queried. - [Medallion layers: bronze, silver and gold](): A way to organise a data platform into three layers of increasing quality: bronze holds data as it arrived, silver holds it cleaned and conformed, and gold holds it shaped for specific uses. - [Data products and data mesh](): A data product is a dataset treated like a product, with an owner, a promise to its consumers and a clear interface; data mesh is an approach that lets the teams who know the data own and publish such products themselves. - [Data team roles](): The jobs people do on a data platform, such as data engineer, analytics engineer, analyst, data scientist, ML engineer, platform engineer and data product owner, what each is responsible for, and where they overlap. ## 02 Storage - [Overview](): How a lakehouse physically stores data: object storage, columnar files and partitioning at the bottom, the Iceberg table format on top of them, the jobs that keep tables healthy, and the catalog through which every engine finds a table. - [Object storage and the S3 API](): Object storage keeps data as whole objects, each a sequence of bytes plus a little metadata, addressed by a bucket and a key and read or written over HTTP. - [Columnar files and Parquet](): Parquet is an open file format that stores a table's values column by column, encoded and compressed, with statistics that let readers skip the parts they do not need. - [Partitioning](): Partitioning splits a table's files into groups by the value of a key, such as the day, so that a query filtering on that key reads only the matching groups. - [Apache Iceberg: table format basics](): Iceberg is an open table format: a set of rules for storing a table as plain files in object storage so that many engines can read and write it safely, as if it were a database table. - [Iceberg maintenance](): Regular jobs that keep an Iceberg table fast and its storage under control: compacting small files, expiring old snapshots and removing files that nothing references. - [Iceberg internals: manifests and planning](): How Iceberg's metadata files are structured, and how an engine uses them to find the files a query needs without listing storage. - [Query performance on lake tables](): Query performance on lake tables comes mostly from reading less: fewer files, fewer row groups and fewer columns. - [Catalogs](): A technical catalog is the service that maps table names to each table's current metadata, coordinates commits, and controls who may read or write each table. - [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](): The main open table formats solve the same problem, turning files into tables, with different designs and ecosystems; this page compares them and explains how they interoperate. ## 03 Ingestion - [Overview](): How data gets from source systems into the platform's raw layer: loading from databases, APIs and files, streaming through Kafka and capturing database changes; the problems every pipeline meets, such as duplicates, late events and changing schemas; how to design the events applications publish; and, optionally, the outbox pattern, device data and stream processing. - [Ingestion patterns](): Ingestion is getting data from source systems into the platform's raw layer, either by copying everything each time (a full load) or only what changed since the last run (an incremental load). - [Kafka and Kafka Connect](): Apache Kafka is a distributed log that stores streams of events durably and lets many applications write and read them independently; Kafka Connect is its framework for moving data between Kafka and other systems without writing custom code. - [Change data capture](): Change data capture (CDC) turns every insert, update and delete in a database into an event, usually by reading the database's own transaction log, so other systems can follow its changes. - [Delivery guarantees and idempotency](): A delivery guarantee says how often a message can arrive when things fail: at most once, at least once, or exactly once. Idempotency is the property that processing the same message twice has the same effect as processing it once. - [Late and out-of-order data](): Events often reach the platform later than they happened and in a different order; pipelines must decide which time they use and how long they wait for stragglers. - [Schema evolution](): Schema evolution is changing the structure of data over time, adding, removing, renaming or retyping fields, without breaking the systems that write and read it. - [Event design](): How to design the events that applications publish, so that they are useful as data: what an event represents, how it is named, which fields every event carries, and how it changes over time. - [Writing to Iceberg from Kafka](): Moving events from Kafka topics into Iceberg tables, usually with a sink connector that buffers records, writes data files and commits them to the table at a fixed interval. - [The outbox pattern](): An application publishes events reliably by writing them to an outbox table in the same database transaction as its business change; change data capture then delivers the outbox rows to an event stream. - [IoT and device data](): Data from many small devices, such as sensors, scanners and vehicles, sent as frequent small messages over unreliable networks, usually through an MQTT broker before it reaches the platform. - [Stream processing](): Continuous computation over unbounded event streams, keeping state between events, so that results such as counts, joins and alerts are updated within seconds of each event. ## 04 Modelling and transformation - [Overview](): How raw data becomes trustworthy tables: the engines that run SQL on the lake, the shapes analytical data takes, turning change events into current state and history, combining sources, and building, testing, publishing and rebuilding transformations safely, and the conventions and alternative models around them. - [SQL query engines](): A SQL query engine runs SQL over data it does not own: it reads tables from object storage and other systems through connectors, spreads the work across many machines, and returns the result. - [Dimensional modelling](): A way to shape data for analysis as facts, the measurable events of the business, surrounded by dimensions, the context used to filter and group them. - [Deduplication and merge](): Turning raw records, with their duplicates and change events, into one correct row per entity: choosing the winning record for each key and applying it to the clean table with MERGE. - [History and slowly changing dimensions](): Ways to keep the past values of data that changes over time, such as a customer's address, so that results can reflect the world as it was when an event happened. - [Multi-source conformance](): Combining records of the same kind of thing from several sources into one consistent view: mapping their fields and codes, matching records that describe the same real-world entity, and deciding which values win. - [dbt](): dbt is a tool for building transformations as SQL SELECT statements in version-controlled files; it works out their order, runs them on a query engine, and tests and documents the results. - [Testing data pipelines](): Checking that transformation code is correct and that the data it produces meets expectations, using several kinds of test at different points between a code change and a published table. - [Write-audit-publish](): A pattern in which new data is written where consumers cannot see it, checked, and only then made visible in one atomic step. - [Backfill and replay](): Re-running a pipeline for past periods, to fill a new table with history, to correct data after a bug, or to apply changed logic, without disturbing the current runs or the people reading the results. - [Naming conventions](): Agreed rules for naming schemas, tables, columns and keys, so that a name alone says what a dataset is, which layer it belongs to and how it can be used. - [Other modelling approaches: Data Vault, one big table and activity schema](): Alternatives and complements to star schemas: Data Vault for integrating and historising many sources, one big table for simple consumption, and activity schema for modelling behaviour as a single stream of events. ## 05 Governance and security - [Overview](): Who is responsible for data and how promises about it are kept, how people find and trust it, and how access is controlled: identities and tokens, roles and attributes, row and column restrictions, tenants, personal data, retention and erasure, where data may live, how it is encrypted and how access is audited. - [Governance roles](): The people responsible for data and what each of them decides: owners, stewards, custodians, consumers and the platform team. - [Data contracts](): A data contract is a written, machine-readable agreement between the producer of a dataset and its consumers about its structure, meaning, quality and service levels, checked automatically. - [Data quality dimensions](): The aspects in which data can be good or bad, such as accuracy, completeness and timeliness, and how each is measured and watched in production. - [Catalog and lineage](): A business data catalog is where people find, understand and trust data: it lists datasets with their descriptions, owners, terms and quality, and shows lineage, the path data takes from sources to reports. - [Governance as code](): Keeping governance decisions, such as owners, classification tags, glossary links, access grants and policies, in version-controlled files, and having automation apply and check them, instead of setting them by hand in each tool. - [Authentication and authorization: OIDC and JWT](): Authentication establishes who is making a request; authorization decides what they may do. In modern platforms, identity comes from a central identity provider through OpenID Connect and OAuth 2.0, carried to each service as a signed token, often a JWT. - [RBAC and ABAC](): Two ways to decide what an authenticated user may do: role-based access control (RBAC) grants permissions to roles and puts users in roles; attribute-based access control (ABAC) evaluates rules over attributes of the user, the data and the request. - [Row-level security and masking](): Row-level security limits which rows of a table a user can see; column masking changes what they see in sensitive columns, for example showing only the last four digits of a card number. - [Multi-tenancy](): Serving several customers or organisations, called tenants, from one platform while keeping each tenant's data separate and preventing one tenant's load from affecting the others. - [PII and data classification](): Identifying which data relates to people or is otherwise sensitive, labelling it with a classification level, and handling it according to that level, including pseudonymising or anonymising it where possible. - [Privacy lifecycle](): Managing personal data from collection to deletion: keeping it only as long as it is needed, archiving or deleting it on schedule, and erasing one person's data everywhere when they ask. - [Residency and regions](): Keeping data in the geographic locations that law, contracts or customers require, while still allowing analysis that spans regions. - [Encryption and key management](): Protecting data by making it unreadable without a key, both while it moves between systems and while it is stored, and managing those keys so that the right systems can use them and nobody else can. - [Audit logging](): Recording who did what to which data, and when: logins, queries, access changes and administrative actions, kept so that they can be searched later and cannot be quietly altered. ## 06 Serving - [Overview](): How data leaves the platform to be used: metrics defined once for every tool, dashboards and reports in BI tools, APIs for applications, results pushed into operational tools, tables shared with partners, fast stores for user-facing features, and AI agents that query data on people's behalf. - [Semantic layer](): A semantic layer defines business metrics, such as revenue or order count, together with the dimensions they can be grouped by and the joins between tables, once and in business terms, so that every tool that asks for a metric gets the same answer. - [Data APIs and the backend-for-frontend pattern](): A data API serves platform data to applications through a defined HTTP interface instead of direct database access; a backend-for-frontend (BFF) is an API built for one specific application's screens. - [Reverse ETL and activation](): Sending results computed in the data platform, such as customer segments or scores, back into the operational tools where people and systems act on them: CRM, marketing, support and advertising platforms. - [Data sharing](): Giving another team or company read access to tables in place, through a sharing protocol or a catalog, instead of sending them exported copies. - [Low-latency serving stores](): Databases placed next to the lakehouse to answer queries in milliseconds for many concurrent users, kept in sync with the platform: real-time analytical databases for fast aggregates, and key-value stores for fast lookups. - [Agents and MCP](): AI agents are language-model applications that answer questions and carry out tasks by calling tools; the Model Context Protocol (MCP) is an open standard for connecting agents to tools and data, including a data platform's catalog, semantic layer and query engine. - [BI and dashboards](): Business intelligence (BI) tools let people explore data and build reports and dashboards; doing this well means choosing how the tool reaches the data, deciding who builds what, and designing dashboards for the decisions they support. ## 07 Operations - [Overview](): How a platform is run day to day: scheduling and coordinating pipelines, moving changes safely from development to production, knowing when something is wrong and what "on time" means, keeping costs under control, recovering when things go badly wrong, declaring infrastructure as code, and moving from one platform to another. - [Orchestration](): Orchestration decides when each step of the platform's pipelines runs, in what order, and what happens when a step fails. - [Environments and CI](): Separate environments, such as development, test and production, let changes to pipelines be built and checked before they affect real consumers; continuous integration (CI) runs those checks automatically on every proposed change. - [Observability and SLOs](): Observability is being able to tell, from the signals a system emits, whether it is healthy and why it is not; service level objectives (SLOs) state in numbers what healthy means for consumers. - [Cost and FinOps](): Understanding where a data platform's money goes, attributing it to the teams and products that cause it, and reducing it safely, as a shared routine of engineering and finance called FinOps. - [Disaster recovery](): Being able to restore the platform's data and service after a serious failure, such as a deleted table, corrupted data, a lost catalog or an unavailable region, within agreed limits on lost data and downtime. - [Infrastructure as code](): Declaring the platform's infrastructure, such as buckets, catalogs, clusters, roles and permissions, in version-controlled files, and letting tools create and update it to match, instead of configuring it by hand. - [Platform migration](): Moving data, pipelines and consumers from one platform to another, such as from a legacy warehouse to a lakehouse, without breaking what people depend on. ## 08 Data science and ML - [Overview](): How data becomes analysis, experiments and models: the data science workflow, working with lake data from Python, controlled experiments, features and training data, and taking models to production and keeping them accurate. - [The data science workflow](): How a business question becomes a trustworthy answer: framing the question, exploring the data, testing explanations, and deciding whether the result is a one-off analysis, a recurring report or a model. - [Python on the lakehouse](): Working with lakehouse tables from Python: notebooks for exploration, dataframe libraries for analysis and feature work, and the choices that decide where the computation runs. - [Experimentation and A/B testing](): Measuring the effect of a change by randomly splitting users into groups that see different versions, and comparing their outcomes. - [Feature engineering and feature stores](): Feature engineering turns raw data into the inputs a model learns from; a feature store keeps those definitions in one place and provides the same values for training and for prediction. - [Training data management](): Building, versioning and governing the datasets models are trained and evaluated on: what goes in, how it is split, how labels are made, and whether the data may be used for training at all. - [The ML lifecycle and MLOps](): The path from a training dataset to a model whose predictions people and systems use, and MLOps, the practices that make that path repeatable, traceable and safe to change. - [Model serving and monitoring](): Running a trained model to produce predictions, in batches or per request, releasing new versions safely, and watching whether its predictions stay accurate as the world changes. ## 09 Generative AI - [Overview](): How the platform supports generative AI: managing documents, images and audio as data, making them searchable by meaning, building and evaluating applications on language models, choosing between retrieval and fine-tuning, and governing AI systems responsibly. - [Unstructured and multimodal data](): Data that is not rows and columns, such as documents, images, audio and video, stored in the lake alongside tables and made usable by extracting text, structure and metadata from it. - [Embeddings, vector search and RAG](): Embeddings turn text, images or other content into lists of numbers that place similar meanings close together; vector search finds the closest ones quickly; retrieval-augmented generation (RAG) uses those results to give a language model the right documents to answer from. - [LLM applications and evaluation](): Building applications on large language models, and measuring whether they work: choosing models, managing prompts, evaluating answers systematically, and tracing and monitoring them in production. The practices together are often called LLMOps. - [Fine-tuning or RAG](): The ways to adapt a language model to a task, prompting, retrieval, fine-tuning and distillation, what each changes, what data each needs, and how to choose between them. - [Responsible AI and AI governance](): The practices and rules that make machine learning and AI systems accountable: knowing which systems exist, documenting how they were built, checking them for unfair or unsafe behaviour, keeping people in control, and meeting legal obligations such as the EU AI Act. ## Reference - [Glossary](): Every key term from the topic pages, one line each, with a link to the page that explains it. Generated by build_reference.py. - [Further reading](): The books, specifications and articles cited on the topic pages, grouped by section. Generated by build_reference.py from each page's Learn more list.