Glossary#

Every key term from the topic pages, one line each, with a link to the page that explains it. Generated by build_reference.py.

A#

• ABAC: permissions decided by rules over user, resource and context attributes. See RBAC and ABAC.
• Access review: a periodic check of who has access against who needs it. See Audit logging.
• Accuracy: agreement with reality or a trusted source. See Data quality dimensions.
• Acting as the user: calling downstream systems with the user's identity, not a service's. See Agents and MCP.
• Activation: making data usable where actions happen. See Reverse ETL and activation.
• Activity schema: a model with one table of entity activities over time. See Other modelling approaches: Data Vault, one big table and activity schema.
• Adapter: the plugin that connects dbt to an engine. See dbt.
• Additive, semi-additive, non-additive: how a measure can be summed. See Dimensional modelling.
• Aggregate: the business object an event is about, such as one order. See The outbox pattern.
• AI agent: a model-driven application that calls tools to answer or act. See Agents and MCP.
• AI inventory: a register of all AI systems with owners, purposes and risk levels. See Responsible AI and AI governance.
• Allowed lateness: how long a window still accepts updates after its watermark has passed. See Late and out-of-order data.
• Analytics engineer: builds tested models and metrics for consumers. See Data team roles.
• Anomaly detection: flagging values that differ from their usual pattern. See Data quality dimensions.
• Anonymisation: making identification no longer possible. See PII and data classification.
• Apache Arrow: a columnar in-memory format shared by many Python data tools. See Python on the lakehouse.
• Apache XTable: a project that translates metadata between table formats. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• API contract: the stable definition of endpoints and fields that clients rely on. See Data APIs and the backend-for-frontend pattern.
• Append mode: every record becomes a new row. See Writing to Iceberg from Kafka.
• Approximate nearest neighbour (ANN): fast search that finds close vectors with high, not perfect, accuracy. See Embeddings, vector search and RAG.
• Asset: a data object, such as a table, that the orchestrator tracks and keeps up to date. See Orchestration.
• At-least-once: retried until acknowledged; may duplicate messages. See Delivery guarantees and idempotency.
• At-most-once: never retried; may lose messages. See Delivery guarantees and idempotency.
• Atomic commit: a pointer change that succeeds only if the table has not changed since the writer started. See Catalogs.
• Atomic swap: replacing a published table with a staged one in one operation. See Write-audit-publish.
• Audit: the checks run between writing and publishing. See Write-audit-publish.
• Audit log: a record of security-relevant actions: who, what, which resource, when, result. See Audit logging.
• Authentication: verifying who is making a request. See Authentication and authorization: OIDC and JWT.
• Authorization: deciding whether a request is allowed. See Authentication and authorization: OIDC and JWT.

B#

• Backend-for-frontend (BFF): an API dedicated to one client application. See Data APIs and the backend-for-frontend pattern.
• Backfill: running a pipeline for past periods. See Backfill and replay.
• Backward compatibility: new readers can read old data. See Schema evolution.
• Batch serving: scoring on a schedule and writing predictions to a table. See Model serving and monitoring.
• Bin-packing: compaction that combines files without reordering rows. See Iceberg maintenance.
• Blocking test: a test whose failure stops publishing. See Testing data pipelines.
• Bounded data: a finite set with a known end, as processed by a batch. See Processing patterns: batch, micro-batch and streaming.
• Branch: a named, independent line of snapshots in a table. See Write-audit-publish.
• Breaking change: a change that existing readers or writers cannot handle under the agreed compatibility mode, or that changes meaning; removing a field consumers use is breaking even when the registry allows it. See Schema evolution.
• Breaking change (Data contracts): a change that can break consumers, requiring a new major version. See Data contracts.
• Broker: the server that receives MQTT messages and forwards them to subscribers. See IoT and device data.
• Bronze (raw): data as delivered, append-only, kept for rebuilding. See Medallion layers: bronze, silver and gold.
• Bucket: a named container for objects, with a region and access policy. See Object storage and the S3 API.
• Bucket versioning: keeping previous versions of overwritten or deleted objects. See Disaster recovery.
• Bucketing: assigning rows to a fixed number of groups by a hash of a key. See Partitioning.
• Business catalog: a searchable inventory of datasets with business context. See Catalog and lineage.
• Business glossary: agreed definitions of business terms. See Catalog and lineage.
• Business intelligence (BI): tools and practices for reporting and exploring business data. See BI and dashboards.
• Business key: the identifier of an entity in the source, such as order_id. See Deduplication and merge.
• Business metadata: meaning and responsibility, such as descriptions, owners and terms. See Catalog and lineage.

C#

• Canary release: sending a small share of traffic to a new version first. See Model serving and monitoring.
• Cardinality: the number of distinct values of a column. See Partitioning.
• Casting: converting a value to a declared type when reading or transforming. See Schema-on-write and schema-on-read.
• Catalog: the service that knows where each table's current metadata.json is. Every commit goes through it. See Apache Iceberg: table format basics.
• Catastrophic forgetting: a fine-tuned model losing general abilities it had before. See Fine-tuning or RAG.
• CDC metadata columns: operation, source position and commit time kept alongside each change row. See Writing to Iceberg from Kafka.
• Certification: marking a dashboard or dataset as trusted and supported. See BI and dashboards.
• Change event: one row change, with before, after, operation and source metadata. See Change data capture.
• Change-based sync: sending only rows that changed since the last sync. See Reverse ETL and activation.
• Checkpoint: a consistent snapshot of state and source offsets, used to recover. See Stream processing.
• Chunk: a section of a document, embedded and retrieved as a unit. See Embeddings, vector search and RAG.
• Class imbalance: when one outcome is much rarer than others. See Training data management.
• Classification level: a label, such as confidential, that decides handling rules. See PII and data classification.
• Classification tag: a label on a column or table marking its sensitivity. See Governance as code.
• Client credentials flow: how a service obtains a token as itself. See Authentication and authorization: OIDC and JWT.
• CloudEvents: a CNCF specification for common event metadata. See Event design.
• Clustering: placing similar values together, by sorting or Z-ordering, so statistics can skip. See Query performance on lake tables.
• Column bounds: the lowest and highest value of a column in one data file. See Iceberg internals: manifests and planning.
• Column chunk: the values of one column within one row group. See Columnar files and Parquet.
• Column mask: an expression that hides or reduces a column's values for a user. See Row-level security and masking.
• Column-level lineage: lineage between individual columns. See Catalog and lineage.
• Column-oriented storage: all values of a column stored together. See OLTP vs OLAP.
• Command: a request for something to happen, which may be refused. See Event design.
• Commit interval: how often written files are committed to the table. See Writing to Iceberg from Kafka.
• Compacted topic: a topic that keeps the latest record per key. See Kafka and Kafka Connect.
• Compaction: rewriting small files, and their delete files, into fewer large files. See Iceberg maintenance.
• Compatibility mode: the rule a registry enforces for a subject, usually one per topic key or value. See Schema evolution.
• Completeness: no missing records or required values. See Data quality dimensions.
• Concept drift: change in the relationship between inputs and outcomes. See Model serving and monitoring.
• Concurrency: the number of queries served at the same time. See Low-latency serving stores.
• Conditional write: a write that succeeds only if the object is absent or unchanged. See Object storage and the S3 API.
• Conformed: using the same keys, names and codes across sources. See Medallion layers: bronze, silver and gold.
• Conformed dimension: a dimension shared, with the same meaning, by several facts. See Dimensional modelling.
• Confounder: a factor that influences both a cause and an effect, creating a false association. See The data science workflow.
• Connector: a configured source or sink in Kafka Connect. See Kafka and Kafka Connect.
• Connector (SQL query engines): a plugin that lets the engine read and write one kind of system. See SQL query engines.
• Consistency: agreement of the same fact across tables and systems. See Data quality dimensions.
• Consumer: uses data within agreed purposes and terms. See Governance roles.
• Consumer (Data contracts): anyone who relies on the dataset. See Data contracts.
• Consumer group: consumers that share the work of reading a topic, with their own committed offsets. See Kafka and Kafka Connect.
• Continuous integration (CI): automatic checks run on every proposed change. See Environments and CI.
• Control and treatment: the groups that see the current and the changed version. See Experimentation and A/B testing.
• Control topic: the topic on which sink tasks and the coordinator exchange commit information. See Writing to Iceberg from Kafka.
• Coordinator: the process that plans a query and schedules work. See SQL query engines.
• Correlation ID: an ID that links events belonging to one process. See Event design.
• Cosine similarity: a measure of how close two vectors point in the same direction. See Embeddings, vector search and RAG.
• Cost attribution: assigning spend to teams, products or workloads. See Cost and FinOps.
• Credential vending: the catalog issuing short-lived storage credentials scoped to one table. See Catalogs.
• Cross-border transfer: data, or access to it, moving from one jurisdiction to another. See Residency and regions.
• Cross-reference table: the record of which source IDs belong to which platform ID. See Multi-source conformance.
• Crypto-shredding: making data unreadable by destroying its encryption key. See Privacy lifecycle.
• Curated dashboard: a reviewed, owned dashboard intended as an official source. See BI and dashboards.
• Cursor: a position token returned by an API to continue reading. See Ingestion patterns.
• Cursor pagination: paging with a token that marks where the next page starts. See Data APIs and the backend-for-frontend pattern.
• Customer-managed key: an encryption key controlled by the organisation, not the provider. See Encryption and key management.
• Cutover: moving consumers from the old platform to the new one. See Platform migration.

D#

• DAG: a directed acyclic graph of tasks and their dependencies. See Orchestration.
• Data analyst: answers business questions with reports and analysis. See Data team roles.
• Data API: an HTTP interface returning records from platform data. See Data APIs and the backend-for-frontend pattern.
• Data contract: a machine-readable agreement on a dataset's schema, meaning, quality and service levels. See Data contracts.
• Data custodian: operates the systems that store and move the data. See Governance roles.
• Data diff: comparing the output of a changed model with production. See Environments and CI.
• Data drift: change in the distribution of a model's inputs. See Model serving and monitoring.
• Data engineer: builds ingestion and the reliable lower layers. See Data team roles.
• Data lake: files of any format in object storage, without table rules. See Data lake, warehouse and lakehouse.
• Data localisation: a legal requirement that certain data stays within a country. See Residency and regions.
• Data minimisation: collecting and keeping only what is needed. See PII and data classification.
• Data owner: accountable for a dataset; makes decisions about its use and quality. See Governance roles.
• Data product: a published dataset with an owner, interface, contract and documentation. See Data products and data mesh.
• Data product owner: accountable for a data product's value to its consumers. See Data team roles.
• Data residency: the location where data is stored and processed. See Residency and regions.
• Data scientist: uses statistics and models to investigate and predict. See Data team roles.
• Data sovereignty: data being subject to the laws of where it is held, or of who holds it. See Residency and regions.
• Data steward: manages definitions, access reviews and quality issues day to day. See Governance roles.
• Data test: a query that returns rows breaking a business rule. See Testing data pipelines.
• Data Vault: a modelling method using hubs, links and satellites for integration and history. See Other modelling approaches: Data Vault, one big table and activity schema.
• Data warehouse: an analytical database that manages its own storage format. See Data lake, warehouse and lakehouse.
• Dataframe: an in-memory table manipulated with code. See Python on the lakehouse.
• Dataset version: an identifiable, reproducible state of a dataset, such as a table tag. See Training data management.
• Datasheet for datasets: documentation of a dataset's origin, content and permitted uses. See Responsible AI and AI governance.
• Declarative: describing the desired end state rather than the steps to reach it. See Infrastructure as code.
• Declarative configuration: describing the desired state, and letting tools make the systems match it. See Governance as code.
• Decommissioning: switching off the old platform once nothing uses it. See Platform migration.
• Delete file: records which rows a later commit removed. Position deletes point at row positions; equality deletes match on column values. See Apache Iceberg: table format basics.
• Deletion vector: a bitmap marking deleted rows of one data file, in format version 3. See Iceberg maintenance.
• Deletion vector (Table formats compared: Iceberg, Delta Lake, Hudi and Paimon): a per-file record of deleted rows, used by Delta Lake and Iceberg v3. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• Denormalised model: related data combined into wider tables to avoid joins. See OLTP vs OLAP.
• Deterministic matching: matching on exact, normalised identifiers. See Multi-source conformance.
• Dialect translation: converting SQL and functions from one engine to another. See Platform migration.
• Dimension: an attribute that metrics are grouped or filtered by. See Semantic layer.
• Dimension table: descriptive context for facts, one row per member. See Dimensional modelling.
• Direct identifier: a value that identifies a person on its own. See PII and data classification.
• Distillation: training a smaller model to reproduce a larger model's outputs on a task. See Fine-tuning or RAG.
• Domain: a business area with its own team, data and language, such as orders or payments. See Data products and data mesh.
• Drift: a difference between declared and actual configuration. See Governance as code.
• Drift (Infrastructure as code): differences between declared and actual infrastructure. See Infrastructure as code.
• Dual write: writing to two systems separately, with no way to commit both together. See The outbox pattern.
• Dynamic masking: masking applied at query time, depending on the user. See Row-level security and masking.

E#

• Effective dates: valid_from and valid_to of a version. See History and slowly changing dimensions.
• Egress: data transferred out of a region or cloud, usually charged. See Cost and FinOps.
• Embedding: a vector representing the meaning of a piece of content. See Embeddings, vector search and RAG.
• Encoding: a compact representation of values, such as dictionary or run-length. See Columnar files and Parquet.
• Encryption at rest: protecting stored data. See Encryption and key management.
• Encryption in transit: protecting data on the network, usually with TLS. See Encryption and key management.
• Enforcement point: the component that applies filters and masks. See Row-level security and masking.
• Entity: the thing features describe, such as a customer or product. See Feature engineering and feature stores.
• Envelope: standard metadata fields carried by every event. See Event design.
• Envelope encryption: encrypting data with data keys that are themselves encrypted by a master key. See Encryption and key management.
• Environment: an isolated place to build and run pipelines, such as dev, CI or prod. See Environments and CI.
• Erasure request: a request to delete all personal data about one person. See Privacy lifecycle.
• Error budget: the amount of failure the SLO allows. See Observability and SLOs.
• Evaluation set: versioned inputs with criteria or references for good outputs. See LLM applications and evaluation.
• Event: a record of a fact that happened. See Event design.
• Event ID: a unique, producer-generated ID used by consumers to deduplicate. See The outbox pattern.
• Event listener: an engine plugin that receives events, such as completed queries. See Audit logging.
• Event time: when the event happened. See Late and out-of-order data.
• Exactly-once: each message affects the result once, through deduplication or transactions. See Delivery guarantees and idempotency.
• Exactly-once processing: each event affects state once, even after failures. See Stream processing.
• Example: one case to predict for, with an entity and a time. See Training data management.
• Exchange: moving intermediate rows between workers, for example to group or join by key. See SQL query engines.
• Expand and contract: making a breaking change in compatible steps. See Schema evolution.
• Experiment tracking: recording parameters, data, metrics and outputs of every run. See The ML lifecycle and MLOps.
• Explainability: the ability to give reasons for a model's predictions. See Responsible AI and AI governance.
• Exploratory data analysis (EDA): examining data to understand it before testing hypotheses. See The data science workflow.
• Export: a copy of data sent to someone, outside the provider's control. See Data sharing.
• Exposure: the moment a unit actually experiences its variant. See Experimentation and A/B testing.
• Extract: reading data out of a source system. See ETL, ELT and EtLT.
• Extract (BI and dashboards): a copy of data stored in the BI tool and refreshed on a schedule. See BI and dashboards.

F#

• Fact table: rows of events or measurements with numeric measures and dimension keys. See Dimensional modelling.
• Failover: switching service to a replica in another location. See Disaster recovery.
• Fairness check: comparing a model's performance or outcomes across groups. See Responsible AI and AI governance.
• Fast-forward: moving a branch to a later snapshot of another branch. See Write-audit-publish.
• Fault-tolerant execution: retrying failed parts of a query instead of the whole query. See SQL query engines.
• Feature: an input variable to a model. See Feature engineering and feature stores.
• Federated governance: shared rules agreed by domain and platform representatives, enforced automatically, with all other decisions left to domains. See Data products and data mesh.
• Federation: one query reading from several systems through connectors. See SQL query engines.
• File catalog table: a table with one row per file, describing and governing it. See Unstructured and multimodal data.
• File skipping: dropping files whose column bounds cannot match a filter. See Query performance on lake tables.
• Fine-tuning: further training a model on task examples to change its behaviour. See Fine-tuning or RAG.
• FinOps: a practice of shared ownership of cloud cost between engineering and finance. See Cost and FinOps.
• Footer: metadata at the end of the file: schema, chunk positions and statistics. See Columnar files and Parquet.
• Format version: v2 added row-level deletes; v3 adds deletion vectors and new data types. Engines differ in which versions they support. See Apache Iceberg: table format basics.
• Forward compatibility: old readers can read new data. See Schema evolution.
• Framing: turning a question into a precise metric, population, window and decision. See The data science workflow.
• Freshness: how old the data is when someone reads it. See Processing patterns: batch, micro-batch and streaming.
• Full compatibility: both directions. See Schema evolution.
• Full load: copying the complete source dataset on every run. See Ingestion patterns.

G#

• Generic, singular and unit tests: column checks, custom queries, and logic tests on fixed inputs. See dbt.
• GitOps: keeping systems in sync with configuration stored in Git. See Infrastructure as code.
• Gold (curated): tables shaped for a specific use, with owners and stable definitions. See Medallion layers: bronze, silver and gold.
• Golden record: the combined record after survivorship. See Multi-source conformance.
• Grain: what exactly one fact row represents. See Dimensional modelling.
• Grain in the name: naming a table after what one row represents. See Naming conventions.
• Granularity: how much change one event describes. See Event design.
• Groundedness: whether an answer is supported by the retrieved sources. See LLM applications and evaluation.
• Guardrail: a check that blocks or corrects unsafe inputs or outputs. See LLM applications and evaluation.

H#

• Headless semantic layer: a standalone service exposing metrics through APIs to any tool. See Semantic layer.
• Hidden partitioning: the table derives partition values from a column by a stored rule. See Partitioning.
• High-risk AI system: under the EU AI Act, a system in a listed sensitive area with strict requirements. See Responsible AI and AI governance.
• Hive-style partitioning: partition values written into file paths as key=value. See Partitioning.
• Home region: the region a record belongs to, decided by a rule such as residence. See Residency and regions.
• Hub: a table of unique business keys. See Other modelling approaches: Data Vault, one big table and activity schema.
• Human oversight: people able to review, override and monitor an AI system. See Responsible AI and AI governance.
• Hybrid search: combining vector similarity with keyword matching. See Embeddings, vector search and RAG.
• Hypothesis: a candidate explanation that predicts something checkable in the data. See The data science workflow.

I#

• Iceberg REST catalog: the HTTP API, defined by the Iceberg project, that engines use to talk to a catalog. See Catalogs.
• Idempotency key: a stable ID used to recognise repeats of the same event. See Delivery guarantees and idempotency.
• Idempotent model: a model that gives the same result when re-run for the same period. See Backfill and replay.
• Idempotent operation: one that gives the same result when repeated. See Delivery guarantees and idempotency.
• Idempotent producer: a Kafka producer whose retries are deduplicated by the broker. See Delivery guarantees and idempotency.
• Identifier mapping: the field used to match a row to a record in the target. See Reverse ETL and activation.
• Identity provider: the service that authenticates users and issues tokens. See Authentication and authorization: OIDC and JWT.
• Identity resolution: assigning one platform ID to all records of one entity. See Multi-source conformance.
• Incremental load: copying only what changed since the previous run. See Ingestion patterns.
• Incremental model: a model that processes only new or changed rows on each run. See dbt.
• Indirect identifier: a value that can identify a person in combination with others. See PII and data classification.
• Infrastructure as code (IaC): managing infrastructure through declarative, version-controlled files. See Infrastructure as code.
• Ingestion time: when the event reached the platform. See Late and out-of-order data.
• Inventory: the list of assets and consumers on the old platform, with usage. See Platform migration.
• Isolation model: how tenants are separated: shared tables, schema per tenant, or stack per tenant. See Multi-tenancy.

J#

• JWT: a signed token carrying claims, verified with the issuer's public keys. See Authentication and authorization: OIDC and JWT.

K#

• Key: the full name of an object within a bucket. See Object storage and the S3 API.
• Key management service (KMS): a service that holds keys and performs key operations under access control. See Encryption and key management.
• Key-value store: a database that returns records by key with very low latency. See Low-latency serving stores.

L#

• Label: the known outcome a model learns to predict. See Training data management.
• Lakehouse: open files in object storage plus an open table format and a catalog. See Data lake, warehouse and lakehouse.
• Lance: an open columnar format designed for multimodal data with fast random access. See Unstructured and multimodal data.
• Landing: writing data to the raw layer as delivered, with load metadata. See Ingestion patterns.
• Latency: how long processing takes, from an event arriving to its result being available. See Processing patterns: batch, micro-batch and streaming.
• Lateness: the delay between event time and processing time. See Late and out-of-order data.
• Lazy evaluation: building a query plan first and running it once, so only needed data is read. See Python on the lakehouse.
• Leakage: features containing information not available at prediction time. See Feature engineering and feature stores.
• Least privilege: granting only the access a user or service needs. See RBAC and ABAC.
• Lineage: the dependencies between datasets, jobs and reports. See Catalog and lineage.
• Link: a table of relationships between business keys. See Other modelling approaches: Data Vault, one big table and activity schema.
• Live query: a dashboard query sent to the engine each time it is viewed. See BI and dashboards.
• LLM-as-a-judge: using a model to score outputs against criteria. See LLM applications and evaluation.
• LLMOps: practices for building, evaluating and operating LLM applications. See LLM applications and evaluation.
• Load: writing data into the target store. See ETL, ELT and EtLT.
• Load metadata: columns added during loading, such as source system and ingestion time. See ETL, ELT and EtLT.
• Log-based CDC: reading changes from the database's transaction log. See Change data capture.
• Logical date: the period a run processes, independent of when it runs. See Orchestration.
• LoRA: a parameter-efficient fine-tuning method that trains small added weight matrices. See Fine-tuning or RAG.
• LSM tree: a log-structured merge tree, a storage design optimised for frequent writes. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.

M#

• Manifest: a list of files with per-column statistics, used to skip files that cannot match a query. See Apache Iceberg: table format basics.
• Manifest (Iceberg internals: manifests and planning): an Avro file listing data or delete files with their partition and column metrics. See Iceberg internals: manifests and planning.
• Manifest list: the per-snapshot list of manifests, with a partition summary for each. See Iceberg internals: manifests and planning.
• Mapping: aligning fields, types and codes of several sources to one structure. See Multi-source conformance.
• Materialization: how a model is built: view, table, incremental, ephemeral. See dbt.
• MCP host: the application in which the agent runs and connects to servers. See Agents and MCP.
• MCP server: a service that exposes tools over MCP. See Agents and MCP.
• MERGE: one statement that inserts, updates and deletes rows in a target based on a source. See Deduplication and merge.
• Metadata column: a column added by the platform, such as _ingested_at. See Naming conventions.
• Metadata file (metadata.json): the root of the tree, holding the schema, partitioning and snapshot list. See Apache Iceberg: table format basics.
• Metadata table: a read-only table exposing metadata, such as $files or $manifests. See Iceberg internals: manifests and planning.
• Metric: a named calculation with business meaning, such as revenue. See Semantic layer.
• Metric, log, trace: numeric measurements over time, event records, and the path of one request or run across components. See Observability and SLOs.
• Metrics API: an interface that returns metric values by dimension. See Semantic layer.
• Metrics API (Data APIs and the backend-for-frontend pattern): an interface returning aggregated metric values. See Data APIs and the backend-for-frontend pattern.
• ML engineer: puts models into production and keeps them running. See Data team roles.
• MLOps: practices for building, deploying and operating models reliably. See The ML lifecycle and MLOps.
• Model: one SELECT statement in a file, built into a table or view. See dbt.
• Model card: documentation of a model's purpose, performance, evaluation and limits. See Responsible AI and AI governance.
• Model Context Protocol (MCP): an open standard for exposing tools, resources and prompts to agents. See Agents and MCP.
• Model registry: a catalog of model versions with metadata, status and approvals. See The ML lifecycle and MLOps.
• Model signature: the declared input and output schema of a model. See The ML lifecycle and MLOps.
• Module: a reusable package of infrastructure declarations. See Infrastructure as code.
• MQTT: a light publish and subscribe protocol for devices and unreliable networks. See IoT and device data.
• Multimodal data: data combining several types, such as images with text. See Unstructured and multimodal data.
• Multipart upload: uploading a large object in parts that become visible together. See Object storage and the S3 API.

N#

• Namespace: a grouping of tables, like a schema in a database. See Catalogs.
• Naming convention: an agreed rule for names, enforced in review or CI. See Naming conventions.
• Noisy neighbour: a tenant whose load degrades performance for others. See Multi-tenancy.
• Normalised model: each fact stored once, across many related tables. See OLTP vs OLAP.
• Notebook: an interactive document of code, results and notes. See Python on the lakehouse.

O#

• OAuth 2.0: a framework for issuing access tokens to clients. See Authentication and authorization: OIDC and JWT.
• Object lock: write-once retention that prevents deletion for a period. See Disaster recovery.
• Observability: understanding a system's state from the signals it emits. See Observability and SLOs.
• OCR: reading text from images of documents. See Unstructured and multimodal data.
• ODCS: the Open Data Contract Standard, a YAML format for contracts. See Data contracts.
• Offline and online store: historical feature values for training, and latest values for low-latency serving. See Feature engineering and feature stores.
• Offset: a record's position within a partition. See Kafka and Kafka Connect.
• On-behalf-of identity: the user for whom a service performs an action. See Audit logging.
• One big table (OBT): a single wide denormalised table for a specific use. See Other modelling approaches: Data Vault, one big table and activity schema.
• Online serving: scoring per request behind an API, with low latency. See Model serving and monitoring.
• Open table format: a specification that turns a set of files into a table with transactions and a schema. See Data lake, warehouse and lakehouse.
• Open table format (Table formats compared: Iceberg, Delta Lake, Hudi and Paimon): a specification that turns files into tables with versions and atomic commits. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• Open-weight model: a model whose weights can be downloaded and run on own infrastructure. See LLM applications and evaluation.
• OpenID Connect: an identity layer on OAuth 2.0 that adds login and ID tokens. See Authentication and authorization: OIDC and JWT.
• OpenLineage: an open standard for emitting lineage events from running jobs. See Catalog and lineage.
• Orphan file: a file in the table location that no snapshot references. See Iceberg maintenance.
• Out-of-core processing: handling data larger than memory by spilling to disk. See Python on the lakehouse.
• Outbox routing: sending outbox rows to topics by event type or aggregate. See The outbox pattern.
• Outbox table: a table in the service's database holding events to publish. See The outbox pattern.
• Output port: the published interface of a data product. See Data products and data mesh.

P#

• Page: the unit of encoding and compression inside a column chunk. See Columnar files and Parquet.
• Parallel run: running old and new pipelines side by side and comparing outputs. See Platform migration.
• Parsing: extracting text and structure from documents. See Unstructured and multimodal data.
• Partition: an ordered, append-only part of a topic; the unit of ordering and parallelism. See Kafka and Kafka Connect.
• Partition key: the column, or derived value, that decides which partition a row belongs to. See Partitioning.
• Partition pruning: skipping partitions that cannot match a query's filter. See Partitioning.
• Partition summary: the lowest and highest partition values in one manifest. See Iceberg internals: manifests and planning.
• Partition transform: the rule that derives partition values, such as day(ordered_at) or bucket(16, customer_id). See Apache Iceberg: table format basics.
• Partition-scoped run: a run that processes and writes exactly one period. See Backfill and replay.
• Peeking: repeatedly checking results and stopping early. See Experimentation and A/B testing.
• Personal data (PII): information relating to an identified or identifiable person. See PII and data classification.
• Physical deletion: the data no longer exists in any file, as opposed to being hidden from queries. See Privacy lifecycle.
• Plan: a preview of the changes an apply would make. See Infrastructure as code.
• Platform engineer: builds and operates the shared data platform. See Data team roles.
• Platform team: provides shared tooling and enforces cross-cutting rules. See Governance roles.
• Point-in-time join: joining on a key and a time within the validity range. See History and slowly changing dimensions.
• Point-in-time join (Feature engineering and feature stores): matching each example with feature values as they were at its time. See Feature engineering and feature stores.
• Policy as code: access and handling rules written in files, often in a policy language, and tested. See Governance as code.
• Policy decision point: the component that evaluates access rules. See RBAC and ABAC.
• Policy enforcement point: the component that applies the decision to a request. See RBAC and ABAC.
• Policy engine: a service that evaluates policies at request time, such as Open Policy Agent. See Governance as code.
• Power: the probability of detecting an effect of a given size if it exists. See Experimentation and A/B testing.
• Pre-aggregation: a stored summary table used to answer common queries quickly. See Semantic layer.
• Pre-signed URL: a time-limited link that grants read access to one object. See Data sharing.
• Predicate pushdown: using a filter to skip data before reading it. See Columnar files and Parquet.
• Predicate pushdown (Query performance on lake tables): passing a filter to the table reader so data can be skipped before it is read. See Query performance on lake tables.
• Prediction log: a record of every prediction with inputs and model version. See Model serving and monitoring.
• Prefix: the start of a key, used for listing and access rules; what consoles show as a folder. See Object storage and the S3 API.
• Prefix (Naming conventions): a name start that marks a layer or table type, such as clean_ or fct_. See Naming conventions.
• Primary and guardrail metrics: what should improve, and what must not get worse. See Experimentation and A/B testing.
• Probabilistic matching: matching on scored similarity across several fields. See Multi-source conformance.
• Processing time: when the pipeline handles the event. See Late and out-of-order data.
• Producer: the team that creates and changes the dataset. See Data contracts.
• Projection: reading only the columns a query needs. See Columnar files and Parquet.
• Projection pushdown: reading only the columns a query uses. See Query performance on lake tables.
• Promotion: moving a model version to a higher status, such as production, under defined rules. See The ML lifecycle and MLOps.
• Prompt injection: instructions hidden in data that try to steer the model. See Agents and MCP.
• Prompting: guiding a model with instructions and examples in each request. See Fine-tuning or RAG.
• Provider: the party that owns and shares the data. See Data sharing.
• Pseudonymisation: replacing identifiers with pseudonyms while the re-identification key is kept apart. See PII and data classification.
• Puffin file: a file of extra table statistics, such as distinct value estimates, or deletion vectors. See Iceberg internals: manifests and planning.
• Pull and push: the platform fetches data, or the source sends it. See Ingestion patterns.
• Pull request: a proposed change, reviewed and checked before merging. See Environments and CI.
• PyIceberg: the Python library for reading and writing Iceberg tables. See Python on the lakehouse.

Q#

• QoS level: MQTT's delivery guarantee per message: 0, 1 or 2. See IoT and device data.
• Query plan: the steps an engine will run, shown by EXPLAIN. See Query performance on lake tables.
• Query-based CDC: finding changes by querying a change column on a schedule. See Change data capture.

R#

• RACI: a matrix of who is Responsible, Accountable, Consulted and Informed per decision. See Governance roles.
• Randomisation unit: what is assigned to a group, such as a user. See Experimentation and A/B testing.
• Range read: reading only part of an object by byte offsets. See Object storage and the S3 API.
• Rate limit: the maximum number of API calls a target allows per period. See Reverse ETL and activation.
• Raw layer: the copy of source data kept as delivered, the basis for re-running transformations. See ETL, ELT and EtLT.
• RBAC: permissions granted to roles; users get permissions through roles. See RBAC and ABAC.
• Real-time OLAP: databases that answer aggregate queries in milliseconds on fresh data. See Low-latency serving stores.
• ReBAC: permissions decided from relationships between users and objects. See RBAC and ABAC.
• Rebuild: recreating a layer from the one below after a logic change or a bug. See Medallion layers: bronze, silver and gold.
• Recipient: the party that reads shared data with its own tools. See Data sharing.
• Reconciliation: comparing outputs to find and explain differences. See Platform migration.
• Reconciliation test: compares counts or totals across tables or systems. See Testing data pipelines.
• Recovery point objective (RPO): the maximum acceptable data loss, as time. See Disaster recovery.
• Recovery time objective (RTO): the maximum acceptable time to restore. See Disaster recovery.
• ref(): a reference to another model, from which dbt builds the dependency graph. See dbt.
• Regional stack: a full set of storage, catalog and compute for one region. See Residency and regions.
• Relationship: how tables join, declared once in the layer. See Semantic layer.
• Remote signing: the catalog signing each storage request on the engine's behalf. See Catalogs.
• Replay: reprocessing data from the raw layer or from a log. See Backfill and replay.
• Replication slot: PostgreSQL's record of how far a consumer has read the log. See Change data capture.
• Reprocessing window: the number of recent days a batch job recomputes on each run. See Late and out-of-order data.
• Reproducibility: the ability to rerun an analysis on the same data and code and get the same result. See The data science workflow.
• Resource group: an engine setting that limits resources for a set of queries. See Multi-tenancy.
• Resource tagging: labelling cloud resources with owner and purpose. See Cost and FinOps.
• Retained message: the last message on a topic, kept by the broker for new subscribers. See IoT and device data.
• Retention: how long or how much data a topic keeps. See Kafka and Kafka Connect.
• Retention period: how long snapshots are kept, which is also how far back time travel reaches. See Iceberg maintenance.
• Retention period (Privacy lifecycle): how long a kind of data is kept, and why. See Privacy lifecycle.
• Retrieval-augmented generation (RAG): answering with a model that is given retrieved documents. See Embeddings, vector search and RAG.
• Retrieval-augmented generation (RAG) (Fine-tuning or RAG): adding retrieved documents to each request. See Fine-tuning or RAG.
• Retry: re-running a failed task automatically after a delay. See Orchestration.
• Reverse ETL: syncing platform results into operational tools. See Reverse ETL and activation.
• Rollback: making an earlier snapshot the current state of a table. See Disaster recovery.
• Row filter: a condition that limits which rows a user sees. See Row-level security and masking.
• Row group: a horizontal slice of a file, holding one column chunk per column. See Columnar files and Parquet.
• Row-oriented storage: all columns of a row stored together. See OLTP vs OLAP.
• Run parameter: the period a run processes, passed in instead of taken from the clock. See Backfill and replay.
• Runbook: written steps for responding to an alert. See Observability and SLOs.

S#

• Sample ratio mismatch (SRM): a difference between planned and actual group sizes that signals a bug. See Experimentation and A/B testing.
• Sandbox schema: a personal or project area for experimental tables. See The data science workflow.
• Satellite: a table of descriptive attributes and their history, attached to a hub or link. See Other modelling approaches: Data Vault, one big table and activity schema.
• Scan: reading a large part of a table rather than looking up single rows. See OLTP vs OLAP.
• Scheduler: the component that starts runs and launches tasks when ready. See Orchestration.
• Schema: the declared names, types and constraints of a table's columns. See Schema-on-write and schema-on-read.
• Schema drift: unannounced changes in the structure of source data. See Schema-on-write and schema-on-read.
• Schema enforcement: rejecting writes that do not match the schema. See Schema-on-write and schema-on-read.
• Schema registry: a service that stores schema versions and checks compatibility. See Schema evolution.
• Schema test: checks columns, types and constraints. See Testing data pipelines.
• Secrets manager: a service that stores and distributes credentials securely. See Encryption and key management.
• Self-serve platform: shared infrastructure that lets teams build and publish products on their own. See Data products and data mesh.
• Self-service analytics: users exploring governed data on their own. See BI and dashboards.
• Semi-structured data: records with structure that may vary, such as JSON. See Schema-on-write and schema-on-read.
• Sensor: a task that waits for a condition before downstream tasks start. See Orchestration.
• Separation of storage and compute: data stored once, processed by engines that can be scaled or swapped independently. See Data lake, warehouse and lakehouse.
• Sequence number: the order of commits, used to decide which deletes apply to which files. See Iceberg internals: manifests and planning.
• Service account: an identity for a non-human client. See Authentication and authorization: OIDC and JWT.
• Service level: a promise about availability, such as data ready by 07:00. See Data contracts.
• Service level agreement (SLA): a promise to customers, with consequences if broken. See Observability and SLOs.
• Service level indicator (SLI): the measured quantity, such as completion time. See Observability and SLOs.
• Service level objective (SLO): a target for an SLI over a period, such as 98% of days. See Observability and SLOs.
• Serving store: a database optimised for fast application queries, fed by the platform. See Data APIs and the backend-for-frontend pattern, Low-latency serving stores.
• Shadow deployment: running a new model on live traffic without using its predictions. See Model serving and monitoring.
• Share: a named set of tables or views made available to a recipient. See Data sharing.
• Sharing protocol: an open API for discovering and reading shared tables. See Data sharing.
• Showback and chargeback: reporting cost to teams, or billing it to their budgets. See Cost and FinOps.
• SIEM: a system that collects security logs and detects suspicious patterns. See Audit logging.
• Silver (clean): typed, deduplicated, conformed entities. See Medallion layers: bronze, silver and gold.
• Sink connector: a Kafka Connect component that writes topic records to an outside system. See Writing to Iceberg from Kafka.
• Slim CI: building only changed models and their dependants, reusing production for the rest. See Environments and CI.
• Slowly changing dimension: a dimension whose attributes change over time. See History and slowly changing dimensions.
• Snapshot: the state of the table after one commit. See Apache Iceberg: table format basics.
• Snapshot (Change data capture): the initial read of a table's current rows before streaming changes (not an Iceberg snapshot). See Change data capture.
• Snapshot expiry: removing old snapshots and the files only they referenced. See Iceberg maintenance.
• Snapshot table: a full copy of a table per day or period. See History and slowly changing dimensions.
• Soft delete: marking a row as deleted instead of removing it; it keeps the stored sequence, so a replay cannot bring a deleted row back. See Deduplication and merge.
• Source: a raw table loaded outside dbt, declared so models can reference it. See dbt.
• Source of truth: the system whose data is authoritative when copies disagree. See Low-latency serving stores.
• Source sequence: an ordering from the source, such as a log position or offset. See Deduplication and merge.
• Star schema: a fact table joined directly to its dimensions. See Dimensional modelling.
• State: what a stream processor remembers between events, such as running totals. See Processing patterns: batch, micro-batch and streaming.
• State (Infrastructure as code): the tool's record mapping declarations to real resources. See Infrastructure as code.
• State (Stream processing): data a job keeps between events, partitioned by key. See Stream processing.
• Storage limitation: the principle of keeping personal data no longer than needed. See Privacy lifecycle.
• Stream processor: an engine that runs continuous jobs over event streams. See Stream processing.
• Stream-table join: enriching events with the current version of a table held in state. See Stream processing.
• Subject: the business object an event is about. See Event design.
• Suffix: a name end that marks a type or meaning, such as _at or _key. See Naming conventions.
• Suppression list: IDs of erased people, used to block reintroduction. See Privacy lifecycle.
• Surrogate key: a key created in the warehouse to identify a dimension row. See Dimensional modelling.
• Survivorship: per-attribute rules for which source value wins. See Multi-source conformance.
• Sync: a configured transfer from a source model to a destination object. See Reverse ETL and activation.
• Synthetic data: generated data that imitates real data. See Training data management.

T#

• Table statistics: row counts and distinct value estimates used by the optimizer. See Query performance on lake tables.
• Tag: a named reference to one snapshot, with its own retention. See Write-audit-publish.
• Target: the configuration that decides where and how a run writes. See Environments and CI.
• Task: one unit of work, such as running a model or loading a file. See Orchestration.
• Technical catalog: maps table names to current metadata and coordinates commits. See Catalogs.
• Technical metadata: structure, such as tables, columns and types. See Catalog and lineage.
• Telemetry: periodic measurements sent by devices. See IoT and device data.
• Tenant: a customer or organisation whose data must be kept separate. See Multi-tenancy.
• Tenant key: the column identifying which tenant a row belongs to. See Multi-tenancy.
• Test fixture: the fixed input rows used by a unit test. See Testing data pipelines.
• Tie-breaker: an extra ordering column that makes the choice deterministic. See Deduplication and merge.
• Time-based split: splitting by date so the model is tested on later data than it learned from. See Training data management.
• Timeline: Hudi's record of actions on a table. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• Timeliness (freshness): availability by the time it is needed. See Data quality dimensions.
• Tokenisation: replacing a sensitive value with a token that can be reversed only by a separate, protected service. See Row-level security and masking.
• Tombstone: a record with a key and no value, marking a deleted key. See Change data capture.
• Tool: a function an agent can call, described by name, purpose and input schema. See Agents and MCP.
• Topic: a named stream of records. See Kafka and Kafka Connect.
• Topic hierarchy: slash-separated topic names that can be matched with + and # wildcards. See IoT and device data.
• Trace: the full record of one request through an LLM application. See LLM applications and evaluation.
• Training run: one execution of training on a specific data version, code version and settings. See The ML lifecycle and MLOps.
• Training, validation and test sets: data for learning, for tuning, and for final evaluation. See Training data management.
• Training-serving skew: a difference between features at training and at prediction time. See Feature engineering and feature stores.
• Transaction: a group of reads and writes that succeeds or fails as a whole. See OLTP vs OLAP.
• Transaction log: Delta Lake's ordered record of commits. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• Transcription: converting speech to text. See Unstructured and multimodal data.
• Transform: cleaning, combining and reshaping data for use. See ETL, ELT and EtLT.
• Type 1: overwrite; no history. See History and slowly changing dimensions.
• Type 2: one row per version, with validity dates. See History and slowly changing dimensions.

U#

• Unbounded data: a stream with no end. See Processing patterns: batch, micro-batch and streaming.
• UniForm: Delta Lake's feature that also writes Iceberg-compatible metadata. See Table formats compared: Iceberg, Delta Lake, Hudi and Paimon.
• Uniqueness: no duplicate entities. See Data quality dimensions.
• Unit cost: cost per unit of value, such as per dashboard view or per million orders processed. See Cost and FinOps.
• Unit test: runs logic on fixed inputs and compares with expected output. See Testing data pipelines.
• Unstructured data: content without a fixed tabular structure, such as documents, images and audio. See Unstructured and multimodal data.
• Upsert mode: records replace or delete existing rows by key. See Writing to Iceberg from Kafka.

V#

• Validity: conformance to type, format and allowed values. See Data quality dimensions.
• Versioned load: loading a new table version and switching readers to it at once. See Low-latency serving stores.

W#

• Watermark: the stored position, such as the highest updated_at loaded, from which the next run continues (not the streaming watermark in Late and out-of-order data). See Ingestion patterns.
• Watermark (streaming): an estimate that no events older than a given time are still to come. See Late and out-of-order data.
• Window: a slice of a stream, such as the last five minutes, used to compute results. See Processing patterns: batch, micro-batch and streaming.
• Window (Stream processing): a time-based grouping of events: tumbling, sliding or session. See Stream processing.
• Winning record: the record chosen as current for a key. See Deduplication and merge.
• Worker: a process that reads data and executes part of the plan. See SQL query engines.
• Write-audit-publish (WAP): write unseen, check, then publish atomically. See Write-audit-publish.
• Write-once storage: storage where records cannot be changed or deleted for a retention period. See Audit logging.