Skip to content

Catalog and lineage#

A business data catalog is where people find, understand and trust data: it lists datasets with their descriptions, owners, terms and quality, and shows lineage, the path data takes from sources to reports.

Before reading

dbt · Governance roles

Why it matters#

In a platform with hundreds of tables, people cannot ask someone every time they need data. Without a catalog they guess, copy the wrong table, or build a new one. Without lineage, nobody can tell what breaks when a column changes, or where a wrong number on a dashboard came from.

How it works#

This catalog is not the technical catalog that engines use to find a table's current metadata. The technical catalog answers "where are the files?"; the business catalog answers "what is this, who owns it, and can I trust it?". Some products combine both, but the jobs are different.

A business catalog holds metadata of several kinds. Technical metadata: tables, columns and types, collected from engines and technical catalogs. Business metadata: descriptions, owners, stewards, classification tags, and links to glossary terms. Operational metadata: when a table was last updated, its quality status, how often it is queried and by whom. Search across all of it is the main feature: people type "revenue" and find the certified table, its definition and its owner.

A business glossary defines the terms the business uses, such as active customer, net revenue or order, once and in plain language. Columns and tables are linked to terms, so that everyone who sees net_revenue sees the same definition.

Lineage shows dependencies. Table-level lineage says fct_daily_sales is built from clean_orders and clean_order_items, which are built from raw_orders and raw_order_items. Column-level lineage goes further: revenue in fct_daily_sales comes from amount in clean_order_items, filtered on status in clean_orders, and that amount comes from the order_items table in the orders database. Column-level lineage answers the precise questions: which dashboards use this column, and which source fields feed this number.

flowchart LR
  SRC[Orders DB<br/>order_items.amount] --> RAW[raw_order_items.amount]
  RAW --> CL[clean_order_items.amount]
  CL --> F[fct_daily_sales.revenue]
  ST[clean_orders.status] -->|filter| F
  F --> D[Dashboard:<br/>Daily revenue]
  CL --> ML[Model features:<br/>avg_order_value]

Metadata is collected in two ways. Pull: the catalog's connectors scan engines, technical catalogs, dbt projects and BI tools on a schedule, and parse SQL to work out lineage. Push: tools emit metadata events as they run. OpenLineage, an open standard under the Linux Foundation, defines events for jobs and datasets, including a facet for column-level lineage; orchestrators, Spark and dbt integrations can send them. dbt alone provides a lot: its manifest contains the model graph, descriptions, tests and owners.

Open-source catalogs include DataHub and OpenMetadata; commercial ones include Atlan, Collibra and Alation; platform suites such as Unity Catalog include catalog features. All depend on the same thing: metadata that is maintained where the work happens, in code and pipelines, not typed into the catalog by hand afterwards.

Certification marks trusted datasets. The owner or steward certifies that a table is documented, tested and supported; search ranks certified tables first, and uncertified ones are marked as use-at-your-own-risk.

Key terms#

  • Business catalog A searchable inventory of datasets with business context.
  • Technical metadata Structure, such as tables, columns and types.
  • Business metadata Meaning and responsibility, such as descriptions, owners and terms.
  • Business glossary Agreed definitions of business terms.
  • Lineage The dependencies between datasets, jobs and reports.
  • Column-level lineage Lineage between individual columns.
  • OpenLineage An open standard for emitting lineage events from running jobs.

Example#

Finance reports that yesterday's revenue on the dashboard is 3% higher than in the accounting system. Tracing it in the catalog:

  1. The dashboard tile Daily revenue reads fct_daily_sales.revenue.
  2. Column lineage shows revenue comes from clean_order_items.amount, filtered on clean_orders.status.
  3. clean_order_items.amount comes from raw_order_items.amount, loaded by CDC from the orders database.
  4. The glossary term revenue, linked to the column, says "tax included, excludes cancelled and refunded orders". The model excludes cancelled orders but not refunded ones.

The steward of fct_daily_sales logs the gap; the custodian adds refunds from clean_payments. Lineage also shows two other dashboards and one model feature built on the same column, whose owners are notified.

Pitfalls#

  • A catalog filled by hand. It is out of date within weeks; collect metadata from code and runs.
  • No owners or descriptions. A list of table names is not a catalog.
  • Table-level lineage only. Impact analysis then flags everything downstream of a table, even for one unused column.
  • Lineage gaps at tool boundaries. Notebooks, scripts and BI tools that are not integrated break the chain.
  • Treating the business and technical catalog as the same decision. One choice rarely serves both jobs well.

Learn more#

Catalogs Governance as code Data quality dimensions PII and data classification