Skip to content

Environments and CI

Separate environments, such as development, test and production, let changes to pipelines be built and checked before they affect real consumers; continuous integration (CI) runs those checks automatically on every proposed change.

Before reading

dbt · Testing data pipelines

Why it matters

A change to one model can break dozens of downstream tables, dashboards and exports. If changes are made directly in production, consumers find the breakage. Environments and CI move that discovery to before the merge, and make releases routine instead of risky.

How it works

Data environments differ from application environments in one important way: the code can be copied easily, but the data cannot. A test environment with no data tests nothing; a full copy of production data is expensive and may expose personal data. The usual answer is a mix:

  • Development: each developer builds models into their own schema, such as dev_dana, reading production raw data or a sample of it, with their own compute.
  • CI: each pull request builds the changed models and everything downstream into a temporary schema, such as ci_pr_412, on a limited sample or a limited time range, then runs the tests and removes the schema.
  • Production: the main branch, built by the orchestrator into the real schemas.

Environments are selected by configuration, not by code. The same dbt project runs everywhere; a target or variable decides the catalog or schema it writes to and the compute it uses. Table names never contain the environment.

CI runs on each pull request and checks several things: the code compiles; linting and naming rules pass; unit tests pass; the changed models and their downstream models build; data tests pass on the CI data; contracts on published models are unchanged, or changed with a proper version; and governance rules hold, such as every new table having an owner. A change that fails any check cannot be merged.

Building only what changed keeps CI fast. dbt can compare the pull request with the production state, select modified models and their downstream models, and read unchanged upstream models from production instead of rebuilding them. This is called slim CI or state-based selection.

A useful extra check compares results: build the changed model in CI and compare it with production, row counts and key metrics by day. A change that was meant only to rename a column but changes revenue by 3% is caught before merge.

flowchart TB
  PR[Pull request:<br/>change to fct_daily_sales] --> C1[Compile, lint,<br/>unit tests]
  C1 --> C2[Build changed models<br/>and downstream into ci_pr_412,<br/>upstream read from prod]
  C2 --> C3[Data tests,<br/>contract check,<br/>compare with prod]
  C3 -->|all pass, reviewed| M[Merge]
  M --> D[Deploy: prod build<br/>by orchestrator]

Deploying is the orchestrator picking up the merged code for the next production run. Changes that need more than that, such as a backfill of history or a new column in a published table, follow their own procedures: backfill, write-audit-publish or contract versioning.

Key terms

  • Environment: an isolated place to build and run pipelines, such as dev, CI or prod.
  • Continuous integration (CI): automatic checks run on every proposed change.
  • Pull request: a proposed change, reviewed and checked before merging.
  • Target: the configuration that decides where and how a run writes.
  • Slim CI: building only changed models and their dependants, reusing production for the rest.
  • Data diff: comparing the output of a changed model with production.

Example

Dana changes fct_daily_sales to exclude refunded orders. The pull request triggers CI:

dbt build --target ci --select state:modified+ --defer --state prod-manifest/

state:modified+ selects fct_daily_sales and everything downstream; --defer reads clean_orders and other unchanged models from production. The models build into ci_pr_412 on the last seven days of data. The reconciliation test passes. No schema check fails, because revenue keeps its name and type, but the change alters what revenue means, which the contract's versioning rule treats as breaking. The pull request therefore also updates the contract's description, bumps its major version and notifies finance. The data diff shows revenue 1.8% lower on each day, which matches the refund rate, and the reviewer approves. After merge, the next nightly run builds the change in production, and a backfill for earlier months is scheduled separately.

Pitfalls

  • Changing production directly. Even small fixes skip every check.
  • Environment names in table names or code. Configuration should decide the target, so code is identical everywhere.
  • CI that rebuilds everything. It becomes too slow to run on every change, so people stop waiting for it.
  • Production personal data in development. Use samples with sensitive columns masked or removed.
  • No comparison with production. Tests pass while the numbers change unexpectedly.

Learn more

  • dbt documentation, continuous integration: https://docs.getdbt.com/docs/deploy/continuous-integration
  • dbt documentation, defer: https://docs.getdbt.com/reference/node-selection/defer

Orchestration · Data contracts · Governance as code · Write-audit-publish