# Write-audit-publish A pattern in which new data is written where consumers cannot see it, checked, and only then made visible in one atomic step. > [!NOTE] > > **Before reading** > > [Testing data pipelines]() ยท [Apache Iceberg: table format basics]() ## Why it matters If a pipeline writes straight into a published table and then runs its tests, consumers can read bad data between the write and the failed test, and someone has to clean it up afterwards. Write-audit-publish (WAP) moves the tests in between. Consumers see either the last good state or the new checked state, never data that failed its checks. ## How it works The pattern has three steps. Write: the pipeline writes its output to a place that consumers do not read. This can be a separate staging table, or a branch of the published table. Audit: tests run against the written data: schema, data and reconciliation tests, and comparisons with the previous version, such as row counts that change by more than an expected amount. Publish: if every blocking check passes, the new data becomes the published version in a single atomic operation. If a check fails, nothing is published; the staged data is kept for investigation and consumers keep reading the previous version. ``` sequenceDiagram participant J as Pipeline participant B as Branch audit participant M as Branch main (published) participant C as Dashboards C->>M: read (yesterday's data) J->>B: write today's fct_daily_sales J->>B: run checks alt all checks pass J->>M: fast-forward main to audit C->>M: read (today's data) else a check fails J-->>J: alert, leave main unchanged C->>M: read (still yesterday's data) end ``` Table formats make publishing cheap. Iceberg supports branches and tags: named references to snapshots. A pipeline creates a branch, writes to it (in Spark, either by table name with a branch suffix or by setting spark.wap.branch on a table with write.wap.enabled), audits it, and then fast-forwards the main branch to the branch's snapshot. Only metadata changes, no data is copied, and readers of main switch to the new state at once. Tags mark specific snapshots, such as a month-end close, and can be kept longer than normal snapshot retention. Without branches, the same effect comes from a staging table followed by an atomic swap: replace the published table's contents, or rename tables, in one operation, if the engine and catalog support it. Some catalogs, such as Project Nessie, and tools such as lakeFS offer branches across many tables, so that several related tables are published together (checked October 2026). A lighter form exists in dbt: dbt build runs each model's tests before building models downstream, so a failed test stops bad data from spreading. But the failing model itself has already been written, and anyone reading it directly sees the bad data. For tables consumers read directly, true WAP needs a separate write location. WAP adds steps and storage, so it is worth it for published tables that people or applications depend on, not for every intermediate table. ## Key terms - Write-audit-publish (WAP) Write unseen, check, then publish atomically. - Audit The checks run between writing and publishing. - Branch A named, independent line of snapshots in a table. - Tag A named reference to one snapshot, with its own retention. - Fast-forward Moving a branch to a later snapshot of another branch. - Atomic swap Replacing a published table with a staged one in one operation. ## Example Publishing fct\_daily\_sales only after checks pass, with Iceberg branches in Spark SQL: ``` -- write ALTER TABLE lake.shop.fct_daily_sales CREATE BRANCH audit_20261028; INSERT INTO lake.shop.fct_daily_sales.branch_audit_20261028 SELECT ... ; -- the day's results -- audit: expect one row per product, revenue within 20% of the same weekday last week SELECT count(*), sum(revenue) FROM lake.shop.fct_daily_sales VERSION AS OF 'audit_20261028' WHERE sales_date = DATE '2026-10-28'; -- publish, only if the checks passed CALL lake.system.fast_forward('shop.fct_daily_sales', 'main', 'audit_20261028'); ALTER TABLE lake.shop.fct_daily_sales DROP BRANCH audit_20261028; ``` On a day when an upstream bug drops half of the orders, revenue falls by 48%, the audit fails, and the dashboards keep showing the previous day's figures with yesterday's date, stale but correct, rather than fresh and wrong. ## Pitfalls - Auditing after publishing. That is testing, not WAP; consumers already saw the data. - Checks that are too loose. An audit that always passes adds steps and no protection. - Forgotten branches. Each holds snapshots and files; drop branches after publishing and set retention. - Publishing related tables one by one. A fact and its dimension can be visible in inconsistent states; publish them together where the catalog allows. - Main moving during the audit. Fast-forward fails if another job, such as compaction, committed to main after the branch was created; schedule maintenance around WAP runs or rebuild the branch. - Engine support. Not every engine can write to branches; check before choosing the approach. ## Learn more - [Apache Iceberg, branching and tagging]() - [Apache Iceberg, Spark procedures (fast\_forward)]() ## Related [Backfill and replay]() [Data quality dimensions]() [Observability and SLOs]() [Environments and CI]()