Platform migration#
Optional
Moving data, pipelines and consumers from one platform to another, such as from a legacy warehouse to a lakehouse, without breaking what people depend on.
Before reading
Why it matters#
Migrations are among the riskiest projects a data team runs. They touch every pipeline and every consumer, take months, and are judged on one thing: whether the numbers people rely on stay the same. Most failed migrations fail not on technology but on scope, on unknown consumers, and on differences nobody noticed until a report changed.
How it works#
A migration runs in stages.
Inventory comes first. List every source, pipeline, table, report, export and consumer on the old platform, using query logs to see what is actually used. Usage data usually shows that a large share of tables is never read; those are retired rather than migrated. Lineage shows what depends on what, which decides the order.
Translation moves pipelines to the new platform. SQL dialects differ, so queries are converted, by tools or by hand, and tested. This is also the moment to fix design problems, but changing logic and platform at the same time makes differences hard to explain; changes in meaning are better made before or after.
Parallel running is the core of a safe migration. Old and new pipelines run side by side on the same inputs for a period, and their outputs are compared every day: row counts, key metrics, and full comparisons (data diffs) of important tables. Differences are explained and fixed until the outputs agree, or the remaining difference is understood and accepted, such as a rounding change.
Cutover moves consumers, one group at a time: dashboards, exports, applications, data science. Replacing the old platform piece by piece, while both run, is known as the strangler fig pattern. Each consumer gets a date, a contact and a way back. Published tables keep their names and contracts where possible, so that consumers change a connection, not their logic.
Decommissioning ends the project. Once no consumer reads the old platform, which query logs can confirm, old pipelines stop and the platform is switched off after a final archive. Migrations that never decommission pay for two platforms indefinitely.
flowchart TB
I[Inventory:<br/>usage, lineage] --> T[Translate pipelines<br/>by dependency order]
T --> P[Parallel run:<br/>daily comparison]
P -->|outputs agree| C[Cut over consumers<br/>group by group]
C --> D[Decommission:<br/>no reads left]
P -->|differences| THistory needs a plan. Historical data is either copied once from the old platform, or rebuilt on the new one from raw sources if they are kept. Rebuilding tests the new pipelines but may not reproduce old numbers exactly, for example if source data has since changed; the choice is agreed with consumers.
Communication is part of the work: a published plan, a date per consumer group, a list of known differences, and a single place to report problems.
Key terms#
- Inventory The list of assets and consumers on the old platform, with usage.
- Dialect translation Converting SQL and functions from one engine to another.
- Parallel run Running old and new pipelines side by side and comparing outputs.
- Reconciliation Comparing outputs to find and explain differences.
- Cutover Moving consumers from the old platform to the new one.
- Decommissioning Switching off the old platform once nothing uses it.
Example#
The shop moves daily sales reporting from a legacy cloud warehouse to its lakehouse.
- Inventory: 640 tables; query logs show 210 read in the last 90 days. 430 are archived, not migrated.
- Translation: fct_daily_sales and its upstream models are rewritten in dbt for Trino; date functions and decimal rounding need manual changes.
- Parallel run: for four weeks, both platforms build fct_daily_sales; a daily job compares revenue per day and category. Two differences appear: refunds were netted on different dates, and one category mapping was out of date in the old system. The first is fixed; the second is accepted, because the new platform is correct, and announced as a known difference.
- Cutover: finance dashboards move first, with the analyst who owns them; exports to the ordering system move a week later.
- Decommissioning: after 30 days with no reads on the old tables, the old pipelines are stopped and the warehouse is archived and switched off.
Pitfalls#
- Migrating everything. Unused tables cost time to move and test; retire them instead.
- Changing logic and platform together. Differences can no longer be explained.
- Skipping the parallel run. The first comparison happens when a consumer complains.
- Unknown consumers. Exports and scripts nobody listed break on cutover day; use access logs.
- Never decommissioning. Two platforms cost twice as much, and people keep using the old one.