07 Operations#
How a platform is run day to day: scheduling and coordinating pipelines, moving changes safely from development to production, knowing when something is wrong and what "on time" means, keeping costs under control, recovering when things go badly wrong, declaring infrastructure as code, and moving from one platform to another.
Before starting
04 Modelling and transformation and 05 Governance and security, in particular dbt, Testing data pipelines and Data quality dimensions. The optional pages on cost and disaster recovery also build on 02 Storage; Infrastructure as code and Platform migration need only the pages of this section.
Pages, in reading order#
-
Orchestration decides when each step of the platform's pipelines runs, in what order, and what happens when a step fails.
-
Separate environments, such as development, test and production, let changes to pipelines be built and checked before they affect real consumers; continuous integration (CI) runs those checks automatically on every proposed change.
-
Observability is being able to tell, from the signals a system emits, whether it is healthy and why it is not; service level objectives (SLOs) state in numbers what healthy means for consumers.
-
4 Cost and FinOps Optional
Understanding where a data platform's money goes, attributing it to the teams and products that cause it, and reducing it safely, as a shared routine of engineering and finance called FinOps.
-
5 Disaster recovery Optional
Being able to restore the platform's data and service after a serious failure, such as a deleted table, corrupted data, a lost catalog or an unavailable region, within agreed limits on lost data and downtime.
-
6 Infrastructure as code Optional
Declaring the platform's infrastructure, such as buckets, catalogs, clusters, roles and permissions, in version-controlled files, and letting tools create and update it to match, instead of configuring it by hand.
-
7 Platform migration Optional
Moving data, pipelines and consumers from one platform to another, such as from a legacy warehouse to a lakehouse, without breaking what people depend on.