# Observability and SLOs Observability is being able to tell, from the signals a system emits, whether it is healthy and why it is not; service level objectives (SLOs) state in numbers what healthy means for consumers. > [!NOTE] > > **Before reading** > > [Orchestration]() ยท [Data quality dimensions]() ## Why it matters Data failures are often silent. A job succeeds but loads half the rows; a table is updated but three hours late; a query that took a minute now takes forty. Without observability, consumers notice first. Without SLOs, nobody agrees what counts as late or broken, so every incident is argued and nothing is prioritised. ## How it works A data platform has two kinds of thing to observe. Pipeline observability covers the jobs and systems: did each task run, succeed, and how long did it take; how far behind are Kafka consumers and CDC connectors; are engines, catalogs and storage available and how busy are they. The signals are metrics, such as durations, lag and error counts; logs from each task; and, increasingly, traces that follow one run across components. Data observability covers the data itself: is each table fresh, is its volume normal, are its distributions and null rates normal, has its schema changed. These are the quality dimensions measured continuously. Lineage connects the two: when clean\_orders is late, lineage shows that the cause is the stopped CDC connector upstream, and that fct\_daily\_sales and three dashboards downstream are affected. An SLO turns a consumer need into a measurable target over a period. It has three parts. A service level indicator (SLI): what is measured, such as the time fct\_daily\_sales finished updating. A target: the condition that counts as good, such as finished by 07:00. An objective: how often the target must be met, such as on 98% of days over 90 days. A service level agreement (SLA) is a promise with consequences, usually to external customers; SLOs are the internal targets, set stricter than any SLA. The difference between 100% and the objective is the error budget: with 98% over 90 days, 2% of 90 days is 1.8 days, so one late day fits in the budget and a second does not. While budget remains, the team can take risks, such as large changes; when it is spent, reliability work comes first. ``` flowchart TB subgraph Signals direction LR M[Metrics:
durations, lag, errors] L[Logs per task] D[Data checks:
freshness, volume] end Signals --> SLI[SLI: time fct_daily_sales
finished updating] SLI --> SLO{SLO: by 07:00
on 98% of days} SLO -->|at risk| AL[Alert owner] SLO --> EB[Error budget
remaining] ``` Alerts should fire on symptoms consumers would notice, or on causes that will soon produce them, and only when someone must act. An alert at 05:30 that fct\_daily\_sales will miss 07:00, because its upstream load is two hours behind, is useful; an alert for every retried task is noise. Each alert has an owner, a severity and a short runbook: what it means and what to check first. SLOs are set per published dataset with its consumers, written into its contract, and reviewed when needs change. Not every table needs one: intermediate tables inherit urgency from the published tables that depend on them. ## Key terms - Observability Understanding a system's state from the signals it emits. - Metric, log, trace Numeric measurements over time, event records, and the path of one request or run across components. - Service level indicator (SLI) The measured quantity, such as completion time. - Service level objective (SLO) A target for an SLI over a period, such as 98% of days. - Service level agreement (SLA) A promise to customers, with consequences if broken. - Error budget The amount of failure the SLO allows. - Runbook Written steps for responding to an alert. ## Example The finance team needs daily sales by 07:00 for their morning meeting. The SLO in the contract of fct\_daily\_sales: | Part | Value | | --- | --- | | SLI | Time the daily build of fct\_daily\_sales completed with tests passed | | Target | By 07:00 Europe/Berlin (the contract's timeOfAvailability) | | Objective | 98% of days over a rolling 90 days | | Alert | At 05:30, if the newest `_ingested_at` in raw\_orders is more than 30 minutes old, or the previous day is not yet marked complete (the build needs about 90 minutes) | | Owner | orders-data on-call | In October the build missed 07:00 twice, both because the CDC connector stopped overnight. That overspent the 90-day budget of 1.8 late days, so the team prioritised a lag alert on the connector over new features. The next month, the connector stopped again; the 01:15 lag alert let the on-call engineer restart it, and the build finished at 06:20. ## Pitfalls - Monitoring jobs but not data. A successful job that loaded nothing raises no alarm. - Alerting on everything. People learn to ignore alerts; alert only where action is needed. - SLOs without consumers. A target nobody depends on is arbitrary; set them with the people who use the data. - 100% objectives. They leave no room for change and are broken by the first incident. - No lineage in incidents. Time is lost finding the cause and the affected consumers. ## Learn more - [Google, Site Reliability Engineering, chapter 4, service level objectives]() - [OpenTelemetry documentation, observability primer]() ## Related [Data contracts]() [Catalog and lineage]() [Cost and FinOps]() [Governance roles]()