Model serving and monitoring¶

Running a trained model to produce predictions, in batches or per request, releasing new versions safely, and watching whether its predictions stay accurate as the world changes.

Before reading

The ML lifecycle and MLOps · Low-latency serving stores

Why it matters¶

A model is useful only while its predictions are used and correct. Serving decides how fast and how fresh predictions are, and how much infrastructure they need. Monitoring catches the slow failure of every model: the world it learned from changes, its accuracy drops, and nothing errors.

How it works¶

There are two ways to serve.

Batch serving scores many entities on a schedule and writes predictions to a table: every customer's churn risk every night, every product's demand forecast every morning. It uses the same engines and orchestration as other pipelines, is easy to rerun, and is enough whenever predictions can be a few hours old.

Online serving runs the model behind an API and scores one request at a time, within milliseconds: fraud risk at checkout, recommendations as a page loads. It needs a model server that is always running, features available with low latency from an online store or computed from the request, and the same reliability practices as any production service. Streaming serving sits between: scoring events as they arrive in a stream processor.

Releasing a new version safely uses patterns from software. A shadow deployment runs the new model alongside the current one on live traffic, logging its predictions without using them, so they can be compared. A canary release sends a small share of traffic to the new model and widens it if metrics hold. An A/B test splits traffic to measure the business effect. Rollback is switching the registry's production pointer back to the previous version; it should take minutes and be tested.

Monitoring covers three layers. Service health: latency, errors, throughput for online models; job success and duration for batch. Data drift: the inputs change, such as a new product category, a different customer mix, or a feature broken upstream that is suddenly null. Comparing input distributions with those in training, using measures such as the population stability index, catches these early. Model quality: accuracy measured when true outcomes become known, by joining predictions with actuals. Outcomes can lag by days or months, so drift checks serve as early warning.

flowchart TB
  subgraph Serve
    direction LR
    B[Batch: nightly<br/>scoring to table] ~~~ O[Online: API<br/>per request]
  end
  Serve --> P[(Prediction log:<br/>inputs, output,<br/>model_version)]
  P --> D[Drift: inputs vs<br/>training distribution]
  P --> Q[Quality: join with<br/>outcomes when known]
  D & Q --> A{Degraded?}
  A -->|yes| RT[Alert owner,<br/>retrain or roll back]

Every prediction is logged with its inputs, output, model version and time, as a table. The log is what makes drift, quality, debugging and audits possible. Retraining can be scheduled, or triggered when drift or quality cross a threshold, after which the new model goes through the registry and release steps again.

Key terms¶

Batch serving: scoring on a schedule and writing predictions to a table.
Online serving: scoring per request behind an API, with low latency.
Shadow deployment: running a new model on live traffic without using its predictions.
Canary release: sending a small share of traffic to a new version first.
Data drift: change in the distribution of a model's inputs.
Concept drift: change in the relationship between inputs and outcomes.
Prediction log: a record of every prediction with inputs and model version.

Example¶

The shop scores fraud risk at checkout. Batch scoring would be too late, so the model runs behind an API called by the checkout service, with features from the online store: orders in the last hour, account age, and whether the delivery address is new. The service has an SLO of 50 milliseconds at the 99th percentile.

A new version runs in shadow for a week. Five weeks later, once chargebacks for those orders have arrived, it would have caught 12% more confirmed fraud while blocking 0.1 points more legitimate orders, within the agreed false-decline budget, so it goes to a 10% canary, then to all traffic.

In November, the share of orders with a new delivery address doubles during a gifting campaign. Drift monitoring flags the feature within a day. Chargebacks, the true outcome, arrive weeks later, but the team reviews the model's behaviour on gifting orders at once and adjusts the decision threshold for the campaign period.

Pitfalls¶

Online serving when batch would do. It costs much more to build and run.
No prediction log. Drift, quality and incidents cannot be investigated.
Monitoring only the service. A model can respond quickly and be wrong.
Waiting for outcomes to detect problems. Use drift as an early warning.
Rollback never tested. The first real rollback takes hours instead of minutes.

Learn more¶

Google, Rules of Machine Learning, monitoring and training-serving skew: https://developers.google.com/machine-learning/guides/rules-of-ml
Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapters 7 and 8.

Stream processing · Observability and SLOs · Feature engineering and feature stores · Experimentation and A/B testing