Feature engineering and feature stores¶

Feature engineering turns raw data into the inputs a model learns from; a feature store keeps those definitions in one place and provides the same values for training and for prediction.

Before reading

History and slowly changing dimensions · Python on the lakehouse

Why it matters¶

Most of a model's quality comes from its features, and most production failures of models come from features too: values computed differently in training and in production, or training examples that accidentally contain information from the future. Feature engineering is where data platform skills, such as modelling history, handling late data and testing, decide whether a model works.

How it works¶

A feature is a measurable property of the thing being predicted, such as a customer's number of orders in the last 90 days, the average basket value, or days since the last order. Features are computed from clean and gold tables, mostly as aggregates over time windows, per entity: customer, product, session.

Point-in-time correctness is the central rule. Training data consists of examples, each with a time: "customer c-88, as of 1 March 2026, did they churn in the next 90 days?". Every feature for that example must use only data that existed before 1 March. Joining to the current value of a feature instead, such as today's order count, gives the model information it will not have when predicting, called leakage. Models trained with leakage look excellent in testing and fail in use. Point-in-time joins, as for slowly changing dimensions, match each example with the latest feature value at or before its time.

Late data adds a subtlety: a feature computed for 1 March, using data that arrived on 3 March, uses information unavailable on 1 March. Strict pipelines record when each feature value became available, not only the period it describes.

Training-serving skew is the second classic failure: the code that computes features for training differs from the code used at prediction time, perhaps Python in a notebook and Java in a service. The model then sees inputs at prediction time that do not match what it learned from. The defence is one definition used for both.

A feature store provides that. It holds feature definitions as code, computes and stores feature values over time in an offline store, usually lakehouse tables, for building training sets with point-in-time joins, and copies the latest values to an online store, a low-latency key-value database, for real-time prediction. It also catalogs features so teams can find and reuse them, with owners and lineage. Feast is a common open-source feature store; ML platforms include their own.

flowchart LR
  G[(Clean and gold<br/>tables)] --> D[Feature definitions<br/>as code]
  D --> OFF[(Offline store:<br/>feature values over time)]
  OFF -->|point-in-time join| TR[Training set]
  OFF -->|latest values| ON[(Online store:<br/>key-value)]
  ON --> SV[Real-time prediction]
  OFF --> BS[Batch prediction]

A feature store is not always needed. A team with batch predictions only can keep feature tables in the lakehouse, built by dbt with point-in-time logic, tested and documented like other tables. A feature store earns its cost when features must be served in real time, or when many teams share features.

Key terms¶

Feature: an input variable to a model.
Entity: the thing features describe, such as a customer or product.
Point-in-time join: matching each example with feature values as they were at its time.
Leakage: features containing information not available at prediction time.
Training-serving skew: a difference between features at training and at prediction time.
Offline and online store: historical feature values for training, and latest values for low-latency serving.

Example¶

Customer features for a churn model, as a table of values over time, built daily:

-- shop.feat_customer_daily: one row per customer per day
SELECT c.customer_id,
       d.date_day                                           AS as_of_date,
       count(o.order_id)                                    AS orders_90d,
       avg(o.amount)                                        AS avg_order_value_90d,
       date_diff('day', CAST(max(o.ordered_at) AS date), d.date_day) AS days_since_last_order_90d
FROM shop.dim_date AS d
CROSS JOIN shop.clean_customers AS c
LEFT JOIN shop.clean_orders AS o
  ON o.customer_id = c.customer_id
 AND o.ordered_at <  d.date_day
 AND o.ordered_at >= d.date_day - INTERVAL '90' DAY
WHERE d.date_day BETWEEN DATE '2024-01-01' AND current_date
GROUP BY c.customer_id, d.date_day;

This filters on order time only; a strict version also requires each row's load time to be before as_of_date, which needs the _ingested_at column carried into clean_orders. avg_order_value_90d is NULL for customers with no orders in the window rather than 0, so models must handle missing values explicitly instead of reading "no orders" as "orders worth 0". days_since_last_order_90d is also NULL beyond 90 days, and those are the likeliest churners; a full days-since-last-order feature comes from a separate, unwindowed max(ordered_at) before as_of_date.

The training set joins each labelled example, customer and date, to the row with the same as_of_date, so no order after that date is counted. Nightly batch scoring uses the latest as_of_date from the same table, so training and scoring share one definition. The cross join is for illustration; in practice the table is built incrementally, one day at a time.

Pitfalls¶

Joining training examples to current feature values. The model learns from the future.
Two implementations of one feature. Training and serving disagree; keep one definition.
Ignoring when data arrived. Late-loaded data leaks into past feature values.
A feature store before it is needed. Lakehouse feature tables serve batch use well.
Features without owners or tests. They break silently when upstream tables change.

Learn more¶

Feast documentation, concepts: https://docs.feast.dev/getting-started/concepts
Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapter 5, feature engineering.

Training data management · Model serving and monitoring · Low-latency serving stores · Data contracts