08 Data science and ML¶

How data becomes analysis, experiments and models: the data science workflow, working with lake data from Python, controlled experiments, features and training data, and taking models to production and keeping them accurate.

Pages, in reading order¶

1. The data science workflow: How a business question becomes a trustworthy answer: framing the question, exploring the data, testing explanations, and deciding whether the result is a one-off analysis, a recurring report or a model.
2. Python on the lakehouse: Working with lakehouse tables from Python: notebooks for exploration, dataframe libraries for analysis and feature work, and the choices that decide where the computation runs.
3. Experimentation and A/B testing: Measuring the effect of a change by randomly splitting users into groups that see different versions, and comparing their outcomes.
4. Feature engineering and feature stores: Feature engineering turns raw data into the inputs a model learns from; a feature store keeps those definitions in one place and provides the same values for training and for prediction.
5. Training data management: Building, versioning and governing the datasets models are trained and evaluated on: what goes in, how it is split, how labels are made, and whether the data may be used for training at all.
6. The ML lifecycle and MLOps: The path from a training dataset to a model whose predictions people and systems use, and MLOps, the practices that make that path repeatable, traceable and safe to change.
7. Model serving and monitoring: Running a trained model to produce predictions, in batches or per request, releasing new versions safely, and watching whether its predictions stay accurate as the world changes.

Previous and next section¶

07 Operations · 09 Generative AI