The largest part of the work.
WHAT DATA PREPARATION INVOLVES
Collecting Cleaning Handling missing values Encoding categories Scaling numerical values Splitting into training, validation and test sets
WHY THE SPLIT MATTERS MOST
A model evaluated on data it trained on tells you nothing.
WHAT TO NEVER DO
Let information from the test set influence training.
WHAT THAT LOOKS LIKE
Scaling using statistics computed across everything Selecting features using the whole dataset Any preparation step fitted before splitting
WHAT THAT IS CALLED
Leakage, and it produces models that appear excellent and fail in production.
WHAT TO DO
Split first, then fit every transformation on the training set only.
WHAT TO CHECK IN THE DATA
Class balance Outliers Duplicates Values that could not occur Whether the data reflects the situation the model will face
WHY THAT LAST POINT
A model learns the data it was given, including its biases.
WHAT TO DOCUMENT
Where the data came from, when, and what was done to it.
