Data for model building.
WHAT DISTINGUISHES IT FROM ANALYTICAL DATA
It must reflect what will be available at prediction time, not what is known afterwards.
WHAT LEAKAGE IS
Information in training data that would not be available when predicting.
WHY IT IS THE MOST DAMAGING ERROR
The model performs excellently in testing and fails entirely in production.
WHAT CAUSES IT
Features computed from the future relative to the prediction point Target information encoded in a feature Data cleaned using statistics from the whole dataset
WHAT POINT-IN-TIME CORRECTNESS MEANS
Every feature reflecting only what was known at the prediction moment.
WHAT THAT REQUIRES
Data models recording when each value became known.
WHY THAT IS A DATA ENGINEERING PROBLEM
Most warehouses store current state, which cannot answer it.
WHAT TO BUILD
History with validity periods, for anything used as a feature.
WHAT TO BE CAREFUL WITH IN SPLITTING
Random splits where data is ordered in time Related records appearing on both sides
WHAT TO SPLIT BY
Time, for anything predicting the future.
WHAT TO DOCUMENT
The exact query producing the training set, and when it ran.
WHY
Otherwise results cannot be reproduced.
