Making results repeatable.
WHAT MUST BE REPRODUCIBLE
The dataset The transformations The code The environment The random state
WHAT MAKES DATASETS IRREPRODUCIBLE
Querying a table that has since changed No record of the query Filtering applied interactively
WHAT SOLVES IT
Snapshotting the training set, or querying a versioned table as of a timestamp.
WHAT TIME TRAVEL PROVIDES
Reading a table as it was at a point, where the format supports it.
WHY THAT IS VALUABLE
Training sets become reproducible without duplicating data.
WHAT TO RECORD FOR EVERY EXPERIMENT
The data reference The code version Parameters The environment Results
WHAT EXPERIMENT TRACKING TOOLS PROVIDE
That record, automatically.
WHAT NOTEBOOKS MAKE DIFFICULT
Reproducibility, since cells run out of order and state persists invisibly.
WHAT TO DO
Use them for exploration, and move anything that matters into versioned code.
WHAT TO NEVER SHIP
A notebook as a production pipeline.
WHAT TO TEST
That rerunning produces the same result.
