Correcting history.
WHEN IT IS NEEDED
A bug is found A source is corrected A definition changes New logic must be applied to history
WHAT MAKES IT POSSIBLE
Raw data retained Tasks parameterised by period Idempotent processing
WHAT MAKES IT IMPOSSIBLE
Transformation applied at extraction Tasks using the current date implicitly Non-idempotent loading
WHAT TO PLAN BEFORE RUNNING ONE
How long it will take What load it places on systems Whether consumers see partial results during it
WHAT TO DO ABOUT THAT LAST POINT
Build into a separate location, then switch.
WHAT TO LIMIT
Concurrency, so the backfill does not starve normal operation.
WHAT TO DO FOR VERY LARGE BACKFILLS
Process in ordered batches, with progress recorded.
WHY ORDERED
So an interruption resumes rather than restarting.
WHAT TO VERIFY AFTER
That totals match expectations for reprocessed periods That nothing outside the intended range changed
WHAT TO COMMUNICATE
That historical figures have changed, and why.
WHY
Someone has a report with the old numbers.
