Processing only what changed.
WHY IT MATTERS
Reprocessing everything becomes infeasible as volume grows.
WHAT IT REQUIRES
Identifying what is new or changed Merging it into the existing result correctly Handling data arriving for periods already processed
WHAT THE STRATEGIES ARE
Append only, for immutable events Merge on a key, replacing matched rows Delete and reinsert a partition
WHAT PARTITION REPLACEMENT SUITS
Time-partitioned data where a whole day may be reprocessed.
WHY IT IS ROBUST
It is idempotent, and handles late data within the replaced window.
WHAT A LOOKBACK WINDOW IS
Reprocessing recent periods each run, to capture late arrivals.
WHAT TO SET IT TO
Longer than the observed lateness.
WHAT TO MEASURE
How late data actually arrives.
WHY
The window is otherwise guesswork.
WHAT DRIFT MEANS HERE
Incremental results diverging from what a full rebuild would produce.
WHAT CAUSES IT
Late data outside the window Source corrections Logic changes applied only to new rows
WHAT TO DO
Rebuild fully on a schedule, and compare.
WHAT TO ALERT ON
A difference between incremental and full results.
