How data arrives.
WHAT THE PATTERNS ARE
- Full extraction: reading everything each time
- Incremental extraction: reading only what changed
- Change data capture: reading the database's change log
- Event streaming: the source emitting events
- File delivery: files placed on a schedule
- Interface polling: requesting data periodically
WHAT FULL EXTRACTION SUITS
Small tables Sources with no reliable change indicator
WHAT IT COSTS
Load on the source, and time, growing with volume.
WHAT INCREMENTAL EXTRACTION REQUIRES
A reliable way to identify changes: a timestamp, an incrementing identifier, or a version.
WHAT GOES WRONG WITH TIMESTAMPS
Rows updated without the timestamp changing Clock differences Rows arriving with an earlier timestamp after extraction
WHAT TO DO ABOUT THAT LAST POINT
Overlap the window, and handle duplicates.
WHAT DELETES BREAK
Incremental extraction, since a deleted row simply stops appearing.
WHAT TO DO
Use change data capture, or reconcile periodically with a full comparison.
WHAT TO RECORD FOR EVERY EXTRACTION
What was extracted, when, and the watermark reached.
WHY
So a failure resumes correctly rather than duplicating or skipping.
