The summary.
DELETES BREAK INCREMENTAL EXTRACTION
A deleted row simply stops appearing. Use change data capture, or reconcile periodically with a full comparison.
Store the watermark only after successful processing — updating before means a failure skips data permanently.
RETRIES ARE INEVITABLE, SO INGESTION MUST BE IDEMPOTENT
Otherwise every retry duplicates data. Stage, verify, then promote atomically so consumers never see partial loads.
STORE RAW RESPONSES AND FILES BEFORE PARSING
So parsing can be corrected without re-extracting, which is frequently impossible.
READ DELIMITED FILES BY HEADER NAME, NEVER BY POSITION
Column order changes load the wrong data into the wrong columns, silently. Fail loudly on a missing column rather than filling with nulls.
Specify the encoding explicitly rather than relying on detection.
ZERO ROWS LOADED SUCCESSFULLY IS THE COMMONEST SILENT FAILURE
Alert on record counts, not only on errors.
A SOURCE CHANGING MEANING WITHOUT CHANGING STRUCTURE IS THE WORST CASE
Nothing detects it. Most schema surprises are a communication failure — establish who tells you before a source changes.
