Making extraction dependable.
WHAT RELIABILITY REQUIRES
- Idempotence: running twice produces the same result
- Resumability: a failure restarts from where it stopped
- Observability: you know what ran and what it produced
- Isolation: one source failing does not stop others
WHAT IDEMPOTENCE REQUIRES
A deterministic key per record Loading that replaces rather than appends duplicates
WHY IT MATTERS SO MUCH
Retries are inevitable, and a non-idempotent pipeline duplicates data on every retry.
WHAT A WATERMARK IS
The point up to which data has been successfully processed.
WHERE TO STORE IT
Durably, updated only after successful processing.
WHY AFTER
Updating before means a failure skips data permanently.
WHAT TO DO ON PARTIAL FAILURE
Either complete or roll back, not leave half-loaded data.
WHAT STAGING PROVIDES
Loading to a temporary location, verifying, then promoting atomically.
WHY THAT PATTERN
Consumers never see partial data.
WHAT TO ALERT ON
Extraction not running Record counts far from expected Duration far from normal
WHY COUNTS SPECIFICALLY
Zero rows loaded successfully is the commonest silent failure.
