Reading changes from the source.
WHAT IT IS
Capturing inserts, updates and deletes by reading the database's transaction log.
WHAT IT PROVIDES
Every change, including deletes Minimal load on the source Low latency
WHY MINIMAL LOAD
It reads the log rather than querying tables.
WHAT IT REQUIRES
Database configuration enabling log access Appropriate permissions A tool that understands the log format
WHAT THE OUTPUT LOOKS LIKE
A stream of change events, each with the operation, the old values and the new.
WHAT TO DO WITH THEM
Apply them to a target, or retain them as a history.
WHAT RETAINING THEM PROVIDES
A complete record of how data changed, which the source may not keep.
WHAT INITIAL LOAD REQUIRES
A snapshot, then changes from a consistent point.
WHY THAT IS DELICATE
Missing changes between the snapshot and the stream loses data silently.
WHAT TOOLS HANDLE
That coordination, which is why building it yourself is inadvisable.
WHAT TO MONITOR
Lag behind the source Whether the log is being retained long enough
WHAT HAPPENS IF LAG EXCEEDS LOG RETENTION
The stream cannot resume, and a full reload is required.
