Knowing they worked.
WHAT TO MONITOR
Whether each run completed Duration, against normal Row counts produced
Freshness: how old the newest data is
Quality test results Cost, where usage-billed
WHY FRESHNESS MATTERS MOST TO CONSUMERS
A dashboard showing stale data without saying so is worse than one that is unavailable.
WHAT TO SURFACE TO USERS
When the data was last updated.
WHAT TO ALERT ON
Failure Freshness exceeding a threshold Row counts outside expected bounds Quality tests failing Duration far above normal
WHY ROW COUNTS SPECIFICALLY
Successful runs producing zero or half the usual rows are the commonest silent failure.
WHAT TO SET BOUNDS FROM
Historical values, rather than fixed guesses.
WHAT NOT TO ALERT ON
Anything nobody will act on.
WHAT TO REVIEW
Alerts that fired and were ignored.
WHAT LINEAGE PROVIDES
Knowing what a table depends on, and what depends on it.
WHY IT MATTERS OPERATIONALLY
It answers who is affected when something fails, immediately.
WHAT TO GENERATE IT FROM
The transformation definitions, automatically.
WHAT TO COMMUNICATE ON FAILURE
Who is affected, and when it will be fixed.
