When things run.
WHAT TIME-BASED SCHEDULING ASSUMES
That inputs are ready by the scheduled time.
WHY THAT ASSUMPTION FAILS
Sources are late, and the pipeline processes incomplete data silently.
WHAT SENSORS PROVIDE
Waiting for a condition: a file arriving, a table updated, an external signal.
WHAT THAT ACHIEVES
Running when data is ready rather than when the clock says so.
WHAT TO ALWAYS ADD TO A SENSOR
A timeout, and an alert.
WHY
Waiting forever is indistinguishable from working.
WHAT CROSS-PIPELINE DEPENDENCIES REQUIRE
One pipeline knowing another completed.
WHAT TO AVOID
Coupling by scheduling one an hour after another and hoping.
WHAT TO USE INSTEAD
Explicit signalling of completion.
WHAT CATCH-UP BEHAVIOUR IS
Running missed intervals after downtime.
WHAT TO DECIDE
Whether that is wanted.
WHY
A pipeline down for a week may attempt a week of runs simultaneously.
WHAT TO CONFIGURE
Concurrency limits.
WHAT TO CONSIDER ABOUT TIME ZONES
That schedules and data periods may differ, and daylight changes affect some regions.
WHAT TO SET SCHEDULES IN
One time zone, consistently.
