Coordinating work.
WHAT AN ORCHESTRATOR DOES
Defines tasks and their dependencies Schedules them Executes them in the correct order Retries failures Reports what happened
WHY IT IS NECESSARY
Pipelines involve many steps with dependencies, and running them by schedule alone breaks whenever timing varies.
WHAT SCHEDULING ALONE CANNOT DO
Ensure a step runs only after its input is actually ready.
WHAT A DEPENDENCY GRAPH PROVIDES
Explicit ordering, and parallelism where steps are independent.
WHAT THE COMMON ORCHESTRATORS PROVIDE
Definition in code A view of runs and their state Retries and alerting Backfilling
WHAT BACKFILLING IS
Running a pipeline for past periods.
WHY IT MATTERS
Bugs are found after the fact, and history must be corrected.
WHAT IT REQUIRES
Tasks parameterised by period, rather than assuming today.
WHAT TO NEVER DO
Write tasks that implicitly use the current date.
WHY
They cannot be backfilled, and a delayed run processes the wrong period.
WHAT TO PASS EXPLICITLY
The period each run is processing.
WHAT TO MAKE EVERY TASK
Idempotent, so rerunning is safe.
