Moving and transforming data.
WHAT PIPELINES DO
Move data from sources, transform it, and load it into a destination.
WHAT OPTIONS EXIST
Managed pipeline services, for substantial processing Scheduled queries within the analytical database Functions triggered by events Managed transfer services for common sources
WHAT TO CHOOSE
The simplest that handles your volume.
WHY
Substantial pipeline frameworks carry operational complexity most workloads do not need.
WHAT MANY ORGANISATIONS ACTUALLY NEED
Loading data into the analytical database, and transforming it there with scheduled queries.
WHAT TO ESTABLISH
Volume Frequency Whether processing must be immediate or can be batched
WHY THAT LAST QUESTION
Batch processing is substantially simpler and cheaper.
WHAT TO DESIGN FOR
Reruns, since pipelines fail and must be repeated.
WHAT THAT REQUIRES
Processing that produces the same result when repeated.
WHAT TO PARTITION BY
Date, almost always.
WHY
It makes reprocessing a single day possible, and controls query cost.
WHAT TO MONITOR
Completion, freshness, and row counts.
WHAT TO ALERT ON
A pipeline not completing, and counts outside expected ranges.
