The summary.
A PIPELINE THAT RUNS SUCCESSFULLY AND PRODUCES WRONG NUMBERS IS WORSE THAN ONE THAT FAILS
Because nobody notices. The central responsibility is that the numbers people act on are correct.
ASK WHETHER ONE MACHINE COULD DO THIS
The answer is usually yes, and distributed tooling is frequently adopted for volumes a single machine would handle easily.
Measure the actual volume before choosing an architecture.
CHOOSE BATCH UNLESS LATENCY GENUINELY REQUIRES OTHERWISE
Most requirements described as real-time are satisfied by running every few minutes. Ask what decision changes if the data is an hour old.
ANALYTICS DOES NOT BELONG IN THE PRODUCTION DATABASE
Analytical queries scan large volumes and compete with the application.
NEVER ASSUME A SOURCE WILL KEEP ITS CURRENT STRUCTURE
Build so change is detected at ingestion rather than discovered downstream by someone reading a wrong number.
MAINTENANCE IS THE MAJORITY OF A PIPELINE'S TOTAL EFFORT
Build for whoever maintains it, including future you. Document where data came from and what was done to it.
