The summary.
LOAD RAW, THEN TRANSFORM IN PLACE
Transformation is far easier to test, version and correct than extraction. Never discard information during extraction — you cannot recover what was never stored.
TEST KEY UNIQUENESS AND NON-NULLITY ON EVERY MODEL
Duplicate keys silently inflate every downstream aggregate, and nothing else catches it.
KEEP STAGING MECHANICAL
Renaming, casting, null handling, deduplication — nothing else. Then it never requires business discussion.
INCREMENTAL LOGIC DRIFTS FROM WHAT A FULL REBUILD WOULD PRODUCE
Rebuild fully on a schedule and compare. Measure how late data actually arrives rather than guessing the lookback window.
SHUFFLES DOMINATE DISTRIBUTED PROCESSING PERFORMANCE
Filter before joining, broadcast small tables, and handle skewed keys separately — one oversized partition is what fails on memory.
SPECIFY TYPES AND DATE FORMATS EXPLICITLY
Identifiers read as numbers lose leading zeros, and day-month ordering is local while tools frequently default to month-day.
RECONCILIATION AGAINST A KNOWN FIGURE CATCHES WHAT STRUCTURAL TESTS MISS
Investigate disagreement before publishing, not after someone notices.
