Engineering practice for data.
WHAT TO VERSION CONTROL
Transformation code Pipeline definitions Configuration Schema expectations Tests
WHAT TO NEVER KEEP OUTSIDE IT
Anything production depends on.
WHY
Undocumented manual changes are how pipelines become unmaintainable.
WHAT ENVIRONMENTS TO MAINTAIN
Development, for building A staging environment, with representative data Production
WHAT REPRESENTATIVE MEANS
Similar structure and enough volume to expose problems, without sensitive values.
HOW TO PRODUCE THAT
Sampling and masking from production.
WHAT TO RUN ON EVERY CHANGE
Tests A build against development data
WHAT TO REVIEW
Changes affecting business definitions, with whoever owns them.
WHY
A transformation change silently alters reported figures.
WHAT TO DOCUMENT IN A CHANGE
What changed, and whether historical figures are affected.
WHAT TO COMMUNICATE
Changes to metric definitions, before they take effect.
WHAT TO AVOID
Editing production queries directly Logic living only in a reporting tool Untracked scheduled jobs on someone's machine
