Controlling spend.
WHAT DRIVES COST
Compute for queries and processing Storage Data transfer Managed service charges
WHAT MAKES IT DIFFICULT
Cost is incurred by many people, in many places, with no immediate feedback.
WHAT TO ESTABLISH
Attribution: which team, pipeline or dashboard caused what.
HOW
Tagging and labelling of resources and queries.
WHAT TO MONITOR
Total spend against budget Cost per pipeline Most expensive queries Growth trend
WHAT CAUSES UNEXPECTED COST
Queries scanning whole tables Dashboards refreshing frequently against large tables Pipelines rebuilding everything rather than incrementally Development work on production-sized data Data retained with no retention policy
WHAT TO IMPLEMENT
Query scan limits Timeouts Materialised results for frequently repeated queries Partitioning and clustering
WHY MATERIALISATION HELPS
A dashboard querying a prepared table costs a fraction of one aggregating raw data.
WHAT TO SET FROM THE START
Budget alerts.
WHY THAT MATTERS HERE
Charges are in foreign currency, and a single careless query can be expensive.
WHAT TO REVIEW MONTHLY
Spend, by cause.
