Knowledgebase

Transformation Frameworks Print

  • dataengineering, data, guide, howto, solution, zillionkinghost, hosting, support
  • 0

Managing transformation as code.

WHAT THEY PROVIDE

Transformations defined as select statements, materialised as tables or views Dependencies resolved automatically Testing built in Documentation generated Version control of the whole model

WHY DEPENDENCY RESOLUTION MATTERS

Models reference each other, and the framework determines the correct build order.

WHAT THAT REPLACES

Manually ordered scripts, which break whenever a dependency is added.

WHAT MATERIALISATION OPTIONS EXIST

Views, computed at query time Tables, rebuilt fully Incremental tables, updated with new data only Ephemeral, inlined into consumers

WHAT TO USE WHEN

Views for small, cheap logic Tables where rebuild is affordable Incremental where rebuild is not

WHAT INCREMENTAL REQUIRES

A reliable way to identify new or changed rows Handling of late-arriving data Periodic full rebuild to correct drift

WHY THE PERIODIC REBUILD

Incremental logic accumulates small errors over time.

WHAT TESTING PROVIDES

Assertions about uniqueness, nulls, relationships and accepted values.

WHAT TO TEST ON EVERY MODEL

That the key is unique and not null.

WHY THAT ONE ALWAYS

Duplicate keys silently inflate every downstream aggregate.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot