Knowledgebase

Handling Schema Changes Print

  • dataengineering, data, guide, howto, solution, zillionkinghost, hosting, support
  • 0

When sources change shape.

WHAT CHANGES OCCUR

Columns added Columns removed Columns renamed Types changed Meanings changed without any structural change

WHY THAT LAST ONE IS WORST

Nothing detects it, and every downstream number becomes wrong.

WHAT TO DETECT AUTOMATICALLY

The schema differing from what is expected.

WHAT TO DO ON DETECTION

Alert, and decide deliberately.

WHAT NOT TO DO

Automatically adopt the new schema without review.

WHY

An accidental change in the source silently propagates.

WHAT ADDITIVE CHANGES ALLOW

Proceeding, with the new column ignored or adopted.

WHAT REMOVALS AND TYPE CHANGES REQUIRE

A decision, since downstream consumers break.

WHAT A CONTRACT PROVIDES

An agreement with the source owner about structure and meaning.

WHY THAT MATTERS ORGANISATIONALLY

Most schema surprises are a communication failure, not a technical one.

WHAT TO ESTABLISH

Who to tell before changing a source.

WHAT TO VERSION

Your schema expectations, in version control.

WHAT TO TEST

That the pipeline fails clearly on an unexpected schema, rather than producing wrong output.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot