Knowledgebase

Integrating Data From Several Sources Print

  • dataengineering, data, troubleshooting, guide, howto, solution, zillionkinghost, hosting
  • 0

Combining systems that disagree.

WHAT THE PROBLEMS ARE

The same entity identified differently in each system Different definitions of the same concept Conflicting values Different update frequencies Different levels of quality

WHAT ENTITY RESOLUTION IS

Determining that records in different systems refer to the same thing.

WHAT MAKES IT HARD

No shared identifier Names spelled differently Values entered inconsistently

WHAT APPROACHES EXIST

Exact matching on a shared key, where one exists Deterministic rules on combinations of attributes Probabilistic matching, scoring similarity Manual review of uncertain matches

WHAT TO BUILD

A mapping table recording which records correspond, and how that was determined.

WHY A TABLE

So decisions are visible, correctable, and stable.

WHAT TO DO WHEN SOURCES CONFLICT

Establish which is authoritative, per field.

WHY PER FIELD

One system may be authoritative for contact details and another for financial status.

WHAT TO DOCUMENT

That decision, explicitly.

WHAT TO RETAIN

The source values, alongside the resolved one.

WHY

So a wrong resolution can be traced and corrected.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot