Where data comes from.
WHAT THE COMMON SOURCES ARE
Application databases Event streams from applications Third-party service interfaces Files delivered on a schedule Manual uploads Log files Sensor and device data
WHAT EACH BRINGS
- Databases: structured, but coupled to the application's model
- Interfaces: structured, but rate limited and subject to change
- Files: simple, but inconsistent
- Manual uploads: unavoidable, and always problematic
WHY MANUAL UPLOADS ARE PROBLEMATIC
Formats change, columns move, values are typed by hand.
WHAT TO DO ABOUT THEM
Validate strictly, and reject clearly rather than accepting bad data.
WHAT TO ESTABLISH FOR EVERY SOURCE
Who owns it How it changes, and whether you are told What its update frequency is What its quality is actually like Whether it can be re-read
WHY THAT LAST POINT MATTERS
Sources you cannot re-read must be captured reliably the first time.
WHAT TO NEVER ASSUME
That a source will keep its current structure.
WHAT TO BUILD FOR
Change, detected rather than discovered downstream.
