The most common and most troublesome source.
WHAT GOES WRONG WITH DELIMITED FILES
Separators appearing within values Inconsistent quoting Line breaks inside fields Character encoding differing between files Headers present sometimes and not others Column order changing
WHY THAT LAST POINT IS DANGEROUS
Reading by position loads the wrong data into the wrong columns, silently.
WHAT TO DO
Read by header name, and verify the expected columns exist.
WHAT TO DO WHEN A COLUMN IS MISSING
Fail loudly, rather than proceeding with nulls.
WHAT ENCODING PROBLEMS LOOK LIKE
Characters appearing as symbols, particularly in names.
WHAT TO SPECIFY EXPLICITLY
The encoding, rather than relying on detection.
WHAT TO VALIDATE ON ARRIVAL
Row count against any manifest Expected columns Types Ranges of key values
WHAT TO DO WITH FILES THAT FAIL
Move them aside, and alert.
WHAT NOT TO DO
Process partially and continue.
WHAT TO RECORD
Which file produced which rows.
WHY
It is the only way to trace a wrong value back to its origin.
WHAT TO NEVER ACCEPT AS A SOURCE FORMAT
A spreadsheet, where it can be avoided.
