Moving data between systems.
WHAT TO BUILD IN
Validation of what arrives, before acting on it A record of what was processed Safety when run twice Clear failure rather than partial completion
WHY VALIDATE FIRST
An import of a malformed file can corrupt more than it adds.
WHAT TO CHECK
Row counts against expectations Required fields present Totals against a known figure
WHAT TO DO ON VALIDATION FAILURE
Stop, and report precisely what was wrong.
WHAT TO DO ABOUT DUPLICATES
Record an identifier for each record processed, and skip anything already handled.
WHY
Because imports are rerun after apparent failures.
WHAT TO LOG
What was processed, what was skipped, and what failed.
WHAT TO PRODUCE
A summary at the end.
WHAT TO BE CAREFUL WITH
Character encoding Date formats Leading zeros in reference codes Very large files held in memory
WHAT TO TEST
With a real file, including awkward rows.
WHAT TO MONITOR
That the import ran and produced plausible counts.
