Practical transformation code.
WHAT THE COMMON LIBRARIES ARE
A long-established tabular library Newer libraries with better performance and lazy evaluation Interfaces to distributed engines Analytical engines usable in process
WHAT TO USE FOR MODEST DATA
Whatever is familiar, on one machine.
WHAT TO BE CAREFUL WITH
Memory use, which is frequently several times the file size Row-by-row iteration, which is very slow Operations producing copies unnecessarily
WHAT TO PREFER
Vectorised operations over loops Reading only required columns Processing in chunks where data is large
WHAT LAZY EVALUATION PROVIDES
Optimising the whole chain, and processing larger than memory.
WHAT TO ALWAYS SPECIFY
Types on reading.
WHY
Inference is slow and produces inconsistent types between files.
WHAT TO BE CAREFUL WITH IN TYPES
Identifiers read as numbers, losing leading zeros Dates parsed with the wrong day and month order Mixed types in a column silently becoming text
WHY THE DATE POINT MATTERS HERE
Day-month ordering is standard locally and month-day elsewhere, and tools default differently.
WHAT TO DO
State the format explicitly, always.
WHAT TO TEST
Against a small known input with a known expected output.
