How data is physically stored.
WHAT TEXT FORMATS PROVIDE
Human readability Universal support
WHAT THEY COST
Large size Slow parsing No type information
Ambiguity: separators within values, quoting, encoding
WHY THAT AMBIGUITY MATTERS
Delimited files with embedded separators are a permanent source of corruption.
WHAT COLUMNAR FORMATS PROVIDE
Compression, frequently several times better Reading only required columns Embedded types and statistics Predicate pushdown, skipping data that cannot match
WHAT TO USE FOR ANALYTICAL STORAGE
A columnar format, essentially always.
WHAT ROW-BASED BINARY FORMATS SUIT
Streaming and message payloads, where whole records are read.
WHAT SCHEMA EVOLUTION MEANS
Reading old files with a newer schema.
WHAT THE FORMATS HANDLE DIFFERENTLY
Adding, removing and renaming columns.
WHAT TO PLAN
How schema changes are handled, before they occur.
WHAT COMPRESSION TO CHOOSE
Faster codecs for frequently read data Higher-ratio codecs for archival
WHAT FILE SIZING MATTERS FOR
Very many small files perform badly on object storage and distributed engines.
WHAT TO AIM FOR
Files large enough to be efficient, compacted periodically.
