Matching approach to volume.
WHAT SMALL DATA MEANS
Fits comfortably in memory on one machine.
WHAT TO USE
A database, a script, a spreadsheet where appropriate.
WHY THAT MATTERS
Most organisations never leave this category, and the tooling for larger scale is unnecessary overhead.
WHAT MEDIUM DATA MEANS
Too large for memory, comfortable on one machine's storage.
WHAT TO USE
A columnar database or a single-machine analytical engine.
WHAT MANY PEOPLE UNDERESTIMATE
How much one modern machine can handle.
WHAT THAT MEANS PRACTICALLY
Distributed processing is frequently adopted for volumes a single machine would manage easily.
WHAT LARGE DATA MEANS
Genuinely requiring several machines.
WHAT THAT INTRODUCES
Coordination Partial failure Substantially more operational complexity
WHAT TO ESTABLISH BEFORE ADOPTING DISTRIBUTED TOOLING
The actual volume, measured.
WHAT TO ASK
Could one machine do this?
WHY
The answer is usually yes, and the simpler system is better in every respect.
WHAT TO PLAN FOR
Growth, but measured growth rather than imagined.
