Storage-based platforms.
WHAT A DATA LAKE IS
Object storage holding files, in any format, at low cost.
WHAT IT PROVIDED
Cheap storage of everything, including data with no defined use yet.
WHAT WENT WRONG WITH EARLY LAKES
No schema enforcement No transactions No way to update or delete reliably Accumulation of files nobody understood
WHAT THAT PRODUCED
Stores nobody could query with confidence.
WHAT TABLE FORMATS ADDED
Transactional guarantees over files Schema tracking and evolution Time travel to previous versions Efficient updates and deletes Statistics enabling files to be skipped
WHAT THE MAIN FORMATS ARE
Several open formats, each providing broadly these capabilities.
WHAT THEY ENABLE
Warehouse-like behaviour over cheap storage, queryable by several engines.
WHY THAT MATTERS
Storage and compute are separated, and you are not locked to one engine.
WHAT TO STILL ENFORCE
Schema, quality and documentation.
WHY
The format prevents corruption, not disorder.
WHAT TO AVOID
Storing everything indefinitely with no catalogue.
