Components and how they fit.
WHAT THE LAYERS ARE
- Sources: applications, databases, files, external services
- Ingestion: getting data in
- Storage: where it lives
- Processing: transforming it
- Serving: making it available
- Orchestration: scheduling and coordinating
- Observability: knowing it worked
- Governance: controlling access and quality
WHAT A DATA WAREHOUSE IS
A store optimised for analytical queries over structured data.
WHAT A DATA LAKE IS
Storage holding raw data in its original form, of any structure.
WHAT A LAKEHOUSE IS
A combination: lake storage with warehouse-like structure and guarantees.
WHY THAT CONVERGENCE HAPPENED
Lakes were cheap but unreliable to query; warehouses were reliable but expensive and inflexible.
WHAT AN OPERATIONAL DATABASE IS FOR
Serving an application, with many small reads and writes.
WHY IT SUITS ANALYTICS BADLY
Analytical queries scan large volumes, which competes with the application.
WHAT THAT MEANS
Analytics belongs somewhere other than the production database.
WHAT TO BUILD FIRST
The smallest thing that answers the actual question.
WHAT NOT TO BUILD FIRST
A platform, before you have a use for it.
