Processing across machines.
WHAT THEY PROVIDE
Splitting work across machines Handling of machine failure Processing volumes exceeding one machine
WHAT THE DOMINANT FRAMEWORK IS
A general-purpose engine supporting batch and streaming, with interfaces in several languages.
WHAT THE PROGRAMMING MODEL IS
Operations on distributed collections, executed lazily and optimised as a whole.
WHAT LAZY EVALUATION MEANS
Nothing runs until a result is requested.
WHY THAT MATTERS
The engine optimises the whole chain, which it could not do step by step.
WHAT A SHUFFLE IS
Redistributing data across machines, required by joins and grouping.
WHY IT DOMINATES PERFORMANCE
It moves data over the network, which is far slower than computing.
WHAT REDUCES SHUFFLES
Filtering before joining Broadcasting small tables rather than shuffling both sides Partitioning data consistently
WHAT DATA SKEW IS
Some partitions far larger than others.
WHAT IT CAUSES
Most machines finishing while one continues, and frequently failing on memory.
WHAT CAUSES IT
Uneven distribution of the join or grouping key, often a dominant value or nulls.
WHAT TO DO
Handle the dominant values separately, or add randomness to spread them.
WHAT TO ESTABLISH BEFORE ADOPTING ANY OF THIS
That one machine genuinely cannot do it.
