Doing many things at once.
WHAT PARALLELISM IS
Executing parts of a computation simultaneously.
WHAT TYPES EXIST
- Data parallelism: the same operation on many data elements
- Task parallelism: different operations simultaneously
- Pipeline parallelism: stages operating on a stream
WHAT SHARED MEMORY PARALLELISM IS
Threads on one machine, sharing memory.
WHAT DISTRIBUTED MEMORY PARALLELISM IS
Processes on separate machines, communicating by messages.
WHAT THE STANDARD FOR THAT IS
A message passing interface, used across the field.
WHAT AMDAHL'S OBSERVATION STATES
Speedup is limited by the portion that cannot be parallelised.
WHY THAT MATTERS
A program ninety per cent parallel cannot exceed a tenfold speedup, however many processors are added.
WHAT THAT ARGUES FOR
Identifying and reducing the serial portion first.
WHAT LIMITS SCALING IN PRACTICE
Communication overhead Load imbalance between workers Memory bandwidth Synchronisation points
WHAT LOAD IMBALANCE COSTS
Everyone waits for the slowest.
WHAT TO MEASURE
Scaling: how performance changes as resources increase.
WHAT POOR SCALING INDICATES
Communication or imbalance dominating.
