Larger datasets.
THE FIRST PROBLEM
Loading everything into memory.
WHAT TO DO INSTEAD
Process in a stream: read, handle, discard, repeat.
WHAT THAT ENABLES
Files larger than available memory.
WHAT GENERATORS ARE
A way of producing values one at a time rather than building a whole collection.
WHY THEY MATTER
Memory stays constant regardless of volume.
WHAT TO AVOID
Building a large list you only iterate over once Repeatedly searching a list, where a set or dictionary would be immediate Nested loops over large collections
THAT LAST POINT
The common cause of a script that works on sample data and never finishes on real data.
WHAT TO DO ABOUT SLOWNESS
Measure before optimising.
Find where the time actually goes, rather than guessing.
WHAT USUALLY HELPS MOST
A better data structure Doing work once instead of repeatedly Avoiding repeated file or network access inside a loop
WHEN TO CONSIDER SPECIALIST LIBRARIES
When standard approaches genuinely cannot cope.
