What changed.
THE THREE FACTORS
- Data: the internet produced text and images at a scale previously unimaginable
- Computation: graphics processors made training large models practical
- Techniques: the transformer architecture, and methods for training at scale
All three arriving together produced the change.
WHY NOT EARLIER
The ideas behind neural networks are decades old.
They did not work well until there was enough data and enough computation to train them properly.
THE SCALING OBSERVATION
Researchers found that larger models trained on more data reliably performed better.
That turned progress into an engineering and investment question as much as a research one.
WHAT EMERGED UNEXPECTEDLY
Capabilities that were not designed for: following instructions, reasoning through problems, writing code.
These appeared as models grew, rather than being programmed.
WHAT REMAINS UNSOLVED
Reliable factual accuracy Knowing what it does not know Genuine reasoning about novel situations Learning efficiently from few examples
THE HONEST ASSESSMENT
Remarkable progress in specific capabilities, with fundamental limitations unaddressed.
