The architecture behind modern language models.
WHY IT MATTERS
Almost every current language model is built on this architecture.
Understanding roughly what it does explains several of their behaviours.
THE KEY IDEA
Attention: when processing a word, the model weighs how relevant every other word in the input is.
That lets it handle relationships across long distances in text.
WHAT IT REPLACED
Earlier approaches processed text sequentially, word by word, and struggled with long-range relationships.
Transformers process the whole input together, which is faster and handles context better.
WHY THIS ENABLED LARGE MODELS
The architecture parallelises well, so training can use enormous computation efficiently.
WHAT IT EXPLAINS
Why models handle long context Why they are sensitive to how input is phrased Why the order and structure of your prompt affects the output
BEYOND TEXT
The same architecture is used for images, audio and other data.
WHAT YOU NEED TO REMEMBER
The model weighs everything in your input against everything else. What you include, and how you structure it, genuinely matters.
