The architecture behind modern AI.
THE DEFINITION
A neural network architecture using attention: when processing a word, the model weighs how relevant every other word in the input is.
WHY IT MATTERS
Almost every current language model is built on it.
WHAT IT REPLACED
Earlier approaches processed text sequentially and struggled with relationships across distance.
Transformers process the whole input together.
WHY THAT ENABLED LARGE MODELS
The architecture parallelises well, so training can use enormous computation efficiently.
WHAT IT EXPLAINS
Why models handle long context Why they are sensitive to how input is phrased Why the structure of your prompt affects the output
BEYOND TEXT
The same architecture is used for images, audio and other data.
WHAT TO REMEMBER
The model weighs everything in your input against everything else. Structure matters.
RELATED TERMS
Attention, large language model, context window.
