Every model you have heard of — Claude, GPT, Gemini — is built on one idea published in 2017. It has an intimidating name and a genuinely simple core: when reading a word, look at the other words that matter.
You do not need the maths. You do need the intuition, because it explains why models are good at some things and unreliable at others, and why context length is the number everyone argues about.
Before 2017, language models read strictly left to right, one word at a time, carrying a running summary. This works for short sentences and falls apart on long ones — by the end of a paragraph the summary has been overwritten so many times that the beginning is effectively gone.
You can feel the problem yourself. Read this: “The trophy would not fit into the brown suitcase because it was too large.”
What does it refer to? The trophy. Now change one word: “…because it was too small.” Suddenly it is the suitcase.
Nothing about the word it changed. To resolve it, you have to look back at two nouns and weigh them against an adjective at the end. A running summary cannot do that. It needs to look around.
Attention lets every word look at every other word at once, and decide which ones matter for interpreting it.
That is genuinely it. Each word asks a question — roughly “what do I need to know to understand myself here?” — and every other word offers an answer. The strength of each answer becomes a weight. Words with high weights shape the meaning; words with low weights are largely ignored.
The blue word is the one being interpreted. The bright words are the ones it is attending to.
Watch what happens at it. The model is not scanning backwards hoping to find a noun. It is weighing trophy and suitcase simultaneously against large, and the weights settle on the right one.
The architecture is called a transformer because each layer transforms every word’s representation using information from the others. Stack dozens of those layers and early ones settle grammar, later ones handle meaning and reference. Nobody designed that division of labour — it emerged from training, and researchers discovered it afterwards.
It reads everything at once. Unlike the old left-to-right approach, a transformer processes all the words in parallel. That parallelism is the reason these models could be trained on the scale they were — and it is why GPUs, which do thousands of things simultaneously, became the hardware of the field.
Comparing everything to everything is expensive. Ten words means a hundred comparisons. A thousand words means a million. The cost grows with the square of the length.
This is the real reason context windows are finite and why long conversations get slow and expensive. It is not an arbitrary limit a company chose. It is arithmetic.
Every single word you see generated is one full pass through all four stages.
Stage four is worth sitting with. The model produces one token, adds it to the text, and starts over — re-reading everything, including what it just wrote. A 500-word answer is 650-odd complete passes.
That is why responses stream in rather than appearing at once, and why a model can contradict itself mid-paragraph. It is not consulting a plan. At each step it is choosing the next token given everything so far.
Why order matters in your prompts. Instructions buried in the middle of a long document get less weight than the same instructions at the start or end. This is a measured effect, not folklore — and it is why “put the important thing last” is common advice that actually works.
Why models lose the thread in long conversations. Attention is spread across everything. The more there is, the thinner it spreads.
Why they cannot revise. A token, once generated, is part of the input for every token after it. There is no going back to fix an earlier sentence — which is why asking a model to check its own work in the same response rarely helps, and asking in a fresh message often does.
Attention means every word can look at every other word and decide what matters. Everything else — the cost of long contexts, instructions getting lost in the middle, one token at a time with no revising — follows from that one mechanism.