Part of the Pristone Academy AI Technical Track
Attention is all you need — how transformers read everything at once.
The one idea that made modern AI possible: every word gets to look at every other word, in parallel.
To understand a word, you have to know which other words it depends on.
Take the sentence “The animal didn't cross the street because it was too tired.” What does “it” refer to? You resolved that instantly by linking “it” back to “animal.” That act — letting one word pull context from the right other words — is exactly what the attention mechanism does, for every word, simultaneously.
Attention is the heart of the transformer, the architecture behind GPT, Claude, and essentially every modern language model. The 2017 paper that introduced it was titled, with some confidence, “Attention Is All You Need.” The title was a direct shot at the era it ended: the previous models all leaned on recurrence (reading word by word) or convolution (scanning local windows). The paper's claim was that you could throw both out and keep only attention. They were right. Once attention existed, the field changed overnight.
Try it. Pick a word below and watch how much attention it pays to every other word. Click “it” and notice where the weight lands — then switch sentences and watch the same word point somewhere new.
1. Pick a word — that's the query:
2. See how much it attends to every other word (the keys). Brighter = more attention:
it attends hardest to animal. That link is exactly how the model resolves what “it” refers to — and the same sentence flips the answer when the noun changes.
Weights shown are illustrative, chosen to make the idea visible. A real transformer learns them from data and runs many such patterns (heads) at once.
Before attention, models read text one word at a time, left to right, and forgot the beginning by the time they reached the end. Attention lets every word see every other word at once — no forgetting, and the whole thing runs in parallel on a GPU.
Query, key, value — match, weight, and blend.
From each word's embedding, the model derives three vectors by multiplying it with three learned weight matrices. Then it does a weighted lookup.
What this word is looking for. “I'm a pronoun — who's my antecedent?”
What each word offers. “I'm a singular animal noun.” A query scores high against keys it matches.
The content actually pulled in once a match is found — the information that gets blended into the result.
The dot product of a query with every key gives a score: how relevant is each word to this one? Those scores are passed through softmax to become weights that sum to one — a distribution of attention, exactly the bars you saw in the demo. The output is the values, blended in those proportions. The whole operation is one tidy formula:
Why divide by √dₖ?
Dot products of long vectors grow large, and large numbers push softmax into a corner where one weight is roughly one and the rest are roughly zero — gradients vanish and learning stalls. Dividing by √dₖ (the square root of the key dimension) keeps the scores in a sane range so the softmax stays soft. It is a small detail that makes the whole thing trainable.
Run it many times in parallel, and hide the future during training.
One attention pass captures one kind of relationship. So the model runs many in parallel — each a head with its own Q/K/V matrices. One head tracks grammar, another long-range references, another tone. Their outputs are concatenated. Parallel perspectives, fused.
A language model predicts the next word, so during training it must not peek ahead. A causal mask sets the attention score for any future word to −∞ before softmax, zeroing its weight. Each word can only attend to itself and what came before.
Why it replaced RNNs and CNNs
The previous generation paid for distance. Recurrent networks processed text strictly in sequence, each step waiting on the last — slow to train and prone to forgetting across long passages. Convolutional ones saw only local windows and had to stack many layers to reach across a sentence. Attention has two decisive advantages over both: parallelism (all words processed at once, so GPUs scream) and long-range dependencies (a word at position 1 connects to position 5,000 just as directly as to its neighbor). Speed plus memory — that combination is why transformers won.
The old architectures ran fixed plays. The RNN was a strict passing chain — point guard to shooting guard to forward to center, in order, every possession, the early read fading before it reached the end. The CNN was local zone vision — each defender watching only their patch, needing layers of relays to see the whole floor.
Attention is the point guard with full-court vision: one glance reads every teammate at once and decides who matters right now. No passing order, no local blind spots — the far corner as clear as the near one. That is the entire upgrade.
Every word attends to every other word, so the cost grows with the square of the sentence length. Double the context, quadruple the work. This quadratic cost is the central reason long context windows are expensive — and a hot area of research.
Three things to keep about attention.
Queries match against keys to produce weights; the output is a weighted blend of values. That's the whole idea — everything else is detail.
By letting every word see every other at once, attention unlocked both GPU-scale training and true long-range understanding. RNNs could do neither.
Multi-head attention runs many lookups in parallel, each catching a different relationship — grammar, reference, tone — then fuses them.
Causal masking is what makes a transformer a next-word predictor: it can only ever look backward.
Attention lets every word ask “who's relevant to me?” and pull in a weighted blend of everything that answers — done in parallel, across many heads, it's the entire engine of the modern language model.
If you want to go from “I get the analogy” to “I could implement this,” that's the jump a 1:1 session is built for.
Frequently asked questions
More free lessons
What AI actually is, and how learning from data differs from following rules.
How a machine sorts a pile of points into clean groups with no labels.
How a machine keeps the few directions that matter and throws away the rest.
How AI redistributes work across the layers of a job — and where humans stay scarce.
Curious who's behind these lessons? See the proof of work.