Self-attention derived on a whiteboard, with one sentence (video)
The written note on Q, K, V and the √dk answers two questions about the attention equation. This one builds the whole equation from nothing, on a whiteboard, with a single four-word sentence as the running example: "The river bank flooded". Two teachers, ten minutes, every number worked by hand. The chapter list and the worked numbers are below the video so you can check each step.
You know "bank" means the edge of a river because "river" is next to it. Self-attention is that move, made into arithmetic: every token asks a question (its query), wears a name tag (its key) and carries a package (its value). A dot product scores each question against each name tag, the scores are divided by √dk so the softmax does not saturate, the softmax turns a row of scores into weights that sum to 1, and the weights blend the packages. "bank" walks in ambiguous and walks out meaning river bank.
Contents
01Chapters
| Time | Chapter | What goes on the board |
|---|---|---|
| 0:00 | The sentence | "The river bank flooded"; bank circled; river and money-bank sketches; the formula lands |
| 0:41 | The transformer in one picture | tokens → embeddings + position → a block of self-attention and feed-forward with residuals and norm, stacked N times → a probability over the next token |
| 2:08 | Every word does three jobs | a question (query), a name tag (key), a package (value) |
| 3:04 | One tool: the dot product | q = (2, 1), k = (2, 0), q·k = 4; a key pointing away scores 0 |
| 3:49 | Where Q, K and V come from | x WQ, x WK, x WV; why three matrices and not one |
| 4:48 | Score every query against every key | the 4×4 grid of Q KT; the bank row reads 1, 4, 3, 2 |
| 5:40 | Why divide by √dk | spread grows with dk; the softmax goes one-hot; variance back to 1 |
| 6:43 | Softmax: scores → a recipe | 0.7, 2.8, 2.1, 1.4 → 6%, 54%, 27%, 13% |
| 7:38 | Spend the weights on the values | the packages blend into bank′ = (0.61, 0.27); bank → river bank |
| 8:33 | Many heads, and a mask | heads in parallel, concat, WO; −∞ above the diagonal → weight 0 |
| 9:40 | One more time, in one breath | the formula with every symbol annotated |
02The worked example, so you can check the video
The video uses two-dimensional queries and keys so every number fits on a board. Real models use 64 or 128 per head; nothing in the derivation changes except the spread of the scores, which is exactly what chapter 7 is about.
The four tokens, and the toy query, key and value vectors the video assigns to them:
| Token | Query q | Key k | Value v | What the value "means" |
|---|---|---|---|---|
| The | (1, 0) | (0, 1) | (0, 0) | nothing much |
| river | (1, 1) | (2, 0) | (1, 0) | water |
| bank | (2, 1) | (1, 1) | (0, 1) | bank-ness |
| flooded | (0, 1) | (1, 0) | (0.5, 0) | some water |
In a real model each of these comes from the token's embedding x through a learned matrix: q = x WQ, k = x WK, v = x WV. The video's chapter 5 is about why those are three matrices rather than one: with a single matrix, "bank" could only match things that look like "bank". Separate projections let it ask for water-words while "river" answers to that question. Asking and answering become different skills. The written note makes the same argument with the bilinear form.
03Scores: Q KT
Stack the queries as the rows of Q and the keys as the rows of K. Transposing K turns its keys into columns, so Q KT does every dot product at once. Row i, column j is how much token i cares about token j:
| The | river | bank | flooded | |
|---|---|---|---|---|
| The | 0 | 2 | 1 | 1 |
| river | 1 | 2 | 2 | 1 |
| bank | 1 | 4 | 3 | 2 |
| flooded | 1 | 0 | 1 | 0 |
Check one cell: qbank · kriver = 2·2 + 1·0 = 4. A big dot product means the two vectors point the same way; a key pointing away from the query, such as (−1, 2), scores −2 + 2 = 0. That is the whole matching rule, and it is one multiply-and-add per pair, which is why GPUs are so good at it.
04Why divide by √dk
A dot product adds up dk terms. If each component has mean 0 and variance 1, the sum has variance dk, so its spread is √dk: about 1.4 for the toy example, 8 for a 64-dimensional head. Feed a softmax scores that are 8 or 16 apart and it goes almost one-hot, one weight near 1 and the rest near 0. The softmax Jacobian is si(δij − sj), which is near zero everywhere in that state, so almost no gradient reaches WQ and WK and the attention pattern never learns to move.
Var = = 1
Dividing by √dk puts the scores back on unit scale no matter how wide the head is. It is a fixed softmax temperature, chosen so that width does not change the learning dynamics. Not magic, just a sensible scale. For the toy example the bank row becomes 1/√2, 4/√2, 3/√2, 2/√2 = 0.71, 2.83, 2.12, 1.41.
05Softmax: the bank row becomes a recipe
Softmax exponentiates each score and divides by the sum, so every weight is positive and the row adds up to 1:
| The | river | bank | flooded | sum | |
|---|---|---|---|---|---|
| scaled score s | 0.71 | 2.83 | 2.12 | 1.41 | |
| es | 2.03 | 16.9 | 8.33 | 4.11 | 31.4 |
| weight | 0.065 | 0.539 | 0.266 | 0.131 | 1.000 |
So "bank" will listen to "river" about half the time, to itself about a quarter, a little to "flooded", and mostly ignore "The". Every token gets its own row, its own recipe for who to listen to. The full matrix is softmax(Q KT / √dk), and the whole thing has been about how to earn one row of it.
06Spending the weights on the values
The last factor, V, is where the weights get spent. The new vector for "bank" is the weighted sum of the four value packages:
bank′ = 0.065·(0, 0) + 0.539·(1, 0) + 0.266·(0, 1) + 0.131·(0.5, 0) = (0.60, 0.27)
River's value said "water" and bank's said "bank-ness", and the mix is mostly water with some bank. The vector that leaves the layer means river bank. The ambiguity was resolved by context, not by a dictionary, and the only arithmetic involved was dot products, an exponential and a weighted sum. (The board in the video shows 0.61 for the first component; the exact value is 0.604, so the rounding on the board is one digit generous.)
Attention(Q, K, V) = softmax V
Read it left to right now: queries ask, keys answer, Q KT scores the match, √dk keeps the scores sane, softmax turns scores into a recipe, and the recipe blends the values.
07Heads and the causal mask
One attention pattern is a bottleneck: "bank" may need "river" for meaning and "flooded" for grammar at the same time. So a layer runs several attentions in parallel, each with its own WQ, WK and WV, each in a smaller space of dmodel / h dimensions. Their outputs are concatenated and mixed by one more matrix, WO. That is multi-head attention, and it costs about the same as one wide head.
A model that generates text one token at a time cannot look at tokens it has not produced yet. Every score to the right of the diagonal is set to −∞ before the softmax, which becomes a weight of exactly 0. That is causal attention, and it is what GPT-style decoders use. The video ends where the block's other half begins: the feed-forward network, where each token digests what it gathered on its own. Attention mixes across tokens; the MLP thinks within one; residual, normalise, repeat, dozens of layers deep.
08Takeaways
- Attention is "look at the neighbours" made into arithmetic. The sentence resolves "bank" the same way the layer does.
- Query, key and value are three jobs. Ask, be found, hand over. Matching happens between queries and keys; content flows through values.
- The dot product is the matching rule. Agree → big score; disagree → near zero or negative; one multiply-and-add per pair.
- Q KT is every pair at once. Row i, column j is how much i cares about j.
- √dk keeps the scores at unit variance. Without it the softmax goes one-hot and its gradient vanishes.
- Softmax turns a row of scores into a recipe that sums to 1; the recipe blends the values into a context-aware vector.
- Heads run the same thing in parallel; the mask hides the future. Everything else in the block works one token at a time.