Why attention has separate Q, K and V, and why it divides by √d_k

01The equation, with the parts named

Scaled dot-product attention, as written in the Transformer paper, is:

Attention(Q, K, V) = softmaxQ K^T√d_k V

Every token in the sequence starts as one vector x. Three learned matrices turn it into three different vectors:

The query of token i is dotted with the key of every token j. Those scores are divided by √d_k, passed through a softmax so they sum to 1 across j, and used as weights on the values. The output for token i is a weighted average of other tokens' values, where the weights say how much i cares about each j.

Both questions in this note are about choices in that one line: why W_Q, W_K and W_V are three matrices rather than one, and why √d_k is there at all.

02What one matrix cannot do

Suppose attention used a single learned matrix W to score how much token i should attend to token j:

score(i, j) = x_i^T W x_j

This is a bilinear form, and for a symmetric W it is exactly symmetric: x_i^T W x_j = x_j^T W x_i. Even for a general W, the score is a fixed function of the pair, with no way for a token to behave differently as the one asking and as the one being asked about. And if the same vectors were then averaged as the output, the thing that decides relevance and the thing that gets passed along would be the same object.

That rules out three things a language model needs, and the three projections exist to supply them.

03Asymmetric roles: addressing versus content

Attention between two tokens is not a symmetric relationship. A verb looking for its direct object has a question to ask. The noun that is the direct object has a property that answers it. The verb should attend strongly to the noun; there is no reason the noun should attend equally strongly back to the verb.

Separate projections make that asymmetry learnable. W_Q maps the verb's vector into a query that says, in effect, "I need an object". W_K maps the noun's vector into a key that says "I am a noun phrase in object position". The dot product between those two is high. The noun's query and the verb's key are produced by the same matrices from different inputs, and nothing forces their dot product to match.

So a token can advertise itself differently from how it searches. The key is a token's address; the query is the address a token is looking for. One matrix cannot express an address and a search as different things.

04Where to look, separated from what to pass

The second split is between Q and K on one side and V on the other.

Q and K together decide the attention weights: the pattern of who attends to whom. V decides what actually travels along those connections. Return to the verb and its object. The matching is done on grammatical role, but the payload the verb wants is the noun's meaning, its tense-agreement features, whatever the next layer needs to know about that object. The thing that makes the noun findable and the thing worth transmitting once it is found are different pieces of information, and in a single-matrix design they would have to be the same vector.

There is also a training reason. If the vector used for matching were also the vector used as output, every gradient that said "be easier to find for this kind of query" would also be a gradient on "carry this content", and the two signals would fight. With W_K and W_V separate, the routing can improve without disturbing the payload and the payload can improve without disturbing the routing. The decoupling is what lets both be learned cleanly.

05Different dimensions, and multi-head attention

Because the value projection is its own matrix, the values can live in a different space from the queries and keys. Queries and keys must share a dimension, d_k, since they are dotted together. Values have their own dimension, d_v, and the output of attention has that dimension too. Nothing ties the matching space to the output space.

That freedom is what makes multi-head attention cheap. With a model width of 512 and 8 heads, each head projects the tokens down to d_k = d_v = 64, runs attention in that small space, and the 8 outputs are concatenated back to 512. Each head gets its own W_Q, W_K and W_V, so each head can learn a different notion of relevance: one head tracks syntactic dependencies, another tracks coreference, another attends to the previous token. The total parameter count is about the same as one full-width head, and the model gets several relationship types in parallel instead of one.

06The dot product grows with the head width

Take a query q and a key k, both in ℝ^d_k, and assume their components are independent random variables with mean 0 and variance 1. That is a fair description of what the projections produce at initialisation, and roughly what layer normalisation keeps them near during training.

The dot product is a sum of d_k products:

q · k = ∑_i=1^d_k q_i k_i

Each product q_i k_i has mean 0, because q_i and k_i are independent with mean 0. Its variance is Var(q_i) · Var(k_i) = 1. The products are independent of each other, so variances add:

𝔼[q · k] = 0,    Var(q · k) = ∑_i=1^d_k Var(q_i k_i) = d_k

So the standard deviation of a raw attention score is √d_k. The scores are not on a fixed scale; they get wider as the head gets wider.

The right column is the typical gap between two scores drawn from that distribution, which is what the softmax compares. A gap of 11 means one weight is e¹¹, about 60,000, times the other. A gap of 32 is a ratio of about 10¹⁴. At that point the softmax is not weighting; it is picking.

07Softmax saturation and the vanishing gradient

Softmax turns a vector of scores z into probabilities s_i = e^z_i / ∑_j e^z_j. When the scores are spread over tens of units, the largest one takes almost all the mass and every other probability is close to 0. The output looks one-hot.

That is bad for the model, because attention is supposed to blend several tokens, but it is worse for training. The derivative of the softmax is:

∂s_i∂z_j = s_i (δ_ij − s_j)

Read it for the saturated case. For the winning token, s_i ≈ 1, so the diagonal term is s_i(1 − s_i) ≈ 0. For every other token, s_i ≈ 0, so every term involving it is ≈ 0 as well. The whole Jacobian is close to zero. Whatever gradient arrives at the attention weights from the loss, almost none of it passes back through the softmax to the scores, and from there to W_Q and W_K.

The attention pattern is then frozen at whatever it was at initialisation, which was random. The head cannot learn where to look, because the signal that would teach it is multiplied by a number near zero at every step. Deeper models make it worse: the vanishing compounds through layers.

08What the scaling does, and what it does not

Dividing the scores by √d_k divides their standard deviation by √d_k, which takes it from √d_k back to 1:

Varq · k√d_k = d_kd_k = 1

Now the softmax inputs sit at unit scale no matter how wide the head is. Score gaps are of order 1, probabilities are spread rather than one-hot, the softmax Jacobian has meaningful entries, and gradients flow to the query and key projections. The same architecture can be scaled from d_k = 64 to d_k = 256 without retuning anything about attention, which is a large part of why the Transformer scaled at all.

A few things worth being precise about:

• It is a temperature. Dividing the logits by a constant is exactly a softmax temperature of √d_k. The choice is principled rather than tuned: it is the one value that makes the variance independent of width.
• It assumes unit-variance inputs. The derivation needs q and k components to have variance about 1. Initialisation schemes and layer normalisation are what make that true in practice. If the projections drift to large norms during training, scores can still grow, which is why some later models add normalisation on queries and keys directly.
• It does not change the argmax. Scaling is monotone, so the most-attended token is the same with or without it. What changes is how sharply the mass concentrates, and therefore whether anything can be learned about it.
• Sharp attention is not always wrong. A trained model does produce near one-hot attention where that is correct, for example a head that copies the previous token. The scaling prevents that from being the starting point for every head before any learning has happened.

09Takeaways

1. One matrix gives a symmetric, static score. A token could not ask for one thing and advertise another.
2. Query and key split addressing from being addressed. The key is a token's address; the query is the address it is looking for.
3. Value splits routing from payload. What makes a token findable and what it hands over are different, and separating them keeps their gradients from fighting.
4. Separate projections allow separate dimensions. That is what lets multi-head attention run several small heads for the price of one wide one.
5. A dot product of unit-variance vectors has variance d_k. Raw scores spread with the square root of the head width.
6. Wide scores saturate the softmax, and a saturated softmax has zero gradient. The attention pattern would never move from its random start.
7. Dividing by √d_k restores unit variance. It is a fixed temperature chosen so that width does not change the learning dynamics.