Attention lets tokens read each other
A token on its own knows only itself. Attention is how "it" finds out what "it" refers to — the one operation that lets context actually flow between positions.
The whole point: mixing context
Attention enables each token to read information from earlier tokens in the sequence. Without it, every position would be processed in isolation; with it, a word can absorb meaning from anything that came before.
It's selective, not a blur
A token doesn't soak up all earlier tokens equally. It scores how relevant each earlier token is, then pulls hardest from the ones that matter — a pronoun reaching back to its noun, a verb reaching back to its subject.
This is what "transformer" means
Stacking many of these attention steps is the heart of the transformer. Everything in the next lesson — the block, the layers — is built around repeating this read-from-the-past operation.
Keywords — tap to unfold the plain meaning
Attention is the one operation that lets a token read from earlier tokens — selectively, weighting the relevant ones most.
Query, Key, Value: the three roles
All of attention is built from three little vectors per token. Get what each one is for and the formula in the next station reads itself.
Query — the question this item asks
The Query (Q) is the question the current token poses: "what am I looking for?" It's the vector the token uses to go shopping among everything before it.
Key — the label each item advertises
The Key (K) is the label each earlier token advertises: "here's what I'm about." A Query matches strongly against Keys that answer its question.
Value — the content each item carries
The Value (V) is the actual content each token hands over once it's been matched: "if you pick me, here's what you get." The output is a blend of Values.
The recipe in one line
Score every earlier token by Query·Key, softmax those scores into weights that add up to 1, and output the weighted sum of their Values. That sentence is the entire mechanism.
Keywords — tap to unfold the plain meaning
Query asks, Key advertises, Value carries. Match Q against the Keys, weight by the match, blend the Values.
The scaled dot-product formula
Don't let the symbols scare you. You already know what they mean from Station 02 — this is just that one sentence written in math.
Dot product = the relevance score
A dot product multiplies two vectors' paired numbers and adds them. It's larger when the vectors point the same way — so Q·K is a natural "how well does this Key answer my Query" score.
Why divide by √d_k
When the vectors are long (high d_k, the per-head dimension), dot products grow large and softmax gets razor-sharp. Dividing by √d_k rescales the scores so softmax stays smooth and trainable.
Softmax then weighted sum
Softmax squashes the raw scores into positive weights that add up to 1. Multiplying those weights by V produces the weighted sum of Values — the token's attention output.
Math, decoded — scaled dot-product attention
- Qthe Queries — what each token is asking for
- Kthe Keys — the label each earlier token advertises
- Vthe Values — the content each earlier token carries
- QKᵀevery Query dotted with every Key: the raw relevance scores, one per pair
- √d_ksquare root of the per-head dimension; divides the scores so softmax doesn't get too peaky
- softmax(·)turns those scores into positive weights that sum to 1
- (…)Vweights times Values = the blended output the token walks away with
Read it left to right: score every token-pair with QKᵀ, shrink by √d_k, softmax into weights, then blend the Values. It's exactly the one-line recipe — Query·Key, softmax, weighted sum of Values — written in symbols.
Keywords — tap to unfold the plain meaning
softmax(QKᵀ/√d_k)V — score with the dot product, tame it with √d_k, normalize with softmax, blend the Values.
The causal mask & many heads
Two refinements turn raw attention into the thing that actually powers an LLM: one stops it cheating, the other lets it specialize.
Causal mask — no peeking ahead
The causal mask means a token may only attend to tokens before it, so it can't peek at the future it's trying to predict. Next-token prediction would be trivial — and useless — if a token could read the answer.
Multi-head — many reads at once
Multi-head attention runs several attention computations in parallel, each with its own separate Q/K/V projections. Each head learns a different relationship — one for syntax, one for referential links, and so on — then their notes are combined.
Keywords — tap to unfold the plain meaning
Causal mask = read only the past. Multi-head = many parallel reads, each a specialist, notes combined.
The O(n²) cost — and the three fixes
Attention's superpower — letting every token see every other — is also its bill. That n-squared is the single biggest reason serving LLMs is hard, and the next several lessons are all about taming it.
Quadratic in the sequence length
Because each of n tokens scores against all n tokens, attention complexity is O(n²) in the sequence length. Double the context and the attention work roughly quadruples.
Fix 1 — the KV cache
The KV cache stores the unchanging Keys and Values from past tokens so each new step doesn't recompute them. You'll meet it in depth later; here it's the headline fix for repeated work.
Fix 2 — GQA
Grouped-Query Attention (GQA) lets multiple query heads share fewer K/V heads, shrinking how much Key/Value data has to be stored and moved.
Fix 3 — FlashAttention
FlashAttention computes the scores in fast on-chip SRAM instead of materializing the full score matrix in slow memory — the same answer, far less memory traffic.
Keywords — tap to unfold the plain meaning
Math, decoded
- nthe number of tokens in the sequence (the context length)
- n × neach token scores against every token, so the score matrix has n² entries
- O(n²)work grows with the square of the sequence — the cost the fixes attack
Every Query meets every Key, so the score matrix is n by n. That squared term is why long contexts get expensive fast — and why KV cache, GQA, and FlashAttention exist.
Attention is O(n²) because every token attends to every token. KV cache, GQA, and FlashAttention are the three tools built to flatten that cost.
On your cluster
The theory shows up directly in the config you serve. Peek at the model on the box and you can read GQA and hybrid attention straight off the head counts.
GQA in the config
The example model uses 24 query heads with only 4 KV heads (head_dim 256). Because 6 query heads share each K/V head, the KV cache is 6× smaller than it would be with one K/V head per query head.
Hybrid attention layers
It also mixes attention types: cheap linear attention on most layers, with full attention every 4th layer. The full layers preserve quality where it matters; the linear ones keep the bulk of the network cheap.
Inspect the heads on the 4×H100 box
curl localhost:8000/v1/models # Qwen3.6-27B-FP8 on 4×H100 # config.json reports: "num_attention_heads": 24, "num_key_value_heads": 4, # GQA → 6× smaller KV cache "head_dim": 256
Qwen3.6-27B-FP8 config on the 4×H100 box: 24 query heads / 4 KV heads at head_dim 256 means each K/V head is shared by 6 query heads — a 6× smaller KV cache, directly easing the memory pressure from the O(n²) attention you saw in Station 05. The hybrid schedule (linear attention most layers, full attention every 4th) trims it further.
Keywords — tap to unfold the plain meaning
Check yourself
- What are the three roles a token projects into, and what does each one do?
- What does the √d_k in softmax(QKᵀ/√d_k)V protect against, and what does the causal mask forbid?
- Why is attention O(n²), and which three optimizations are built to flatten that cost?
Your served model wears the theory on its sleeve: 24 query / 4 KV heads is GQA cutting the cache 6×, and every-4th-layer full attention is the hybrid trick in action.