Dense vs MoE; MHA vs GQA vs MQA.
A dense kitchen sends every order through the same crew. An MoE kitchen owns many specialist stations but dispatches each token to only a few. GQA/MQA are shared index-card writers: many query readers reuse fewer K/V records.
Most generative LLM servers repeatedly run a causal decoder stack. Encoder-only models and encoder-decoder models have different request shapes, so identify the architecture before applying decoder assumptions.
The 2017 design had two stacks. The encoder reads the whole input at once; every token attends to every other. The decoder generates left-to-right with masked self-attention (so it cannot peek at future tokens) plus cross-attention that reads the encoder's output. Modern decoder-only LLMs keep only the right half (no encoder and no cross-attention) and normalize before each sublayer (pre-norm) instead of after. Same building blocks, a simpler shape, which is exactly why the serving baseline is decoder-only. Lesson 6 opens up one such decoder block in detail.
Multi-head attention gives every query head its own K/V head (recall from Lesson 5: a head is one independent Query/Key/Value attention; the KV heads are the ones whose Keys and Values get stored). GQA shares each K/V head across a group of query heads; MQA shares one K/V head across all queries. Fewer KV heads shrink the KV cache (the per-request attention memory the model builds up while generating; computed in Lesson 14) and the attention bandwidth (here "bandwidth" means GPU-memory bandwidth (the speed of reading weights and the KV cache from HBM, not network bandwidth)), but the architecture is fixed at training time.
A Mixture-of-Experts block owns many expert MLPs (the feed-forward network, or FFN, also called the MLP, is the big per-token math block in each layer; see Lesson 6), while a router sends each token to a small top-k subset. Total parameters determine storage and placement; active parameters better approximate per-token expert compute. Both matter for inference.
Experts can be imbalanced, tokens may cross GPUs, and capacity limits can drop or reroute work. Expert parallelism trades dense compute for routing, communication, and load-balancing complexity.
Your Qwen3.6 config demonstrates GQA directly: 24 query heads share 4 KV heads, cutting KV width by 6× versus full MHA. It is not a standard top-k MoE, so expert-routing claims are taught as a separate architecture case.
Those are the architecture variants; next, Lesson 8 runs a block all the way to a chosen token.
Dense vs MoE; MHA vs GQA vs MQA.
A dense kitchen sends every order through the same crew. An MoE kitchen owns many specialist stations but dispatches each token to only a few. GQA/MQA are shared index-card writers: many query readers reuse fewer K/V records.
| Decoder-only is the serving baseline | Most generative LLM servers repeatedly run a causal decoder stack |
| The original Transformer, and what LLMs kept | The 2017 design had two stacks |
| MHA, GQA, and MQA change KV width | Multi-head attention gives every query head its own K/V head |
| MoE separates total from active parameters | A Mixture-of-Experts block owns many expert MLPs, while a router sends each token to a small top-k subset |
Most generative LLM servers repeatedly run a causal decoder stack. Encoder-only models and encoder-decoder models have different request shapes, so identify the architecture before applying decoder assumptions.
The 2017 design had two stacks. The encoder reads the whole input at once; every token attends to every other. The decoder generates left-to-right with masked self-attention (so it cannot peek at future tokens) plus cross-attention that reads the encoder's output. Modern decoder-only LLMs keep only the right half (no encoder and no cross-attention) and normalize before each sublayer (pre-norm) instead of after. Same building blocks, a simpler shape, which is exactly why the serving baseline is decoder-only. Lesson 6 opens up one such decoder block in detail.
Multi-head attention gives every query head its own K/V head (recall from Lesson 5: a head is one independent Query/Key/Value attention; the KV heads are the ones whose Keys and Values get stored). GQA shares each K/V head across a group of query heads; MQA shares one K/V head across all queries. Fewer KV heads shrink the KV cache (the per-request attention memory the model builds up while generating; computed in Lesson 14) and the attention bandwidth (here "bandwidth" means GPU-memory bandwidth (the speed of reading weights and the KV cache from HBM, not network bandwidth)), but the architecture is fixed at training time.
A Mixture-of-Experts block owns many expert MLPs (the feed-forward network, or FFN, also called the MLP, is the big per-token math block in each layer; see Lesson 6), while a router sends each token to a small top-k subset. Total parameters determine storage and placement; active parameters better approximate per-token expert compute. Both matter for inference.
Experts can be imbalanced, tokens may cross GPUs, and capacity limits can drop or reroute work. Expert parallelism trades dense compute for routing, communication, and load-balancing complexity.
Your Qwen3.6 config demonstrates GQA directly: 24 query heads share 4 KV heads, cutting KV width by 6× versus full MHA. It is not a standard top-k MoE, so expert-routing claims are taught as a separate architecture case.
Those are the architecture variants; next, Lesson 8 runs a block all the way to a chosen token.