The KV Cache & Its Memory — Inference Engineering

01

What the KV cache actually is

TL;DR · The KV cache stores the keys and values of every earlier token so decode doesn't recompute them — and it only ever grows.

Last lesson the staircase loomed: each new token re-reads the whole past. The KV cache is how the engine stops re-reading — by remembering. But remembering costs memory, and that memory is the quiet thing that decides how many users you can serve.

Keys and values, kept around

During decode, attention compares the new token against every earlier token. The KV cache stores the keys and values of every earlier token so the model avoids recomputing them on each step. It grows linearly with sequence length and batch size.

Query, key, value — three roles

Each token produces three vectors. The query asks "what am I looking for?", the key answers "how do I label myself?", and the value is "the info I hand over if picked." You re-make the query each step, but keys and values of the past never change — so you cache them.

It only grows

One new token means one new key and one new value at every layer, appended to the pile. Nothing is freed mid-sequence. The longer the conversation runs, the taller the cache for that one sequence.

Keywords — tap to unfold the plain meaning

Analogy ❝Recall the line cook whose pantry was hauled across town for every bite — the weights, fetched fresh each step. The KV cache is a different pile: your stack of recipe binders, one new page added per bite you cook this session. It only grows, and every van trip has to haul the whole stack. The pantry is fixed; the binder stack keeps thickening as the order runs long.

The KV cache stores past keys and values so decode skips recomputation. It grows linearly with context length and batch — a pile that only gets taller.

02

The memory formula, decoded

TL;DR · Two lines: per-token bytes are fixed by the model; total cache is that times sequence length times batch — and both factors multiply.

You don't need to profile anything to know how big the cache gets. The model's config has every number you need. Two short lines turn that config into gigabytes.

Line one — fixed by the model

The bytes one token costs is set entirely by the architecture: 2 for both K&V, times the number of layers, times kv_heads × head_dim, times bytes/elem. You can't change this at serving time — it's baked into the weights you loaded.

Line two — what you control

Total cache is per-token bytes × seq_len × batch. Sequence length and batch size are knobs you set at serving time, and both multiply. Double the context or double the concurrency and the cache doubles with it.

The two pieces of the config

kv_heads is how many heads store a K/V per token (32 for Llama 2 7B). head_dim is the length of one head's K (or V) vector (128 numbers). Multiply them and you get the width of one layer's key vector.

Keywords — tap to unfold the plain meaning

Math, decoded

bytes/token = 2 × layers × (kv_heads × head_dim) × bytes/elem

total cache = bytes/token × seq_len × batch

2one slot for the key and one for the value — K & V are stored per token
layersevery transformer layer keeps its own K/V, so the pile multiplies by depth
kv_heads × head_dimthe width of one layer's key (or value) vector for one token
bytes/elemhow many bytes each number takes — FP16 = 2, FP8 = 1
seq_lenhow many tokens are in the sequence; the cache grows one slot per token
batchhow many concurrent sequences share the GPU — each carries its own cache

The first line is fixed by the model. The second line is what you control at serving time — and both factors multiply, which is why long contexts and big batches blow up memory so fast.

Per-token bytes = 2 × layers × (kv_heads × head_dim) × bytes/elem. Total = that × seq_len × batch. The model fixes line one; you set line two.

03

Worked example: Llama 2 7B

TL;DR · Plug Llama 2 7B's config into the formula: 0.5 MiB/token → 2 GiB per 4K sequence → a batch of 8 already eats 16 GiB of cache.

Numbers make it real. Take one well-known model, drop its config into the two lines, and watch the cache outgrow the weights faster than you'd guess.

The config

Llama 2 7B has 32 layers, kv_heads = 32, head_dim = 128, served in FP16 (2 bytes per element). That's plain MHA — multi-head attention, one KV head per query head.

Per token, then per sequence

Per token = 2 × 32 × 4096 × 2 = 524,288 B = 0.5 MiB. At a 4,096-token context, one sequence holds 0.5 MiB × 4096 = 2 GiB of cache. Note kv_heads × head_dim = 32 × 128 = 4096.

Batch of 8 vs the weights

A batch of 8 sequences needs 16 GiB of cache — and the weights themselves are only ≈ 14 GiB. The cache has already overtaken the model. On an 80 GB GPU that leaves ~66 GiB, which is ~66 ÷ 2 ≈ 33 concurrent 4K sequences before you run out.

Math, decoded — Llama 2 7B, FP16

per token = 2 × 32 × (32 × 128) × 2 = 524,288 B = 0.5 MiB

@ 4,096 ctx → 0.5 MiB × 4096 = 2 GiB/seq · batch 8 → 16 GiB · weights ≈ 14 GiB

32 × 128kv_heads × head_dim = 4096 numbers wide per layer per token
0.5 MiBwhat one token of context costs across all 32 layers, K and V
2 GiB/seqa full 4K-token sequence's cache — for a single user
~33 seqs~66 GiB left on an 80 GB GPU ÷ 2 GiB per seq ≈ 33 concurrent 4K sequences

At batch 8 the 16 GiB of cache already exceeds the ≈14 GiB of weights. Memory left after the weights — not the weights — is what decides how many sequences fit.

Keywords — tap to unfold the plain meaning

Llama 2 7B: 0.5 MiB/token, 2 GiB per 4K sequence, 16 GiB at batch 8 — already bigger than its ~14 GiB of weights.

04

Two levers: fewer bytes, fewer KV heads

TL;DR · Shrink the cache two ways: store each number in fewer bytes (quantize to FP8), or store fewer KV heads (Grouped-Query Attention).

Look back at line one of the formula. Only two factors there are negotiable: bytes/elem and kv_heads. Every modern model attacks one or both — and that's the whole story of cheap long-context serving.

Lever one — fewer bytes per element

Quantize the cache: store each K/V number in FP8 (1 byte) instead of FP16 (2 bytes) and the cache halves outright. It's the same vectors, just stored at lower precision.

Lever two — fewer KV heads

Grouped-Query Attention (GQA) keeps all the query heads (so the model stays just as expressive) but lets a group of them share one key/value head. Fewer kv_heads means a directly smaller cache.

The MHA → GQA → MQA spectrum

MHA: 32 query, 32 KV (1:1) — full cache, e.g. Llama 2 7B. GQA: 32 query, 8 KV (4:1 sharing) — 4× smaller, e.g. Llama 3 8B. MQA: 32 query, 1 KV (all share) — tiny cache. Same query expressiveness, ever-shrinking KV.

Query heads stay at 32 the whole way across, so the model stays expressive. Only the KV heads shrink — and the cache shrinks with them.

Keywords — tap to unfold the plain meaning

Two cache levers, both in line one of the formula: fewer bytes per element (FP8) and fewer KV heads (GQA). Query heads stay full, so quality holds.

05

On your cluster: Qwen3.6-27B-FP8

TL;DR · Both levers, live: GQA (24→4 KV heads) and FP8 cut Qwen3.6-27B to 128 KiB/token — yet one 128K request still eats 16 GiB of KV.

The real box uses both levers at once. Watch the per-token cost collapse — and then watch one maxed-out request swallow it right back up.

The config in play

Qwen3.6-27B-FP8 on a 94 GB H100 NVL: 64 layers, 4 KV heads (GQA, 24→4 = 6× reduction), head_dim = 256, and kv FP8 = 1 byte. Both levers, applied together.

Per-token cost collapses

Per token = 2 × 64 × (4 × 256) × 1 = 128 KiB — versus 256 KiB at FP16. But scale it up: a 128K-token request needs 128 KiB × 131,072 = 16 GiB of KV for one sequence.

Math, decoded — Qwen3.6-27B-FP8

per token = 2 × 64 × (4 × 256) × 1 = 128 KiB (vs 256 KiB at FP16)

128K-token request → 128 KiB × 131,072 = 16 GiB KV for ONE sequence

4 × 256just 4 KV heads (GQA 24→4) times head_dim 256 = 1024 wide per layer
× 1FP8 stores each number in 1 byte instead of 2 — halves the cache
128 KiBper-token cache after both levers, down from 256 KiB at FP16
16 GiBone maxed 128K-context request still fills 16 GiB of the KV pool

Both levers shrink the per-token cost sharply — but multiply by a 128K context and a single sequence still claims 16 GiB. The levers buy headroom; they don't make context free.

Cluster note On the 94 GB H100 NVL: GPU memory utilization 0.75 → ~70 GiB for vLLM, of which weights take ~27 GiB, leaving a KV pool of ~43 GiB ≈ 2.7 full-context requests. That's the real reason --max-num-seqs 8 and --max-model-len 131072 can't both be maxed: the cache, not the weights, is the limit. In practice, observed KV usage runs only 0.2–0.5% — RAG prompts are far shorter than the 128K ceiling.

The flags that collide — Qwen3.6-27B-FP8

vllm serve Qwen3.6-27B-FP8 \
  --gpu-memory-utilization 0.75 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 8 \
  --max-model-len 131072

Keywords — tap to unfold the plain meaning

Qwen3.6-27B-FP8: 128 KiB/token after GQA + FP8, yet a 128K request is 16 GiB. A ~43 GiB KV pool holds only ~2.7 full-context requests.

06

The cache, not the weights, caps the batch

TL;DR · Weights load once and sit still; every concurrent request adds its own growing cache. When the caches fill the memory left over, no more requests fit.

Here's the punchline that should reshape how you reason about capacity: the weights are not the bottleneck. The pile of caches is. Once you see it, every batch-size limit makes sense.

Weights are a fixed display

The shelves — HBM, the GPU's memory — are a fixed size. The weights are a permanent display you can't move: loaded once, sitting still. Everything left over is the KV pool.

Each customer eats the leftover shelf

Every concurrent sequence's binders eat the shelf space that's left. Run out, and you can't seat another customer. That's the cap: not the model's size, but how many caches fit in whatever HBM the weights didn't already claim.

Analogy ❝The kitchen's shelves (HBM) are a fixed size. The weights are a permanent display you can't move — bolted to the wall, eating their share of every shelf. Every concurrent customer brings their own stack of recipe binders (their KV cache), and each stack grows as their order runs long. Those binders eat the shelf space that's left over. Run out of shelf, and you simply can't seat another customer — no matter how small the next order is.

SRE note Ever notice a Claude Code session is cheap on repeated turns but pricey to resume the next day? That's this exact KV cache — persisted across requests, given a TTL, and put on a price tag. Same pile, just rented out instead of rebuilt. (Covered in Lesson 20, Prefix Caching & the KV Hierarchy.)

Keywords — tap to unfold the plain meaning

Check yourself

From config alone, write the two lines that give a model's total KV-cache size — and say which line you control at serving time.
Llama 2 7B at FP16: why does a batch of 8 at 4K context (16 GiB cache) already exceed the ~14 GiB of weights?
Name the two levers that shrink the cache, and which factor of the formula each one attacks.
Finish the sentence for a colleague: "We can only batch ~N requests because…"

Each concurrent sequence needs its own KV cache, which grows with context length. Once the caches fill the GPU memory left after the weights, no more requests fit — the cache, not the weights, sets the batch cap.