Prefix Caching & the KV Hierarchy — Inference Engineering

01

Reuse the shared opening

TL;DR · Many requests start with the same long opening; prefill its KV once, then reuse those blocks so later requests skip straight to the new part.

Look at real traffic and most prompts aren't unique — they share a fat, identical opening. Prefix caching is the runtime noticing that and refusing to do the same prefill twice.

Most prompts share a head

A system prompt, a few-shot block, a tool spec — these sit at the front of every request and rarely change. The model's expensive prefill computes KV for that shared head over and over, even though the result is byte-for-byte identical each time.

RAG makes it worse — and better

With RAG, retrieved documents are fetched and pasted into the prompt, which makes prompts long and largely shared. Long shared prefixes are exactly the workload prefix caching was built to win.

Compute the shared KV once

Prefix caching prefills the shared opening a single time, stores its KV blocks, and reuses them. A new request only pays to process the part that's actually new — so TTFT drops.

Keywords — tap to unfold the plain meaning

Analogy ❝It's the shared base sauce, made once. The pantry already has the sauce simmering, so a new order only needs its finishing touches — and the first bite arrives much sooner. You don't re-make the base from scratch for every ticket; you ladle from the pot that's already hot and just add what this dish needs.

Shared openings — system prompts, few-shot, RAG context — produce identical KV. Prefill it once, reuse it, and a new request only pays for what's new.

02

Hash the prefix, hit the cache

TL;DR · The engine hashes each incoming prefix; on a hit it reuses the cached KV blocks and skips re-prefilling them — only the new suffix gets computed.

The mechanism is almost boring, which is why it works: turn the prefix into a key, look it up, and if it's there, don't redo the work.

Hash in, lookup out

The engine hashes each incoming prefix. If that hash is already in the cache — a hit — it reuses the cached KV blocks and skips re-prefilling them. If it's not there — a miss — it does the full prefill and stores the result for next time.

Only the new part is computed

On a hit, prefill resumes at the first token after the shared prefix. The user waits only for the new question to be processed, not the whole repeated opening — which is where the TTFT win comes from.

Hash the prefix, look it up. A hit reuses the cached KV blocks so prefill only has to run over the new suffix; a miss prefills everything and stores it.

Keywords — tap to unfold the plain meaning

Math, decoded

prefill work_hit ∝ L_new instead of L_prefix + L_new

L_prefixlength of the shared opening — system prompt, few-shot, RAG context
L_newlength of the new suffix — the user's actual question this time
∝ L_newon a hit, prefill cost scales only with the new part; the shared prefix is free

Without caching, prefill processes the whole prompt (prefix + new). On a hit, it processes only the new suffix — so when the shared prefix dominates, TTFT drops a lot.

Hash the prefix; a hit reuses cached KV blocks and skips re-prefilling them. Prefill resumes at the new suffix, so the user only waits for what's new.

03

The KV memory hierarchy

TL;DR · Cached KV can live in VRAM, host RAM, or SSD — faster-and-smaller down to bigger-and-slower. If a lower tier is slower to fetch than to re-prefill, just re-prefill.

VRAM is tiny, so you can't keep every prefix hot forever. The hierarchy is how you keep more cache around — but only as far down as it still pays off.

Three tiers, getting colder

Cached KV spans a hierarchy: VRAM (hottest, faster, smaller) → host RAM → local SSD (bigger, slower). The deeper you go, the more you can store, but the longer it takes to fetch a block back.

The break-even rule

Fetching a cached block from a lower tier isn't always worth it. The rule: if a lower tier is slower to fetch than to re-prefill, just re-prefill. A cache only helps when reading it back beats recomputing it.

VRAM → host RAM → local SSD: faster-and-smaller down to bigger-and-slower. The deeper the tier, the more it has to beat a plain re-prefill to be worth fetching.

Keywords — tap to unfold the plain meaning

Math, decoded

fetch only if t_fetch < t_reprefill

t_fetchtime to read the cached KV block back from a lower tier (RAM or SSD)
t_reprefilltime to just recompute that block's KV with a fresh prefill
<the cache only helps when the fetch is genuinely faster than redoing the work

A lower tier is worth using only while fetching from it is faster than re-prefilling. Past that break-even, the "cache" is slower than recomputation — so you re-prefill.

Analogy ❝Think fridge → pantry → cold storage. The fridge is right at the station; the pantry is a few steps away; cold storage is in the basement. If the sauce is in cold storage and the trip down and back takes longer than just simmering a fresh batch at the station, you simmer fresh. A faraway pantry only saves time while the walk is shorter than the cooking.

Cached KV lives across VRAM → host RAM → SSD, getting bigger and slower. If fetching from a lower tier is slower than re-prefilling, re-prefill instead.

04

RadixAttention & partial overlap

TL;DR · SGLang generalizes prefix caching with RadixAttention — a radix tree of all live prefixes — so even partial overlaps between prompts can share cached KV.

Exact-match caching only helps when whole openings are identical. A tree lets prompts that share part of an opening still split the cost at the branch point.

A tree of shared prefixes

SGLang generalizes prefix caching with RadixAttention — a radix tree of all live prefixes. Common openings sit near the root; where two prompts diverge, the tree branches.

Even partial overlaps share

Because matches happen along the tree, prompts don't need identical openings to benefit — even partial overlaps share. Two requests that agree for the first few thousand tokens reuse that shared run, then branch where they differ.

Keywords — tap to unfold the plain meaning

Analogy ❝One base sauce, many finishes. Two tickets share the same simmering base, then one branches to add basil and the other to add chili. You don't keep two separate pots from the start — you keep one shared pot and only split into separate pans at the moment the recipes diverge. The shared simmering is paid for once; only the finishing pans are per-ticket.

RadixAttention (SGLang) keeps a radix tree of all live prefixes, so prompts that share even part of an opening reuse the common run and branch where they differ.

05

Routing must be cache-aware

TL;DR · Prefix caches are per-replica. If the load balancer sends a request to a replica that doesn't hold its prefix, it's a miss — full re-prefill, no savings.

You can build the perfect cache and still get zero benefit if the router scatters matching requests across replicas. The cache only pays off where the request actually lands.

Caches are per-replica

Prefix caches are per replica. Each server holds its own cached KV; a block cached on replica A is invisible to replica B. So a hit depends not just on whether a prefix is cached, but on where.

A blind router throws the win away

If your load balancer sends a request to a replica that doesn't hold its prefix, it's a miss: full re-prefill, no savings. The fix is cache-aware routing — send matching requests to the same replica so they actually hit.

A block cached on Replica A is invisible to Replica B. Cache-aware routing sends matching requests to the holder so they hit; a blind balancer scatters them into full re-prefills.

Keywords — tap to unfold the plain meaning

Analogy ❝Send the order to the station with the sauce. Only one cook's pot is already simmering the shared base; route the matching ticket there and the first bite comes fast. Hand it to a cold station instead and that cook has to simmer the whole base from scratch — the pot two stations over does you no good at all.

Prefix caches are per-replica. A blind load balancer turns hits into misses; cache-aware routing sends matching requests to the same replica so the cache actually pays off.

06

On your cluster

TL;DR · Prefix caching is on in your vLLM, and your traffic is heavily prefill-heavy — the ideal workload for it to win.

This isn't a hypothetical optimization for your box. The flag is already set, and the traffic shape is exactly the one prefix caching was built for.

Turn it on (it already is)

In vLLM, prefix caching is enabled with --enable-prefix-caching. On the cluster this is confirmed in the server args — your vLLM runs it.

Your traffic is the ideal case

Measured on the box, traffic is 19–47:1 prefill-heavy: each request prefills far more tokens than it decodes. Long shared openings plus a heavy prefill ratio is precisely where caching the prefix pays off most.

vLLM server arg (confirmed on the 4×H100 box)

vllm serve Qwen3.6-27B-FP8 \
  --enable-prefix-caching

Cluster note Live-tested 2026-06-19 on the 4×H100 box. --enable-prefix-caching is ON (confirmed in the server args), and traffic measured at 19–47:1 prefill-heavy — the ideal prefix-caching workload. With a fat shared prefix and that ratio, hits skip re-prefilling the opening and TTFT drops.

Keywords — tap to unfold the plain meaning

Check yourself

Explain to a colleague: "Prefix caching helps our RAG app because…" — what makes the KV identical, what gets reused, and what the user no longer waits for?
Cached KV can sit in VRAM, host RAM, or SSD. What's the rule for when you should stop fetching from a lower tier and just re-prefill?
Why does prefix caching need cache-aware routing to actually pay off — what happens on a replica that doesn't hold the prefix?

Your vLLM already runs --enable-prefix-caching, and 19–47:1 prefill-heavy traffic is the ideal case. Shared prefix + cache-aware routing = lower TTFT.