PagedAttention & Batching — Inference Engineering

01

Problem A: static batching idles the GPU

TL;DR · Static batching waits for every request in a batch to finish before starting the next — so the whole batch stalls on its slowest request, leaving roughly 60% of the GPU idle.

A GPU running one request at a time barely uses its compute. The obvious fix is to batch requests. But the obvious way to batch them leaves the GPU mostly empty — and the reason is the loop you already know from earlier lessons.

What static batching does

Static batching groups N requests, runs them together, waits for all of them to finish, then starts the next batch. Simple to build, easy to reason about — and quietly wasteful.

Why it idles

Requests finish at wildly different lengths: one wants 10 tokens, another wants 800. The whole batch stalls on the slowest one while the finished slots just sit there. Measured across a variable workload, roughly 60% of the GPU sits idle.

Keywords — tap to unfold the plain meaning

Analogy · the delivery van ❝Static batching is one shared delivery van that refuses to leave the depot until every customer's order on board is done. A customer who ordered a single coffee is stuck waiting on the customer who ordered a fifty-item banquet. Quick orders wait on slow ones, and the van's empty seats earn nothing the whole trip.

Static batching waits for the whole batch to drain before refilling, so it stalls on the slowest request and idles ~60% of the GPU.

02

Fix A: continuous batching

TL;DR · Continuous batching schedules one decode step at a time: after every step it evicts finished requests and admits waiting ones, so the batch never drains. Worth 4–8× on variable-length workloads.

If the problem is waiting for the whole batch to finish, the fix is to stop thinking in batches and start thinking in single steps. Decide who's in the batch fresh, every single step.

Schedule per step, not per batch

Continuous batching schedules one decode step at a time. After every step it evicts finished requests and admits waiting ones, so a slot freed by a short request is instantly reused — the batch never drains.

The name and the origin

This is iteration-level scheduling — the scheduling unit is one iteration (one decode step), not one whole request. It was introduced by Orca (Yu et al., OSDI'22).

What it's worth

On variable-length workloads — the realistic case — continuous batching is worth roughly 4–8× the throughput of static batching, just by never letting finished slots sit idle.

Same three slots, two policies. Static leaves a slot empty once its request finishes; continuous batching admits a waiting request into that slot on the next decode step.

Keywords — tap to unfold the plain meaning

Analogy · the delivery van ❝Now the van drops finished orders and picks up waiting ones on every single trip. The moment one customer's order is complete, their seat goes to the next person in line — so the van rolls out the depot always full instead of waiting for the banquet order to finish.

Continuous batching (iteration-level scheduling, from Orca) refills the batch every decode step instead of waiting for it to drain — worth 4–8×.

03

Problem B: contiguous KV wastes memory

TL;DR · Early servers reserved one contiguous KV region per request, sized to the maximum possible length. Most requests never reach it, so 60–80% of reserved memory sits empty.

Continuous batching fixes the idle compute. But there's a second, quieter waste — in memory, not compute — and it limits how many sequences you can fit at once no matter how clever your scheduler is.

Reserve-for-the-worst-case

Early servers gave each request one contiguous KV region, sized to the maximum possible length the request might generate. The KV cache must live somewhere, so they grabbed the whole worst-case slab up front.

Most of it never gets used

Most requests never reach that maximum. So 60–80% of reserved memory sits empty, and the box fits far fewer sequences than the raw KV math promised — a memory problem, not a compute one.

Keywords — tap to unfold the plain meaning

Analogy · the pantry shelf ❝It's like reserving a whole pantry shelf for each customer, just in case they order the maximum. Almost nobody does — so the shelves fill the storeroom while standing mostly empty. You run out of shelves long before you run out of food.

Contiguous KV pre-reserves the worst-case length per request, so 60–80% of reserved memory sits empty and you fit far fewer sequences.

04

Fix B: PagedAttention

TL;DR · PagedAttention splits the KV cache into small fixed-size blocks handed out on demand — non-contiguous, as each sequence grows. Waste drops to nearly zero. With continuous batching it gave vLLM up to 24× naïve HuggingFace throughput.

The trick is borrowed straight from operating systems. You don't hand a program one giant contiguous slab of RAM — you hand it pages on demand. Do the same to the KV cache and the empty-shelf waste disappears.

Blocks on demand

PagedAttention splits the KV cache into small fixed-size blocks, handed out on demand as each sequence actually grows. The blocks are non-contiguous — they can sit anywhere in memory — so nothing is reserved before it's needed and waste drops to nearly zero.

Where the idea comes from

It borrows virtual memory / OS paging: just as an operating system maps scattered physical pages into one tidy address space, PagedAttention maps scattered KV blocks into one logical sequence.

The combined payoff

Together with continuous batching, PagedAttention gave vLLM up to 24× the throughput of naïve HuggingFace serving. It was introduced by Kwon et al., SOSP'23 (arXiv 2309.06180).

Contiguous KV nails down a worst-case slab per request and leaves most of it empty. PagedAttention scatters small fixed-size blocks across a shared pool, handed out only as each sequence grows.

Keywords — tap to unfold the plain meaning

Analogy · the pantry rack ❝Instead of a whole shelf per customer, you hand out small bins from a shared rack — and only when an order actually grows enough to need one. The bins don't have to sit next to each other; you just remember which bins belong to which order. Almost no empty space, and the storeroom holds far more orders.

PagedAttention is OS paging for the KV cache: small fixed-size non-contiguous blocks on demand, waste ≈ 0. Plus continuous batching, vLLM hit up to 24× naïve HF.

05

Both fixes, one batch

TL;DR · Two requests share one weight-read but keep separate KV caches. Continuous batching swaps them in and out each step; PagedAttention pages their KV across iterations; chunked prefill keeps long prompts from blocking decode.

The two fixes attack different bills — compute and memory — and they compose. Walk one batch with two requests through both, and you can see exactly which cost is shared and which is paid per sequence.

Shared compute, separate memory

Both requests prefill, then enter the same decode batch. The big win: one weight-read serves both requests (shared compute), but each maintains its own KV cache (a memory cost paid per sequence). That split is the whole reason batching helps.

The two fixes do their jobs

Across the timeline, continuous batching swaps finished requests out and waiting ones in each step, while PagedAttention hands out KV blocks as each sequence grows — paging across iterations so neither request over-reserves.

Chunked prefill

Chunked prefill splits a long prompt into smaller pieces so prefill interleaves with other requests' decode in the same batch — instead of one long prompt freezing everyone else's token generation.

Shared vs per-seq Weights are read once per step for the whole batch — adding a second request is nearly free on the compute side. KV cache is per sequence — each request you add costs more memory, which is exactly why PagedAttention's near-zero waste decides how many you can pack in.

Keywords — tap to unfold the plain meaning

One weight-read serves the whole batch (shared); each request keeps its own KV (per-sequence). Continuous batching swaps, PagedAttention pages, chunked prefill interleaves.

06

On your cluster

TL;DR · A few vLLM flags turn these ideas on; two gauges tell you if the batch is full; and the same paging idea is what makes API prompt caching ~90% cheaper.

All of this ships in vLLM as plain flags. You set the batch width, turn on prefix and chunked prefill, watch two numbers — and you can see the same KV-paging idea show up as a price on the hosted APIs.

The flags that turn it on

--max-num-seqs sets the continuous-batch width (e.g. 8 or 64). --enable-prefix-caching shares KV blocks across requests with the same prefix. --enable-chunked-prefill keeps long prefills from blocking decodes.

The live gauge

Watch num_requests_running versus num_requests_waiting. Running well below your --max-num-seqs while many wait means the batch isn't full — you're leaving throughput on the table.

vLLM flags on the 4×H100 box

vllm serve Qwen3.6-27B-FP8 \
  --max-num-seqs 64 \
  --enable-prefix-caching \
  --enable-chunked-prefill

# live gauge — is the batch full?
num_requests_running   # active decode slots, vs --max-num-seqs
num_requests_waiting   # queued, not yet admitted

The same idea, billed

Hosted-API prompt caching stores the prefill KV for a stable prefix so the next request skips re-prefilling it — the same KV-reuse from --enable-prefix-caching, now a line on your bill.

Prompt caching Cache reads ≈ 0.1× the input price (~90% off); cache writes cost 1.25× for a 5-minute TTL, or 2× for a 1-hour TTL. Mechanically it's PagedAttention's blocks reused: store the prefix's prefill KV once, then later requests with that same prefix skip re-prefilling.

Cluster note On the 4×H100 box, static batching left roughly 60% of the GPU idle on a variable-length workload. Continuous batching is worth 4–8× there; PagedAttention cut KV waste from 60–80% toward zero; together they put vLLM at up to 24× naïve HuggingFace throughput.

Keywords — tap to unfold the plain meaning

Check yourself

Why does static batching idle the GPU?
What is continuous batching?
What does PagedAttention borrow?

Set --max-num-seqs, enable prefix and chunked prefill, and watch running vs waiting. The same KV-paging trick is what makes prompt caching ~90% cheaper.