What Is an Inference Engine? — Inference Engineering

01

The naive loop isn't enough in production

TL;DR · A plain model.generate() loop serves one request at a time — fine in a notebook, but it wastes an expensive GPU when thousands of users are waiting.

Today's win: you'll predict why the naive loop isn't enough in production, and what an inference engine adds. The loop itself is correct — it's the serving around it that's missing.

Notebook to production is a different problem

In a notebook you call model.generate() and wait. In production you serve thousands of users on a costly GPU, each with a latency target. The naive loop has no answer for "many users at once."

One request at a time wastes the GPU

A plain generation loop serves one request at a time. While it grinds through generation for one user, everyone else waits in line and the hardware sits underused.

The engine is the missing layer

An inference engine is the serving layer that wraps the model loop so many requests share the GPU at once, fit in memory, and arrive over a real API with metrics. That layer is the whole of Part II.

Keywords — tap to unfold the plain meaning

Analogy ❝The naive loop is one cook making one order start to finish: take the ticket, cook it, plate it, only then look at the next ticket. The dining room fills up while a perfectly good kitchen stands half-empty. An inference engine turns that one cook into a whole kitchen operation — an expediter calling orders, many plates in flight, the line never idle.

The naive loop is correct but serves one request at a time. Production needs an inference engine — the serving layer that keeps an expensive GPU busy for everyone.

02

Decode is memory-bound — so the GPU sits idle

TL;DR · Generating tokens for one user is memory-bound — limited by how fast data moves from GPU memory, not by math — so the GPU's compute mostly sits idle while everyone else waits.

This is the precise reason the naive loop wastes hardware. The bottleneck isn't the math being slow — it's the GPU waiting on memory, with nothing else to do.

Memory-bound, not compute-bound

One-user decode is memory-bound: limited by how fast data moves from GPU memory, not by how much math the chip can do. The math units finish early and wait on the next chunk of weights to arrive.

Idle compute is wasted money

While the naive loop does that memory-bound decode for one user, the GPU's compute sits mostly idle, and everyone else waits. You're paying for a fast chip that's mostly twiddling its thumbs.

The fix is more work in flight

If many requests run together, the same memory reads feed many users' math at once — so the idle compute fills up. That's batching, and it's the core trick the engine is built around.

Keywords — tap to unfold the plain meaning

Analogy ❝The pantry is at the far end of the building. A single line cook spends the whole shift walking to that faraway pantry and back for one order's ingredients — the chopping itself is instant, it's the trip that's slow. That long walk is memory bandwidth. The cure isn't a faster knife; it's carrying ingredients for many orders on each trip, so the same long walk feeds the whole line. Batching is loading up the cart.

Single-user decode is memory-bound — bottlenecked by data moving from GPU memory, not by math — so compute idles. Batching many requests onto each memory trip is the fix.

03

What an inference engine wraps around the model

TL;DR · An engine wraps the model loop with four things: a scheduler doing continuous batching, a paged KV-cache manager, an OpenAI-compatible API server, and metrics/SLOs.

"Engine" sounds vague until you list its parts. It's exactly four boxes bolted around the same next-token loop you already know — each one fixing a different production problem.

Scheduler with continuous batching

A scheduler keeps the GPU full by mixing many requests together. Continuous batching means new requests can join the running batch at any step, instead of waiting for a fixed batch to finish.

Paged KV-cache manager

The paged KV-cache manager stores each request's attention data in small reusable blocks, so many sequences fit in memory without pre-reserving a huge contiguous slab per request.

OpenAI-compatible API + metrics

An OpenAI-compatible API server exposes the model over standard HTTP endpoints, and metrics / SLOs expose latency and throughput so you can watch and meet service targets.

Requests arrive at an OpenAI-compatible API, the scheduler batches them continuously into the model loop, the paged KV manager holds their attention data, and metrics watch the SLOs.

Keywords — tap to unfold the plain meaning

An engine wraps the model loop with four boxes: scheduler + continuous batching, paged KV-cache manager, OpenAI-compatible API, and metrics/SLOs.

04

Four engines, four signature tricks

TL;DR · vLLM, SGLang, TensorRT-LLM, and Dynamo are four real engines, each known for one headline trick — and vLLM is the one this course runs.

You don't need all four today. But knowing each one's signature move tells you what problem it was built to solve — and why we picked vLLM as the kitchen's operating system.

vLLM — PagedAttention + continuous batching

vLLM is built around PagedAttention (KV cache stored in small reusable blocks) and continuous batching. It's the engine we run — think of it as your kitchen's operating system.

SGLang — RadixAttention

SGLang is known for RadixAttention: prefix sharing, so requests that begin with the same text reuse each other's cached work instead of recomputing it (covered later, in L20).

TensorRT-LLM & Dynamo

TensorRT-LLM leans on compiled kernels — hand-tuned GPU code for maximum speed. Dynamo is known for disaggregated serving: splitting stages across separate machines.

Keywords — tap to unfold the plain meaning

Analogy ❝Four kitchens, four reputations. vLLM packs its pantry into neat reusable bins so nothing's wasted. SGLang notices two tables ordered the same starter and plates it once. TensorRT-LLM has every station's moves drilled to the second. Dynamo splits prep and cooking into separate rooms. Same dish, different operating philosophy — and we cook in the vLLM kitchen.

vLLM = PagedAttention + continuous batching · SGLang = RadixAttention / prefix sharing · TensorRT-LLM = compiled kernels · Dynamo = disaggregated serving. We run vLLM.

05

Your lab setup: vLLM serving Qwen

TL;DR · Your box runs vLLM serving Qwen with an OpenAI-compatible API plus a tokenize and a metrics endpoint — and three tuning flags you'll meet again.

Everything above is concrete on your cluster. The same four-box engine is already running, exposing real endpoints you can curl and real flags you can turn.

The endpoints it exposes

vLLM serves an OpenAI-compatible API: /v1/completions and /v1/chat/completions for generation, /tokenize to inspect tokenization, and a Prometheus /metrics endpoint for the SLOs.

The flags you'll tune

Three tunable flags recur through Part II: --max-num-seqs (how many sequences batch together), --enable-chunked-prefill, and --kv-cache-dtype (the KV cache's number format).

Endpoints on the 4×H100 box (text, not live)

GET  /v1/completions        # generate from a raw prompt
POST /v1/chat/completions   # chat-format generation
POST /tokenize              # inspect token IDs
GET  /metrics               # Prometheus SLO metrics

# tunable launch flags
--max-num-seqs          # sequences batched together
--enable-chunked-prefill
--kv-cache-dtype        # KV cache number format

Cluster note On the 4×H100 box the vLLM deployment serves Qwen (the running deployments are named llm-serving-qwen36 / qwen35-27b-fixed). The same engine that exposes /v1/chat/completions also exposes /tokenize and Prometheus /metrics — one process, all four engine boxes.

Keywords — tap to unfold the plain meaning

Your lab is vLLM serving Qwen with an OpenAI-compatible API, plus /tokenize and /metrics, tuned with --max-num-seqs, --enable-chunked-prefill, and --kv-cache-dtype.

06

On your cluster — why vLLM, not generate()

TL;DR · Hit the same OpenAI-compatible endpoint your engine exposes, and you're talking to the wrapped loop — not a notebook call. That difference is the whole lesson.

One curl tells the story. The model loop is identical to the one from Part I; what changed is everything wrapped around it.

Talk to the engine, not the loop

Sending a request to /v1/chat/completions hands your prompt to the scheduler, which batches it with everyone else's, runs the forward pass, and streams tokens back — the four engine boxes, working as one.

Try it on the 4×H100 box (text, not live)

curl localhost:8000/v1/chat/completions -d '{
  "model": "Qwen",
  "messages": [{"role": "user", "content": "hi"}],
  "max_tokens": 16
}'

Why vLLM We run vLLM instead of plain model.generate() because an engine wraps the model loop with continuous batching (keep the GPU full with many requests), paged KV management (fit more sequences), an OpenAI-compatible API, and metrics.

Keywords — tap to unfold the plain meaning

Check yourself

Complete it: "We run vLLM instead of plain model.generate() because…"
Why does single-user decode leave the GPU's compute mostly idle?
Name the four things an inference engine wraps around the model loop.

We run vLLM, not model.generate(), because the engine adds continuous batching, paged KV management, an OpenAI-compatible API, and metrics around the same loop.