One Request, End to End — Inference Engineering

01

The five-stage journey

TL;DR · One request walks through five stages: validate & render, tokenize & schedule, prefill once, decode & stream, then stop & account.

You've seen the autoregressive loop in isolation. Now watch a real request carry it: HTTP JSON comes in one side, streamed text and a usage receipt come out the other. Everything in between is these five stages.

Input and output, named

A request arrives as HTTP JSON — a model name, a prompt or chat messages, and limits. It leaves as streamed text plus a usage tally counting tokens in and tokens out. The five stages turn one into the other.

The whole path at a glance

Stage 1 validates and renders the request into one exact string. Stage 2 tokenizes, admits, and schedules it. Stage 3 runs prefill once. Stage 4 decodes and streams token by token. Stage 5 stops, releases, and accounts.

The whole lesson on one line. Stages 1–3 and 5 each happen once; stage 4 is the autoregressive loop, run once per generated token.

Keywords — tap to unfold the plain meaning

Analogy · the kitchen ❝An order is validated at the door, rewritten onto the kitchen's ticket format, admitted to a station, prepped in one batch, then plated one bite at a time until the stop marker. The receipt counts both ingredients in and bites out. That's the whole request — and we'll follow the same order through each stage.

Five stages turn HTTP JSON into streamed text plus a usage receipt: validate & render, tokenize & schedule, prefill once, decode & stream, stop & account.

02

Stage 1 — validate & render the template

TL;DR · The API checks the model name and limits, then applies the chat template so roles and special tokens collapse into one exact string.

Before any GPU work, the request has to be made legal and made concrete. Two checks, then one rewrite — and the messy structured input becomes a single flat string the model can read.

Validate first

The API validates the request: is the model name one this server actually hosts, and are the limits (like max tokens) within bounds? A bad name or an impossible limit is rejected here, before any compute is spent.

Render the chat template

Then it applies the model's chat template: the structured system, user, and assistant roles plus any special tokens are stitched into one exact string. The same conversation rendered with the wrong template becomes a different prompt.

Keywords — tap to unfold the plain meaning

Analogy · at the door ❝The order is validated at the door — is this even on the menu? — then rewritten onto the kitchen's own ticket format. The customer's words and the kitchen's shorthand are not the same; the template is what translates one into the other before the ticket ever reaches a station.

Validate the model name and limits, then render roles and special tokens through the chat template into one exact string. Wrong template, wrong prompt.

03

Stage 2 — tokenize, admit, schedule

TL;DR · The tokenizer turns the string into IDs; the engine rejects an oversized context or queues the request — and under load, that queue is where time goes.

Now the exact string meets the engine. It either fits and gets a slot, or it's too big and bounced, or it waits in line. That wait is the part most people forget — and the part that dominates latency when the server is busy.

Tokenize to IDs

The tokenizer converts the rendered string into a list of integer token IDs — the only form the model reads. This also fixes the prompt's length in tokens, which the next checks depend on.

Admit or reject

The engine checks the prompt against the model's context window. An oversized context — prompt plus requested output exceeding the limit — is rejected outright. There's no room to run it, so it never starts.

Schedule into the queue

If it fits, the scheduler places the request in a queue. When a GPU slot opens, it's admitted to run. Under load, requests can wait here for many milliseconds before any model work begins.

Keywords — tap to unfold the plain meaning

Cluster note Under load, the queue — not prefill — dominates TTFT. A healthy, fast-prefill server can still post high time-to-first-token simply because requests sit in the queue. The cure is more capacity or shorter prompts, not a faster prefill kernel. Observed across the 4×H100 box: when concurrency climbs, queue delay swamps the compute it gates.

Tokenize to IDs, reject an oversized context, otherwise queue and wait for a slot. Under load the queue, not the kernel, owns your TTFT.

04

Stage 3 — prefill once

TL;DR · Prefill runs all prompt tokens through the model in one batch, fills the KV cache, and produces the logits for the first output token.

This is the one big push. Every prompt token is processed together in a single pass — and the by-product, the KV cache, is what makes every later step cheap.

One batched pass over the prompt

The scheduler batches the prompt tokens and runs the model over all of them at once. This is prefill: a single forward pass that processes the entire prompt before any output token exists.

Populate the KV blocks

Prefill populates the KV blocks — the per-request KV cache, the attention memory the model builds up while generating. Storing it now means later steps never re-read the whole prompt.

First-token logits

Prefill ends by producing the logits for the first output token. The clock from request arrival to this point is TTFT — and it bundles queue delay together with prefill compute.

Math, decoded

TTFT = queue delay + prefill compute

TTFTtime to first token — request arrival until the first output token is ready
queue delayhow long the request waited in the scheduler before a GPU slot opened
prefill computetime to run the whole prompt through the model in one batched pass

TTFT is not just compute. It's the wait plus the work — which is exactly why, under load, adding capacity (shrinking the wait) helps more than a faster kernel (shrinking the work).

Keywords — tap to unfold the plain meaning

Analogy · prepped in one batch ❝The whole order is prepped in one batch — every ingredient on the ticket laid out together before a single bite is plated. That prep work is stored on the station (the KV cache) so the cook never re-reads the ticket from the top again. The first bite can't leave until the prep is done; that wait-plus-prep is your TTFT.

Prefill processes all prompt tokens in one batch, fills the KV cache, and yields the first-token logits. TTFT = queue delay + prefill compute.

05

Stage 4 — decode & stream

TL;DR · The engine schedules one-token decode steps across all active requests; each token is selected, detokenized, and optionally streamed as an event.

Prefill happened once. Now the loop runs: one token at a time, interleaved across every request the server is juggling. The gap between those tokens has its own name — and it's what makes a stream feel fast or sluggish.

One-token steps, interleaved

The engine repeatedly schedules one-token decode steps across all active requests. Each step advances every running request by a single token, sharing the GPU instead of finishing one request before starting the next.

Select, detokenize, stream

For each request, a token is selected from the logits, detokenized back into text, and — if streaming — sent as a stream event. The user sees text appear token by token rather than all at once.

The gap between tokens

The time between two successive generated tokens is TPOT (time per output token), also called ITL (inter-token latency). It's the steady drip rate of the stream once the first token has landed.

TTFT covers the wait and the first big push. After that, evenly spaced decode steps — each gap is TPOT/ITL — drip out the rest of the answer.

Keywords — tap to unfold the plain meaning

Decode schedules one-token steps across active requests; each token is selected, detokenized, and streamed. The gap between tokens is TPOT / ITL.

06

Stage 5 — stop, release, account

TL;DR · Generation ends on one of several conditions; the KV blocks are freed and a usage receipt tallies tokens in and tokens out.

Every stream stops somehow. When it does, the memory the request was holding has to go back, and the bill has to be written. This is the cleanup that lets the next request take the slot.

Six ways to stop

Generation terminates on any of: the EOS token, a matched stop sequence, the token limit, a client cancellation, a timeout, or an error. Whichever fires first ends the loop for that request.

Release the KV blocks

Once stopped, the request's KV blocks are freed — the attention memory it built up is returned to the pool so a waiting request can use it. Holding memory after the answer is done is pure waste.

Account the usage

Finally, usage is tallied: tokens in (the prompt) and tokens out (the generation). This is the receipt — the same count the kitchen ticket keeps of ingredients in and bites out.

Keywords — tap to unfold the plain meaning

Analogy · the receipt ❝The plating stops at the stop marker — the dish is done, the order's cancelled, or the kitchen calls time. The station is wiped clean for the next order (the KV blocks freed), and the receipt counts both ingredients in and bites out. Nothing leaves the kitchen without that final tally.

Check yourself

Name the five stages a request passes through, in order.
What does TTFT bundle together — and why does that make queue delay matter?
Under load, what actually fixes a high TTFT: a faster prefill kernel, or more capacity?
What's the difference between TTFT and TPOT / ITL?
What two things happen at stop time besides ending the loop?

Stop on EOS, stop sequence, token limit, cancel, timeout, or error; free the KV blocks; tally tokens in and out. The receipt closes the request.