Inference Engineering · Lesson 10 · One Inference Request, End to EndHome · Glossary · Your Lab

One Inference Request, End to End

From HTTP JSON to streamed text and usage counters.

Commit a prediction before revealing the model.
Today's win: trace every ownership boundary in one request and locate where queueing, TTFT (time to first token), TPOT (time per output token), cancellation, and token accounting arise.

The setup

An order is validated at the door, rewritten onto the kitchen's ticket format, admitted to a station, prepped in one batch, then plated one bite at a time until the stop marker. The receipt counts both ingredients in and bites out.

1 · Validate and render the conversation

The API validates model name and limits, then applies the model's chat template. System, user, assistant roles and special tokens become one exact string. A template mismatch can hurt quality even when the server is healthy.

2 · Tokenize, admit, and schedule

The tokenizer converts the rendered string to IDs. The engine rejects an oversized context or places the request in a queue. Admission time plus queue delay occurs before GPU work and contributes to TTFT.

3 · Prefill once

The scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token. Long prompts, cache misses, and competing prefills usually dominate TTFT.

4 · Decode and stream

The engine repeatedly schedules one-token decode steps across active requests. Each token is selected, detokenized, and optionally sent as a stream event. TPOT/ITL measures the gaps.

5 · Stop, release, and account

Generation ends on EOS (EOS = the special end-of-sequence token a model emits to stop), a matched stop sequence, a generated-token limit, cancellation, timeout, or error. KV blocks (the KV cache is the per-request attention memory the model builds up while generating; computed in Lesson 14) are released and usage reports prompt, completion, and total tokens.

On YOUR cluster live-tested · course lab

The connected OpenShift cluster exposes Qwen through vLLM's OpenAI-compatible server. The course probe reads running/waiting requests and prompt/generation counters—the scheduler and phase boundaries in this request path.

Study next: sources & lab companionChat API shape · vLLM OpenAI-compatible server

Final check

← Lesson 9Lesson 11 →
References

Chat API shape · vLLM OpenAI-compatible server

One Inference Request, End to End

From HTTP JSON to streamed text and usage counters.

Today's win: trace every ownership boundary in one request and locate where queueing, TTFT (time to first token), TPOT (time per output token), cancellation, and token accounting arise.

The picture

An order is validated at the door, rewritten onto the kitchen's ticket format, admitted to a station, prepped in one batch, then plated one bite at a time until the stop marker. The receipt counts both ingredients in and bites out.

Validate and render the conversationThe API validates model name and limits, then applies the model's chat template
Tokenize, admit, and scheduleThe tokenizer converts the rendered string to IDs
Prefill onceThe scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token
Decode and streamThe engine repeatedly schedules one-token decode steps across active requests

1 · Validate and render the conversation

The API validates model name and limits, then applies the model's chat template. System, user, assistant roles and special tokens become one exact string. A template mismatch can hurt quality even when the server is healthy.

ONE REQUEST · where each latency number is born clientPOST /v1/chat validate+ chat template tokenizetext → IDs queueadmit · schedule prefillGPU · 1st logit TTFT · everything before the first token (queue delay counts) decode loop 1 token / step · detokenize · stream TPOT / ITL = the gap between tokens stop · release KV · account EOS / stop / limit / cancel / timeout usage = prompt + completion tokens
Notice TTFT covers the whole path before the first token, including queue delay, not just GPU prefill; once decoding starts, TPOT (a.k.a. ITL) measures the gap between tokens, and the KV blocks a request holds are only released when it terminates. This is why a healthy GPU can still show high TTFT under load.

2 · Tokenize, admit, and schedule

The tokenizer converts the rendered string to IDs. The engine rejects an oversized context or places the request in a queue. Admission time plus queue delay occurs before GPU work and contributes to TTFT.

THE BOUNDARY · validate, then render the conversation to one exact string raw request { model, messages: [system, user…], max_tokens, temp } JSON over HTTP validate known model? limits in range? else 400, no GPU apply chat template <|system|> You are… <|user|> Why is decode slow? <|assistant|> one exact string + special tokens a wrong template degrades quality even when the server is perfectly healthy
Notice the model never sees your JSON: the server validates it, then the model's own chat template turns the roles and special tokens into one exact string. This matters because a template mismatch (wrong role markers, missing special tokens) quietly hurts output quality with no error and no latency signal.

3 · Prefill once

The scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token. Long prompts, cache misses, and competing prefills usually dominate TTFT.

WHERE TTFT IS BORN · queue delay is latency the GPU never sees tokenizetext → IDs context checkprompt+max ≤ ctx? no → 400 queue others ahead prefillwhole prompt 1st tok TTFT queue delay + prefill compute under load the queue, not prefill, dominates TTFT — add capacity, not a faster kernel
Notice that the work before the first token is admission plus queueing plus prefill, and the queue is pure waiting the GPU never accounts for. This matters because under load a healthy, fast-prefill server still posts high TTFT — the cure is more capacity or shorter prompts, not a faster prefill kernel.

4 · Decode and stream

The engine repeatedly schedules one-token decode steps across active requests. Each token is selected, detokenized, and optionally sent as a stream event. TPOT/ITL measures the gaps.

DECODE → STOP → RELEASE · the long tail of one request decode loop TPOT terminal condition • EOS token • matched stop string • max_tokens reached • cancel / timeout / error any one ends generation release KVblocks freed for others usage = prompt + completion a cancelled or timed-out request must release its KV too, or capacity leaks billing and rate limits read the completion-token count produced here
Notice that several independent conditions can end a request, and only after one fires are the KV blocks released and usage tallied. This matters operationally: a client that disconnects must still trigger release, or abandoned requests hold KV and quietly shrink the batch every other request shares.

5 · Stop, release, and account

Generation ends on EOS (EOS = the special end-of-sequence token a model emits to stop), a matched stop sequence, a generated-token limit, cancellation, timeout, or error. KV blocks (the KV cache is the per-request attention memory the model builds up while generating; computed in Lesson 14) are released and usage reports prompt, completion, and total tokens.

On YOUR cluster live-tested · course lab

The connected OpenShift cluster exposes Qwen through vLLM's OpenAI-compatible server. The course probe reads running/waiting requests and prompt/generation counters—the scheduler and phase boundaries in this request path.

Check yourself

← Lesson 9Lesson 11 →
References

Chat API shape · vLLM OpenAI-compatible server