From HTTP JSON to streamed text and usage counters.
An order is validated at the door, rewritten onto the kitchen's ticket format, admitted to a station, prepped in one batch, then plated one bite at a time until the stop marker. The receipt counts both ingredients in and bites out.
The API validates model name and limits, then applies the model's chat template. System, user, assistant roles and special tokens become one exact string. A template mismatch can hurt quality even when the server is healthy.
The tokenizer converts the rendered string to IDs. The engine rejects an oversized context or places the request in a queue. Admission time plus queue delay occurs before GPU work and contributes to TTFT.
The scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token. Long prompts, cache misses, and competing prefills usually dominate TTFT.
The engine repeatedly schedules one-token decode steps across active requests. Each token is selected, detokenized, and optionally sent as a stream event. TPOT/ITL measures the gaps.
Generation ends on EOS (EOS = the special end-of-sequence token a model emits to stop), a matched stop sequence, a generated-token limit, cancellation, timeout, or error. KV blocks (the KV cache is the per-request attention memory the model builds up while generating; computed in Lesson 14) are released and usage reports prompt, completion, and total tokens.
The connected OpenShift cluster exposes Qwen through vLLM's OpenAI-compatible server. The course probe reads running/waiting requests and prompt/generation counters—the scheduler and phase boundaries in this request path.
From HTTP JSON to streamed text and usage counters.
An order is validated at the door, rewritten onto the kitchen's ticket format, admitted to a station, prepped in one batch, then plated one bite at a time until the stop marker. The receipt counts both ingredients in and bites out.
| Validate and render the conversation | The API validates model name and limits, then applies the model's chat template |
| Tokenize, admit, and schedule | The tokenizer converts the rendered string to IDs |
| Prefill once | The scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token |
| Decode and stream | The engine repeatedly schedules one-token decode steps across active requests |
The API validates model name and limits, then applies the model's chat template. System, user, assistant roles and special tokens become one exact string. A template mismatch can hurt quality even when the server is healthy.
The tokenizer converts the rendered string to IDs. The engine rejects an oversized context or places the request in a queue. Admission time plus queue delay occurs before GPU work and contributes to TTFT.
The scheduler batches prompt tokens, runs the model, populates KV blocks, and obtains logits for the first output token. Long prompts, cache misses, and competing prefills usually dominate TTFT.
The engine repeatedly schedules one-token decode steps across active requests. Each token is selected, detokenized, and optionally sent as a stream event. TPOT/ITL measures the gaps.
Generation ends on EOS (EOS = the special end-of-sequence token a model emits to stop), a matched stop sequence, a generated-token limit, cancellation, timeout, or error. KV blocks (the KV cache is the per-request attention memory the model builds up while generating; computed in Lesson 14) are released and usage reports prompt, completion, and total tokens.
The connected OpenShift cluster exposes Qwen through vLLM's OpenAI-compatible server. The course probe reads running/waiting requests and prompt/generation counters—the scheduler and phase boundaries in this request path.