Classify the symptom before turning knobs.
Lesson 22 set quality gates; this is the triage manual for when serving itself breaks. Next, Lesson 24 starts cutting bits per number.
A delayed meal can be stuck at the door, waiting for a station, missing ingredients, or burning on the stove. Start with where progress stopped; do not buy a faster oven for a ticketing failure.
Unknown model, malformed roles, tokenizer/chat-template mismatch, and prompt+max_tokens beyond context should fail before GPU work. Report token counts and limits, not a generic 500.
Weights may not fit at load; KV may exhaust under concurrency/context; runtime workspaces can spike; fragmentation can prevent a large allocation. Inspect free memory, configured utilization, active sequences, context lengths, and allocator logs.
A growing queue and rising TTFT with stable TPOT indicates admission/prefill pressure. Rising TPOT suggests decode batch, bandwidth, collectives (collectives = cross-GPU communication such as all-reduce; see Lesson 27), or noisy-neighbor pressure. Add capacity only after locating the violated phase.
Unsupported numeric format (dtype, e.g. FP8 vs FP16; see Lesson 24), compute capability (compute capability = the GPU's CUDA feature-level version), attention backend (attention backend = which attention kernel is used, e.g. FlashAttention; see Lesson 19), model architecture, or compiled shape (a compiled/padded shape = kernels built ahead of time for fixed tensor sizes; see Lesson 26) can fail at startup or silently fall back to a slow path. Record versions and the selected kernels/backends.
Readiness must wait for loaded weights and warmup. Cancellation and shutdown must stop admission, drain or abort active streams intentionally, release KV, and distinguish timeout from server failure.
The live cluster provides real examples from different layers: heavy GPU memory reservation, vLLM running/waiting/KV metrics, time-sliced GPUs, and a diagnosed NVLink placement fault. The symptom must be assigned to the right layer before tuning.
Classify the symptom before turning knobs.
A delayed meal can be stuck at the door, waiting for a station, missing ingredients, or burning on the stove. Start with where progress stopped; do not buy a faster oven for a ticketing failure.
| Reject boundary errors clearly | Unknown model, malformed roles, tokenizer/chat-template mismatch, and prompt+max_tokens beyond context should fail before GPU work |
| Separate memory failures | Weights may not fit at load; KV may exhaust under concurrency/context; runtime workspaces can spike; fragmentation can prevent a large allocation |
| Recognize overload | A growing queue and rising TTFT with stable TPOT indicates admission/prefill pressure |
| Check compatibility | Unsupported dtype, compute capability, attention backend, model architecture, or compiled shape can fail at startup or silently fall back to a slow path |
Unknown model, malformed roles, tokenizer/chat-template mismatch, and prompt+max_tokens beyond context should fail before GPU work. Report token counts and limits, not a generic 500.
Weights may not fit at load; KV may exhaust under concurrency/context; runtime workspaces can spike; fragmentation can prevent a large allocation. Inspect free memory, configured utilization, active sequences, context lengths, and allocator logs.
A growing queue and rising TTFT with stable TPOT indicates admission/prefill pressure. Rising TPOT suggests decode batch, bandwidth, collectives (collectives = cross-GPU communication such as all-reduce; see Lesson 27), or noisy-neighbor pressure. Add capacity only after locating the violated phase.
Unsupported numeric format (dtype, e.g. FP8 vs FP16; see Lesson 24), compute capability (compute capability = the GPU's CUDA feature-level version), attention backend (attention backend = which attention kernel is used, e.g. FlashAttention; see Lesson 19), model architecture, or compiled shape (a compiled/padded shape = kernels built ahead of time for fixed tensor sizes; see Lesson 26) can fail at startup or silently fall back to a slow path. Record versions and the selected kernels/backends.
Readiness must wait for loaded weights and warmup. Cancellation and shutdown must stop admission, drain or abort active streams intentionally, release KV, and distinguish timeout from server failure.
The live cluster provides real examples from different layers: heavy GPU memory reservation, vLLM running/waiting/KV metrics, time-sliced GPUs, and a diagnosed NVLink placement fault. The symptom must be assigned to the right layer before tuning.