Failure Modes & Debugging — Inference Engineering

Classify the symptom before you turn knobs

TL;DR · Find which layer progress stopped at first — boundary, overload, memory, compatibility, or lifetime — then pick the one measurement that tells them apart. Don't guess.

The goal of this lesson: triage common inference failures from the request boundary all the way down to the GPU, and choose the first discriminating measurement instead of guessing. A wrong fix wastes GPU time and hides the real cause.

The two-rule discipline

Classify the symptom before turning knobs. Every failure lives in one layer; identify the layer, then choose the single measurement that discriminates between causes. And for quality work: commit a prediction before revealing the model — decide what you expect before you peek.

Five layers, five symptoms

Inference failures fall into five families: boundary errors (rejected before the GPU), overload (queues and per-token slowdown), memory / OOM (four distinct kinds), compatibility (silent or crash-on-start), and request lifetime (readiness and cancellation). The next five stations are one family each.

First discriminator, not first knob

For each symptom there is a first discriminating measurement — token counts vs. limits, TPOT stability, free vs. reserved memory, kernel/backend version logs. Read that one number first; it points at the layer, and the layer points at the fix.

Keywords — tap to unfold the plain meaning

Analogy ❝A delayed meal can be stuck at the door, waiting for a station, missing ingredients, or burning on the stove. Start with where progress stopped — the ticket never got taken, the line cook is backed up, the pantry is empty, the dish is scorching — and fix that. Do not buy a faster oven for a ticketing failure. The faster oven is the wrong knob when the order never reached the kitchen.

Classify the symptom, find the layer where progress stopped, read the first discriminating measurement — then turn a knob. Never the other way around.

Reject boundary errors clearly

TL;DR · Errors at the request/service interface should fail fast with HTTP 400 and explicit counts — never a generic 500 that hides the cause and burns no insight.

Boundary errors happen before any GPU computation. No GPU time is spent, the client can fix the request and retry — but only if you tell them exactly what was wrong.

What goes wrong at the boundary

Four classic boundary errors: an unknown model name, malformed roles in the chat request, a tokenizer / chat-template mismatch, and a prompt plus max_tokens that exceeds the context window. All are caught at the request/service interface, before the model runs.

Return 400 with the numbers

The correct response is HTTP 400 with explicit token counts and limits — for example, prompt 9,000 + max 2,000 > context 8,192 — not a generic 500. A 500 hides the cause; a counted 400 lets the caller fix and retry immediately.

Keywords — tap to unfold the plain meaning

The check, decoded

prompt_tokens + max_tokens > context_window → reject (400)

prompt_tokenshow many tokens the incoming prompt occupies after tokenizing — e.g. 9,000
max_tokensthe most new tokens the request asked to generate — e.g. 2,000
context_windowthe model's hard limit on prompt + output combined — e.g. 8,192
>if the sum can't fit, there is no point starting; reject before the GPU with the counts shown

9,000 + 2,000 = 11,000 > 8,192, so the request can never fit. Return a 400 that states all three numbers — the client sees exactly what to trim.

SRE note A boundary 400 costs zero GPU seconds and is self-explaining. A generic 500 on the same input costs the same zero seconds but tells the caller nothing — they retry blindly, and your error rate climbs for no reason. Always echo the counts and the limit.

Boundary errors spend no GPU time. Reject them with a 400 that names the counts and limits — never a generic 500 that hides the cause.

Separate the four OOM failures

TL;DR · "Out of memory" is four different bugs — weights, KV cache, workspace, fragmentation. Each has its own trigger and its own measurement; treat them separately.

"OOM" is the most over-collapsed error in serving. Inspect free vs. reserved memory, configured utilization, active sequences, context lengths, and allocator logs — they split one symptom into four distinct stories.

Weights vs. KV cache

Weights OOM happens at load time: the model size exceeds VRAM — measure free memory at startup. KV cache OOM happens under load: concurrency × context length outgrows the reserved cache — measure active sequences times their context.

Workspace vs. fragmentation

Workspace OOM is a transient spike from a kernel's scratch allocation — catch it in allocator spike logs. Fragmentation OOM fails a large allocation even though total free memory exists, because it isn't contiguous — read the allocator's fragmentation report.

Four OOM stories. The measurement column is the discriminator — read it first and the right fix (smaller model, less batch/context, kernel scratch, defragment) follows.

Keywords — tap to unfold the plain meaning

Analogy ❝A pantry can be "out of room" for four very different reasons: the bulk staples never fit on day one (weights), every active order is hoarding shelf space at once (KV cache), one dish briefly grabbed a huge counter to plate it (workspace), or there's plenty of total space but it's scattered in gaps too small for the big tray (fragmentation). Same empty-handed cook, four different fixes.

OOM is four bugs: weights (load), KV cache (concurrency × context), workspace (kernel scratch spike), fragmentation (free but non-contiguous). The measurement tells them apart.

Recognize overload with TPOT

TL;DR · Watch TPOT. If queue and TTFT rise but TPOT stays flat, it's an admission/prefill bottleneck — add capacity. If TPOT itself rises, it's a decode bottleneck.

Overload looks the same from outside — "it's slow" — but one number splits it cleanly. TPOT (Time Per Output Token) is the discriminator: is each token getting slower, or is the queue just longer?

Queue ↑, TTFT ↑, TPOT flat → prefill bottleneck

If the queue grows and TTFT (Time To First Token) climbs but TPOT stays flat, the jam is at admission / prefill. Tokens still generate at full speed once started — you just can't start them fast enough. The fix is add capacity (or shed load), not a faster GPU.

TPOT ↑ → decode bottleneck

If TPOT itself rises — each token genuinely slower — the jam is in decode. Suspect batch size, memory bandwidth, collectives (cross-GPU communication such as all-reduce), or noisy-neighbor pressure. Here speed per token is the problem, so the fixes target the decode step.

Same "it's slow" symptom, two layers. TPOT flat points up to admission/prefill; TPOT rising points into decode.

Keywords — tap to unfold the plain meaning

Analogy ❝Two ways the kitchen runs late. Either the line out the door is long but each plated dish still leaves in the normal time — you need more line cooks, not a faster stove (prefill). Or every dish itself is now cooking slower because the single shared range is jammed and the cooks keep bumping elbows over one road to the pantry — that's the decode step choking on bandwidth and cross-talk. One queue is too long; the other cooking is too slow.

TPOT flat with queue/TTFT rising = prefill bottleneck (add capacity). TPOT rising = decode bottleneck (batch, bandwidth, collectives, noisy neighbor).

Check compatibility — the silent failures

TL;DR · dtype, compute capability, attention backend, architecture, and compiled shape either crash at startup or fall back to a slow path that logs nothing. Record versions and selected kernels.

The dangerous compatibility bugs are the quiet ones. A mismatch may crash loudly at startup — or worse, silently swap in a slow fallback path that never says a word, and you only notice the latency weeks later.

Five things that must match

Five settings silently decide your speed: dtype (e.g. FP8 vs FP16), compute capability (the GPU's CUDA feature-level version), the attention backend (which kernel is active, e.g. FlashAttention), the model architecture, and the compiled / padded shape (kernels pre-built for fixed tensor sizes).

Crash loud, or fall back quiet

A mismatch produces one of two outcomes: a startup crash (loud, easy) or a silent fallback to a slow path (quiet, dangerous). The mitigation: record versions and the selected kernels / backends — the slow path often logs nothing at all, so you must capture what was chosen.

Keywords — tap to unfold the plain meaning

SRE note When startup succeeds but latency is mysteriously bad, suspect a silent fallback. Log the chosen dtype, compute capability, attention backend, and compiled shape on boot — because "it didn't crash" is not the same as "it took the fast kernel." The slow path rarely announces itself.

dtype, compute capability, attention backend, architecture, compiled shape — a mismatch crashes at startup or silently goes slow. Record versions and selected kernels.

Protect request lifetime — and on your cluster

TL;DR · Don't flip readiness green until weights are loaded and warmup is done, and cancel cleanly: stop admission, drain or abort streams, release KV cache, tell timeout from server failure.

The last layer is the request's whole life — from "am I ready for traffic?" to "how do I shut a stream down cleanly?" Get readiness wrong and you send traffic straight into cold-start latency.

Readiness after warmup, not before

Readiness must wait for loaded weights and warmup completion. A readiness that flips green before warmup sends traffic into cold-start latency — the requests technically succeed but every early one is slow, and your dashboards lie about why.

Cancel and shut down on purpose

On cancellation or shutdown: stop new admission, drain or abort active streams intentionally, release the KV cache they held, and distinguish a timeout from a server failure. Each active stream is holding GPU memory; an unclean abort leaks it.

Cluster note Live lab examples on the 4×H100 box: heavy GPU memory reservation, vLLM running / waiting / KV metrics, time-sliced GPUs, and a diagnosed NVLink placement fault. References: the vLLM troubleshooting docs and the Kubernetes Pod-lifecycle docs (readiness, draining, termination).

Triage table — symptom → layer → first discriminator → action

Rejected before GPU      boundary       token counts vs limits     return 400 + counts
Queue↑ TTFT↑ TPOT flat    admission      TPOT stability             add capacity / shed load
TPOT ↑                   decode         per-token latency trend    check batch/bw/neighbor
OOM at load              memory:weights free mem at startup        model > VRAM
OOM under concurrency    memory:KV      active seqs × context      reduce batch or context
OOM transient            memory:wkspc   allocator spike logs       kernel scratch behavior
OOM despite free mem     memory:frag    fragmentation report       defragment / reduce peak
Crash or silent slowdown compatibility  kernel/backend version     verify dtype/arch/cap

Keywords — tap to unfold the plain meaning

Check yourself

A request is rejected before any GPU work — which layer is it, and what should the server return instead of a 500?
Queue and TTFT are rising but TPOT is flat. Which bottleneck is it, and is "buy a faster GPU" the right fix?
Name the four kinds of OOM and the measurement that discriminates each.
Why is a silent fallback more dangerous than a startup crash — and what should you record to catch it?
Why must readiness wait for warmup, and what four steps make a clean cancellation?

Ready only after warmup; cancel by stopping admission, draining/aborting streams, releasing KV, and telling timeout from failure. Classify the symptom, then turn the knob.

Reached the end — nice. This lesson now counts toward your progress.