Practical API Controls — Inference Engineering

Five families of knobs

TL;DR · Every API parameter falls into one of five families — limits, sampling, observability, transport, or constrained decoding — and each family has a different operational effect.

You met the request lifecycle last lesson. Now we sort the dials on the dashboard. The goal isn't to memorize every flag — it's to classify any control by what it actually changes: the work done, the token chosen, what you can see, how it's delivered, or the shape it must fit.

The single skill of this lesson

Given any API parameter, classify it as a generation limit, a sampling control, an observability option, a transport choice, or constrained decoding — and predict its operational effect on cost, latency, or output shape.

Why the family matters more than the name

Names drift between servers: max_tokens in vLLM is max_completion_tokens in newer APIs. But the family is stable. Once you know which family a knob belongs to, you know whether it touches compute, the chosen token, your visibility, the delivery, or the form.

One trap to retire now

Two of these families look like they change the answer but don't change model compute: transport (streaming) only changes delivery, and observability (logprobs, usage) only changes what you can see. Keep them separate from the knobs that truly reshape the work or the token.

Keywords — tap to unfold the plain meaning

Limits and sampling change the run itself; observability, transport, and constrained decoding change what you see, how it arrives, and the form it takes.

Five families: limits, sampling, observability, transport, constrained decoding. Classify the knob, then you can predict its effect.

Bound the request

TL;DR · Limits — max_tokens, stop sequences, the context window, and cancellation — cap the computational work so one request can't run away with your decode slots.

The first family is about boundaries. A generation has to stop somewhere, and several different mechanisms can end it. Get these wrong and a single abandoned request will quietly burn capacity long after the user has left.

The output ceiling

A generated-token ceiling — max_tokens in vLLM, max_completion_tokens in newer APIs — caps computational work by limiting how many tokens the model is allowed to produce. It's the most direct dial on cost per request.

The context limit is a different cap

The context limit (the context window) restricts total tokens — prompt plus output combined. It's not the same as max_tokens: one bounds the whole sequence, the other bounds only the new tokens you generate.

Natural and configured stops

The model emits an EOS (end-of-sequence) token to signal it's done — a special token that ends generation naturally. You can also supply stop sequences: configured strings that end generation early, before max_tokens is reached.

Cancellation must propagate

Timeouts and cancellation have to reach the engine. If they don't, an abandoned request that is not cancelled keeps decoding into the void — holding KV cache memory and a batch slot — even though no one is listening.

Keywords — tap to unfold the plain meaning

Analogy ❝A line cook keeps plating an order until the ticket says "that's all" — the EOS token. A stop sequence is the expediter calling "cut it there" partway down. max_tokens is the kitchen rule that no single order gets more than N plates, no matter what. And cancellation is someone tearing the ticket off the rail when the table walks out — if that message never reaches the cook, they keep plating dishes nobody will ever eat, tying up a burner the next order needed.

Cluster note On the 4×H100 box, an abandoned but un-cancelled request keeps decoding to its max_tokens ceiling, holding KV cache and a batch slot the whole time. max_tokens caps the damage, but it is not a substitute for admission control or a total-context check — set server-side timeouts so cancellation actually frees the slot.

Limits cap work: max_tokens bounds output, the context window bounds prompt+output, stop and EOS end it, and cancellation must propagate or the slot stays busy.

Control sampling

TL;DR · Temperature reshapes the whole distribution; top-p and top-k only truncate the candidate set; the token is then drawn from what remains. Seed aids repeatability but guarantees nothing across engines or hardware.

This is the family people reach for first and understand least. The key distinction: temperature changes the shape of every token's probability, while top-p and top-k only decide which candidates survive. They are not interchangeable.

Temperature reshapes the logits

Temperature reshapes the logits before softmax. At T = 0.2 the distribution is sharp and near-greedy; at T = 1.0 you get the model's own spread; at T = 1.8 it flattens toward random. It touches every token's probability at once.

Top-k is a fixed count

top-k truncates the candidate set to a fixed number. top-k = 3 keeps exactly the 3 most likely candidates and drops the rest — then sampling happens among only those.

Top-p is a cumulative mass

top-p (nucleus sampling) truncates by cumulative probability mass. top-p = 0.9 keeps candidates, highest first, until their probabilities add up to 90% — a set that grows or shrinks with how confident the model is.

Seed helps, but never promises

A seed can improve repeatability of sampling, but it does not guarantee identical output across engines, hardware, batching, or versions. Treat it as a nudge toward reproducibility, not a contract.

Keywords — tap to unfold the plain meaning

Math, decoded

p_i = e^z_i/TΣ_j e^z_j/T

z_ithe raw logit for token i, before temperature touches it
Ttemperature — divide every logit by it before softmax
T < 1divides up: gaps widen, distribution sharpens toward greedy (e.g. T=0.2)
T > 1divides down: gaps shrink, distribution flattens toward random (e.g. T=1.8)
p_ithe reshaped probability — top-p/top-k then cut this set before drawing

Temperature divides the logits before softmax, so it rescales every probability at once. Top-p and top-k come after: they only crop which candidates remain, then one token is drawn from the survivors.

Temperature rescales every bar; top-k cuts to a fixed number of bars; top-p cuts once the kept bars reach 90% of the mass. Only then is a token drawn.

Temperature reshapes the whole distribution; top-p/top-k only truncate the candidate set; the token is drawn from what's left. Seed nudges repeatability — it doesn't guarantee it.

Choose transport & observability

TL;DR · Streaming changes only delivery, not model compute. Logprobs and usage counters change only what you can see — for some extra payload or compute. Neither family reshapes the answer.

These two families are the ones most often mistaken for performance wins. Streaming feels faster; logprobs feel like a deeper look. Both are real and useful — but it's important to know exactly what they do and don't change.

Streaming is delivery, not speed

Streaming reduces time to visible text — the user sees tokens as they're produced. But it does not reduce the model's compute time. The work is identical; only the moment of first visible output moves earlier.

Logprobs cost a little extra

logprobs expose the token scores — the raw next-token probabilities — at some extra payload or compute cost. They're how you inspect the distribution you met earlier, but they aren't free, and they don't change the chosen token.

Usage counters are for accounting

Usage counters report tokens consumed and produced. They're necessary for enforcing limits and for billing — observability that feeds your controls and your invoices, not the generation itself.

Keywords — tap to unfold the plain meaning

Analogy ❝Streaming is the line cook handing you each component the instant it's ready instead of waiting to carry the whole plate — you start eating sooner, but the kitchen did exactly the same amount of cooking in exactly the same time. Logprobs are asking the cook to also write down, for each reach, the runner-up ingredients they almost grabbed: useful insight, but it slows the pass a touch and changes nothing about what landed on the plate. Usage counters are the receipt tallying every ingredient used — for the bill, not the meal.

Transport (streaming) changes delivery, not compute. Observability (logprobs, usage) changes what you see, for a little extra cost. Neither reshapes the answer.

Constrain the output

TL;DR · Constrained decoding masks invalid next tokens during decoding so the output fits a JSON schema or grammar — guaranteeing form, never facts.

The last knob family forces the output into a shape. It's powerful: it eliminates whole classes of parse errors. But it carries a sharp limit that trips up almost everyone — it guarantees the structure, and says nothing about whether the values are true.

It masks invalid tokens as it decodes

JSON schema, grammars, and structured-output modes work by masking invalid next tokens during decoding. At each step, tokens that would break the required form are forbidden, so the model can only continue down valid paths.

What it guarantees

Constrained decoding guarantees syntax, required keys, types, and enums. It eliminates parse errors and retries — a perfectly-formed object comes back every time.

What it can never guarantee

It does not guarantee correct, current, or truthful field values. A perfectly-formed object can still hold a wrong number. As the source puts it: structure validity is not factual validity.

Keywords — tap to unfold the plain meaning

Cluster note Enabling a JSON schema on the cluster eliminates parse errors and retries — every response is well-formed. But a valid object can still carry a wrong value: structure validity is not factual validity. Constrained decoding fixes the form, not the truth, so keep validating the content downstream.

Analogy ❝A grammar is a plating template stamped on the pass: the cook can only place items into the marked slots, so every plate leaves the kitchen in the right shape — entrée here, garnish there, no missing sections. But the template says nothing about whether the dish is seasoned correctly or even the one that was ordered. A flawlessly arranged plate can still be the wrong meal.

Constrained decoding masks invalid tokens to enforce syntax, keys, types, and enums — but a well-formed object can still be wrong. Structure validity is not factual validity.

Treat controls as a contract

TL;DR · Validate unsupported combinations, set server-side ceilings, and document defaults — because parameter support varies by model and server, and a client value is never your safety net.

The knobs only behave if the server agrees to honor them. Support varies by model and server, so the operator's job is to turn a pile of optional parameters into a stable, enforced contract — one that holds even when a client sends something odd.

Validate and bound on the server

Validate unsupported parameter combinations and reject them clearly. Set server-side ceilings so a client can never request more work than you'll allow — client values are requests, not guarantees.

Document the defaults

Document the defaults every control falls back to. When a parameter is unset or unsupported, callers need to know exactly what they'll get — undocumented defaults are silent behavior changes.

Client max_tokens is not admission control

A client max_tokens is not a substitute for admission control or a total context check. It caps one request's output, but only server-side admission control protects the cluster from overload.

Try it on the 4×H100 box — one request, several families at once

curl localhost:8000/v1/completions -d '{
  "model": "Qwen3.6-27B-FP8",
  "prompt": "List two Kubernetes scheduler components as JSON.",
  "max_tokens": 64,
  "stop": ["\n\n"],
  "temperature": 0.2,
  "top_p": 0.9,
  "seed": 42,
  "logprobs": 5,
  "stream": false
}'

Cluster note This single request touches four families: max_tokens + stop (limits), temperature + top_p + seed (sampling), logprobs (observability), and stream (transport). Add a JSON schema and you've used all five. On the cluster, the server still applies its own ceilings on top — the client values here are requests, not the final word.

Keywords — tap to unfold the plain meaning

Check yourself

Which family does each knob belong to — max_tokens, temperature, streaming, logprobs, JSON schema?
Temperature vs top-p/top-k: which reshapes the whole distribution, and which only truncates the candidate set?
What does constrained decoding guarantee, and what can it never guarantee?
Why is a client's max_tokens not a substitute for server-side admission control?

Controls are a contract: validate combinations, set server-side ceilings, document defaults. A client max_tokens is never a substitute for admission control.

Reached the end — nice. This lesson now counts toward your progress.