The knobs that change output, cost, and request lifetime.
Lesson 10 traced one request end to end; here we turn the practical knobs on it. Some controls change the recipe's randomness, some cap the number of bites, some choose whether plates leave one-by-one, and some force the final dish into a mold. They are not interchangeable performance knobs.
A generated-token ceiling (for example max_tokens in vLLM or max_completion_tokens in newer APIs) caps work; the context limit caps prompt plus output. EOS (EOS = the special end-of-sequence token a model emits to stop) and supported stop sequences can end earlier. Timeouts and cancellation must propagate so abandoned requests stop consuming decode slots. Parameter support is model- and server-specific.
Temperature reshapes logits; top-p/top-k truncate the candidate set; a seed can improve repeatability but does not guarantee identical output across engines, hardware, batching, or versions.
Streaming reduces time to visible text but not model compute. Logprobs expose token scores at extra payload or compute cost. Usage counters are necessary for limits and billing.
JSON schema, grammars, and structured-output modes mask invalid next tokens during decoding. They improve syntactic validity but do not guarantee that values are true or semantically correct.
Validate unsupported combinations, set server-side ceilings, and document defaults. Client max_tokens is not a substitute for admission control or a total context check. Next, Lesson 12 zooms out to the engine that runs many such requests at once.
Your vLLM deployments already expose the OpenAI-compatible request surface used in the course. Supported fields and defaults depend on the deployed vLLM/model generation config, so inspect that version before promising an API contract.
The knobs that change output, cost, and request lifetime.
Lesson 10 traced one request end to end; here we turn the practical knobs on it. Some controls change the recipe's randomness, some cap the number of bites, some choose whether plates leave one-by-one, and some force the final dish into a mold. They are not interchangeable performance knobs.
| Bound the request | A generated-token ceiling (for example max_tokens in vLLM or max_completion_tokens in newer APIs) caps work; the context limit caps prompt plus output |
| Control sampling | Temperature reshapes logits; top-p/top-k truncate the candidate set; a seed can improve repeatability but does not guarantee identical output across engines, hardware, batching, or versions |
| Choose transport and observability | Streaming reduces time to visible text but not model compute |
| Constrain the output | JSON schema, grammars, and structured-output modes mask invalid next tokens during decoding |
A generated-token ceiling (for example max_tokens in vLLM or max_completion_tokens in newer APIs) caps work; the context limit caps prompt plus output. EOS (EOS = the special end-of-sequence token a model emits to stop) and supported stop sequences can end earlier. Timeouts and cancellation must propagate so abandoned requests stop consuming decode slots. Parameter support is model- and server-specific.
Temperature reshapes logits; top-p/top-k truncate the candidate set; a seed can improve repeatability but does not guarantee identical output across engines, hardware, batching, or versions.
Streaming reduces time to visible text but not model compute. Logprobs expose token scores at extra payload or compute cost. Usage counters are necessary for limits and billing.
JSON schema, grammars, and structured-output modes mask invalid next tokens during decoding. They improve syntactic validity but do not guarantee that values are true or semantically correct.
Validate unsupported combinations, set server-side ceilings, and document defaults. Client max_tokens is not a substitute for admission control or a total context check. Next, Lesson 12 zooms out to the engine that runs many such requests at once.
Your vLLM deployments already expose the OpenAI-compatible request surface used in the course. Supported fields and defaults depend on the deployed vLLM/model generation config, so inspect that version before promising an API contract.