What Is an LLM? — Inference Engineering

01

It's a next-token predictor

TL;DR · An LLM does exactly one thing: given the text so far, it predicts the single next token — then appends it and does it again.

Everything else in this course — the runtime, the GPUs, the production tricks — exists to make this one tiny prediction happen fast, over and over. Get the loop and the rest is plumbing.

One job, repeated forever

An LLM is a next-token predictor. Given the text so far, it outputs a probability over its whole vocabulary for what comes next, picks one token, appends it to the text, and repeats with the longer text.

It doesn't plan ahead

There's no outline, no draft, no idea of the finished answer. Each step is a fresh guess at just the next token. The "reasoning" you see is what emerges from millions of these one-step guesses chained together.

The knowledge lives in the weights

The numbers that turn "text so far" into "scores for the next token" are the model's weights — billions of fixed parameters learned during training. At inference time they never change; they're just read.

Keywords — tap to unfold the plain meaning

Analogy ❝Picture a line cook glancing at the order ticket so far and predicting the single most likely next ingredient to reach for — not the finished plate. Read "tomato, basil, mozzarella…" and the next reach is probably "olive oil," not "ice cream." The cook never sees the whole dish in advance; each move is just the best next reach given everything written on the ticket up to now.

An LLM outputs one thing per step — a probability over the next token — then appends and repeats. It predicts, it doesn't plan.

02

The raw output is a probability distribution

TL;DR · One step produces a score for every token in the vocabulary; softmax turns those into probabilities, and greedy decoding just takes the top one.

The model never hands you a word. It hands you a number for every possible word, all at once — and "generation" is just deciding which of those numbers to honor.

A score for every token

A single forward pass produces one raw score — a logit — for every token the model knows. Softmax squashes that whole list into probabilities that sum to 1. Greedy decoding just picks the highest one.

It's a probability, never a fact

The model learned that Kubernetes pods are placed by the kube-scheduler — but it stores that as a tendency, not a lookup. Below is real measured output from the cluster: notice it's spread across several plausible continuations, with one clearly favored.

Measured on the cluster — Qwen3.6-27B-FP8 on 4×H100, prompt: "Kubernetes pods are scheduled by the"

" kube"38.9%

" scheduler"18.4%

" Kubernetes"12.6%

" control"3.2%

" cluster"2.3%

" controller"1.9%

Greedy decoding picks " kube" (38.9%) — the start of "kube-scheduler". The model learned pods are placed by the kube-scheduler as a probability, not a stored fact.

Keywords — tap to unfold the plain meaning

Math, decoded

p_i = e^z_iΣ_j e^z_j → Σ p_i = 1

z_ithe raw score (logit) the model gives token i — any real number, can be negative
e^zexponentiate each score so they're all positive and big scores pull far ahead
Σ_j e^z_jadd up the exponentiated scores of every token — the normalizer
p_ieach token's share of the total: a probability between 0 and 1

Softmax turns raw scores (logits) into probabilities that sum to 1: exponentiate every score, then divide each by the total. Greedy decoding just keeps the largest.

The model's raw output is one logit per token; softmax makes it a distribution that sums to 1; greedy takes the top. It's always a probability, never a fact.

03

Training vs inference: two different jobs

TL;DR · Training learns the weights once, at huge cost; inference runs those frozen weights on every single request. This course is about inference.

People say "the model" to mean both jobs, but they're as different as compiling a program and running it. Confusing them is the root of most muddled thinking about cost.

Training: learn the weights

Training reads oceans of text and slowly adjusts the weights until the next-token predictions get good. It happens once per model, costs enormous amounts of compute, and is over before you ever serve a request.

Inference: run the weights

Inference takes those now-frozen weights and runs them forward to answer a prompt. It happens on every request, forever. "Loading the model" means loading that artifact — the weights — into GPU memory so it's ready to run.

Keywords — tap to unfold the plain meaning

Analogy ❝Training is writing the cookbook: you taste millions of dishes, arguing over every recipe, until the book is finished — exhausting, expensive, done once. Inference is a line cook using that finished cookbook to plate orders all day. The cook never rewrites a recipe mid-service; they just read the page and cook. Every order is one more read of the same finished book.

SRE note Training = compile. The weights are the compiled artifact (your .safetensors binary). Inference = run the binary: load the artifact into GPU memory once, then execute it per request. You compile rarely; you run constantly — so this course optimizes the run, not the compile.

Training happens once and learns the weights; inference happens per request and runs them frozen. Compile once, run forever — we optimize the run.

04

Every request is a three-stage pipeline

TL;DR · Every request runs the same loop: tokenize → forward pass → decode/pick → append, repeat. Making this loop fast is the whole course.

This is the map. Almost every term you'll meet later — KV cache, batching, prefill, decode — is just a way to speed up one of these four steps without changing what they do.

Step 1 — Tokenize

Tokenize: chop the incoming text into integer tokens the model understands. This turns "Hello world" into a short list of token IDs.

Step 2 — Forward pass

Forward pass: run those tokens through the weights to produce logits — one score per vocabulary token — for the next position.

Step 3 — Decode / pick

Decode: turn the logits into one chosen token (greedy, or with sampling). This is the actual "which word" decision.

Step 4 — Append & repeat

Append the chosen token to the sequence and loop back to the forward pass. This autoregressive loop runs once per generated token until you hit a stop.

Text in, token out, then the token feeds straight back into the next forward pass. The dashed line is the loop that makes generation possible — and expensive.

Keywords — tap to unfold the plain meaning

Tokenize → forward pass → decode → append & repeat. That four-step loop is the whole request, and the whole rest of the course.

05

It's a loop — and that's where the cost hides

TL;DR · Each new token re-reads the whole sequence so far, so every token costs a little more than the last — a rising staircase the engine exists to flatten.

The loop looks innocent, but it hides the central problem of serving: work per step grows as the answer grows. Every optimization later is a fight against this one staircase.

Each step re-reads everything

To predict token n+1, the naive loop re-reads all n tokens before it. So the first token is cheap, and each later token in autoregressive generation is slightly more expensive than the one before.

A rising staircase

Plot cost per step and it climbs as the sequence grows — the longer the context length, the steeper the later steps. Long outputs don't just cost more in total; each token gets pricier.

The whole engine fights this

The KV cache (you'll meet it soon), batching, and the inference engine all exist to flatten that staircase — to stop re-reading the whole past on every single step.

Each bar is one generated token. Because every step re-reads the whole past, the bars climb — the rising staircase that KV cache and batching are built to flatten.

Keywords — tap to unfold the plain meaning

Math, decoded

work per step ∝ n → total work ∝ n²

nhow many tokens are already in the sequence when this step runs
∝ nnaive per-step work grows in proportion to n — re-read all n prior tokens
∝ n²summing rising per-step costs over the whole output gives quadratic total work

Naively, the work to produce one token grows with the sequence length n, so generating a long answer scales like n² overall. The engine's job is to break that growth.

Analogy ❝Imagine a cook who, after every single bite they plate, re-reads the entire order ticket from the very top before deciding the next bite. Bite one is instant. By bite fifty they're re-reading fifty lines just to add one more. The food isn't harder to cook — the re-reading is what's piling up. Flattening that re-read is the whole game.

Each token re-reads the whole sequence, so cost per step rises with n and total work scales like n². Every optimization ahead exists to flatten that staircase.

06

On your cluster

TL;DR · Ask for one token with logprobs and the server hands you the raw distribution — you can see the next-token guess for yourself.

This isn't theory you have to take on faith. One request, one flag, and the model shows you its hand: the same kind of distribution you saw in Station 02.

Request a single token, with logprobs

Cap the output at one token and ask for logprobs. The server runs exactly one forward pass and returns the top candidates with their log-probabilities — the raw distribution, before any word is committed.

Try it on the 4×H100 box

curl localhost:8000/v1/completions -d '{
  "model": "Qwen3.6-27B-FP8",
  "prompt": "Kubernetes pods are scheduled by the",
  "max_tokens": 1,
  "logprobs": 8
}'

Cluster note Live-tested 2026-06-19 on the 4×H100 box. The logprobs: 8 field returns the eight most likely next tokens with their scores — exactly the " kube" / " scheduler" / " Kubernetes" spread from Station 02. Set max_tokens: 1 so the server stops after a single forward pass and you see the distribution unmuddied by later steps.

Keywords — tap to unfold the plain meaning

Check yourself

What's the only thing an LLM outputs at each step?
Why does each token cost a bit more than the last?
Training vs inference — which happens once, and which happens per request?

One token plus logprobs shows you the raw distribution on your own cluster — the predictor's hand, before it commits to a word.