Quality Evaluation — Inference Engineering

01

Faster only counts if quality holds

TL;DR · Every speedup in this course — quantization, compilation, new sampling, speculative decoding, engine upgrades — must clear a quality check before you ship it.

A change that doubles throughput but quietly degrades answers isn't a win — it's a regression you can't see on a latency dashboard. This lesson is the gate that keeps speed honest.

The core principle

Faster is only better if the output remains acceptable. Speed is worthless if it ships worse answers. So before you accept any speedup, you prove the output is still good enough.

Which changes need this gate

The course optimizations all earn a quality check: quantization, compilation, changed sampling, speculative decoding, and engine upgrades. Each is a performance improvement that needs validation before shipping.

What "acceptable" means

Acceptable is measured against a baseline — the trusted config you're comparing to — and an agreed tolerance, the small amount of drift you're willing to accept. No baseline, no honest comparison.

Keywords — tap to unfold the plain meaning

Analogy ❝A line cook who plates twice as fast but starts skipping the basil isn't faster — they're cooking a different, worse dish. Speed on the wrong plate is just speed at being wrong. Before you praise the new pace, you taste the food against the recipe you trust. Quicker only counts once the plate still passes.

Faster is only better if the output stays acceptable. Every speedup gets a quality check against a trusted baseline before it ships.

02

Define quality for the task, not in general

TL;DR · Pick a metric that matches the actual job — exact match, pass@k, retrieval quality, factuality, schema validity, preference judgments. Perplexity alone is rarely the product metric.

"Is it good?" has no answer until you say good at what. A code endpoint, a RAG search box, and a JSON extractor each fail in different ways — so each needs its own ruler.

Task-specific metrics

Different jobs use different rulers: exact match, pass@k (code execution tests), retrieval quality, factuality review, schema validity, preference judgments, and other domain-specific checks. Choose the one that mirrors how the product is actually used.

Why perplexity isn't enough

Perplexity alone is rarely the product metric. It can stay flat while the user-facing metric regresses — a fluent-looking model that quietly gets the answers more often wrong. Measure the thing users feel, not just held-out text prediction.

Keywords — tap to unfold the plain meaning

Math, decoded

pass@k = # tasks with a correct answer in k tries# tasks

khow many independent attempts (samples) the model gets per task
correct in k triesfor code, "correct" means at least one sample passes the unit tests
÷ # tasksaverage over all tasks, giving a success rate between 0 and 1

Pass@k is the fraction of tasks the model solves given k attempts. For code generation, a task counts as solved if any of the k samples passes its execution tests. Exact match is the stricter k=1, single-correct-answer case.

Pick the ruler Exact match = binary correctness on extraction. Pass@k = unit-test / code-gen success. Retrieval quality = relevance scoring in RAG. Factuality review = groundedness on open-ended outputs. Schema validity = structural correctness of returned fields. Preference judgments = win-rate in human or model comparisons. Perplexity = generic score of how well a model predicts held-out text; lower is better — but rarely the product metric.

Quality is task-specific. Choose a metric that mirrors the real job, and don't trust perplexity alone — it can stay flat while users get worse answers.

03

Sort changes: lossless vs lossy

TL;DR · Lossless changes preserve the output distribution by design and only need a spot-check; lossy changes alter the math and require a full evaluation.

Not every speedup risks quality equally. Knowing which bucket a change falls in tells you whether to spot-check it or run the whole eval suite — and saves you from gating things that can't drift.

Lossless — preserve the distribution

Lossless changes give the same output distribution by design, so they only need a spot-check: prefix / KV caching, exact speculative decoding, and batching / paging. They speed up how tokens are computed, not which tokens come out.

Lossy — require evaluation

Lossy changes alter the numbers the model produces, so they require evaluation: quantization (weights or KV cache), a changed kernel or attention backend, and altered sampling or chat templates. Any of these can shift the distribution.

Keywords — tap to unfold the plain meaning

The bucket decides the effort: lossless gets a quick spot-check, lossy gets the full evaluation against your baseline.

Analogy ❝Stocking the basil right next to the cook's station is lossless — a faster reach, but the exact same dish reaches the plate, so a glance confirms it. Swapping in a cheaper cheese is lossy — it changes the flavor, so now you actually taste every plate before serving. Same kitchen, two very different levels of checking.

Lossless changes (caching, exact spec decoding, batching) keep the distribution and only need a spot-check. Lossy changes (quantization, new kernels, altered sampling) must be evaluated.

04

Control the randomness before you measure

TL;DR · For deterministic outputs, use greedy decoding or fixed seeds; for sampling products, compare distributions across many trials, never one anecdotal sample.

Sampling means the same prompt can give different answers each run. If you don't pin down the randomness, you can't tell whether a difference came from your change or from luck.

When you want determinism

For a fair before/after, remove the noise: use greedy decoding (always take the top token) or fixed seeds so the random draws repeat. Now any difference you see is the change, not chance.

When the product samples

If the product itself uses sampling, a single output is an anecdote. Instead, compare distributions across repeated trials — run many times and look at the spread, not one cherry-picked sample.

Keywords — tap to unfold the plain meaning

Pin the randomness first: greedy or fixed seeds for deterministic tests, and compare distributions over many trials when the product samples. One anecdote proves nothing.

05

Paired gates: quality first, then speed

TL;DR · A change must clear two independent gates — quality within tolerance vs baseline, AND SLO-valid speed on the same workload. Fail quality and it's rejected, however fast it is.

It's tempting to ship anything that's faster. The discipline here is that speed never overrides quality: both gates must pass, and quality has the veto.

Gate 1 — quality

The quality gate: the metric stays within an agreed tolerance versus the baseline. First you require quality is within tolerance — only then do you even look at speed.

Gate 2 — performance

The performance gate: SLO-valid speed measured on the same workload. Comparing speed on a different workload is meaningless — it has to be apples to apples.

Quality has the veto

The gates are independent and both must pass. A faster configuration that fails quality is rejected. Speed cannot buy back lost quality.

Quality is checked first; only a change that clears tolerance reaches the speed gate. A faster config that fails quality is rejected — speed has no veto, quality does.

Keywords — tap to unfold the plain meaning

Two independent gates: quality within tolerance vs baseline, then SLO-valid speed on the same workload. A faster config that fails quality is rejected.

06

Canaries need semantic checks — on your cluster

TL;DR · Offline gates aren't enough for rollout — add sampled online quality signals and a rollback threshold, because latency and error metrics can't catch a fluent-but-worse model.

A canary that only watches latency and errors will wave a degraded model straight through. The model is up, it's fast, it's returning 200s — and it's quietly wrong. You need to watch the meaning, not just the plumbing.

Why ops metrics miss it

Latency and error metrics cannot detect a fluent but worse model. The responses still arrive fast and well-formed; only the content got worse. Those signals are blind to quality.

What a real canary needs

Production rollout needs the offline gates plus sampled online quality signals — a slice of live traffic scored for quality — and a rollback threshold that automatically pulls the change if quality drops too far.

Cluster note On the course cluster, the speedups under test were FP8 weights, an FP8 KV cache, and DFlash speculative serving — real performance wins that all required quality validation before shipping. A typical canary samples roughly 5% of traffic for an online quality signal, paired with a rollback threshold.

Shape of a canary check (illustrative)

canary:
  candidate: "fp8-kv + dflash"
  baseline:  "bf16"
  online_sample: 5%          # sampled quality signal
  quality_gate:  "within tolerance vs baseline"
  perf_gate:     "SLO-valid on same workload"
  rollback_if:   "quality < threshold"

Keywords — tap to unfold the plain meaning

Check yourself

Why is "faster" not enough to accept an inference change on its own?
A change is lossless vs lossy — which one needs full evaluation, and which only a spot-check?
What are the two paired gates, and which one has the veto when a faster config fails?
Why can't latency and error metrics catch a fluent-but-worse model in a canary?

Canaries need semantic checks: offline gates plus sampled online quality signals and a rollback threshold. Latency and errors can't see a fluent model that simply got worse.