Scale on the right signal
A standard web service scales on CPU because CPU tracks how busy it is. An LLM server breaks that habit: the metric that classic autoscalers reach for is exactly the one that lies to you here.
The default metric is the wrong one
Classic autoscaling watches CPU and adds replicas when it climbs. For LLM serving that signal is meaningless — the heavy work runs on the GPU, so CPU stays low even while the server is saturated.
GPU utilization misleads too
The tempting fix — scale on GPU utilization — also misleads. A single decode step can pin the GPU near 100% while the server still has plenty of room to batch more requests, so high util doesn't mean "full".
Scale on demand you can feel
Use signals that track real backlog: concurrency (requests in flight), queue depth (num_requests_waiting), or TTFT (time-to-first-token). When the queue grows or first tokens slow down, that's genuine demand — add replicas.
Keywords — tap to unfold the plain meaning
Scale on concurrency / queue depth (num_requests_waiting) / TTFT — not CPU. GPU utilization is misleading for LLMs.
The cold-start surprise
In a stateless web tier, "add a replica" is nearly instant, so you can scale reactively. With LLMs the new replica takes minutes to wake up — which changes the whole shape of how you scale.
Two ways LLM autoscaling differs
Two things separate this from stateless web autoscaling. First, the scaling metric: concurrency / queue depth / TTFT instead of CPU, because GPU util misleads. Second, the cold start duration.
Cold start runs in minutes
Cold start here means minutes, not seconds: the node pulls a multi-GB container image, then loads tens of GB of weights into VRAM. Until that finishes the replica can serve nothing.
So scale before you're desperate
Because the new replica lags by minutes, reactive scaling arrives too late. You scale early — at the first sign of pressure — and keep some warm capacity ready, so demand is met while the next replica boots.
Keywords — tap to unfold the plain meaning
A new replica costs minutes — image pull plus loading tens of GB of weights into VRAM. Scale up early and keep warm capacity, never purely reactive.
Strategies for the lag
If you can't make the new replica appear instantly, you work around the wait. Each strategy either keeps spare capacity ready or shrinks the cold start itself.
Keep capacity ahead of demand
Baseline + headroom keeps always-on replicas sized to absorb normal bursts. A warm pool goes further: pre-loaded replicas on standby, weights already in VRAM, ready the instant traffic spikes.
Anticipate, don't just react
Predictive scaling adds capacity ahead of known patterns — the daily peak, the scheduled batch job — so the minutes-long boot finishes before the load actually arrives.
Shrink the cold start itself
Faster cold start attacks the wait directly: NVMe weight caching so weights load from fast local disk, smaller images to pull, and quantized variants that are simply fewer bytes to move into VRAM.
Scale-to-zero, with eyes open
Scale-to-zero drops idle replicas to none — cheapest, but the first request after that eats the full minutes-long cold start. Use it only when that first-request latency is acceptable.
Keywords — tap to unfold the plain meaning
Baseline + headroom, warm pool, predictive scaling, faster cold start (NVMe cache, smaller images, quantized variants), and scale-to-zero only when minutes-long first-request latency is OK.
Sizing with Little's Law
Strategy is good, but you still need a number: how many replicas? Little's Law turns the concurrency cap and the latency you measured into a count you can put in your config.
One replica's ceiling
A replica runs at most --max-num-seqs sequences at once — its concurrency cap. Divide that cap by average latency and you get its max throughput in requests per second.
From ceiling to replica count
Now divide your peak QPS (queries per second at the busiest moment) by one replica's throughput. That's roughly how many replicas you need to keep up — the minimum before you add headroom.
Math, decoded
- max-num-seqsthe concurrency cap — how many sequences one replica serves at once (e.g. 8 or 64)
- avg latencyhow long, on average, one request takes start to finish
- Treplicaone replica's max throughput — its concurrency cap divided by latency
- peak QPSqueries per second at the busiest moment you must serve
- Nreplicas needed: peak QPS divided by what one replica can handle
A replica's max throughput ≈ its concurrency cap (--max-num-seqs) ÷ average latency; replicas needed ≈ peak QPS ÷ that. This is Little's Law applied to LLM serving — concurrency relates to throughput through latency.
Keywords — tap to unfold the plain meaning
Replica throughput ≈ concurrency cap ÷ latency; replicas ≈ peak QPS ÷ that. Size from the cap and the latency you actually measured.
HPA & KEDA, two changes
You don't throw out the platform. The familiar Horizontal Pod Autoscaler still drives this — you only swap the metric it watches and adjust how fast it scales each direction.
Change 1 — a custom metric
Run the HPA (Horizontal Pod Autoscaler) or KEDA as usual, but feed it a custom metric — num_requests_waiting, concurrency, or TTFT — instead of CPU. The autoscaler is fine; the default signal isn't.
Change 2 — respect the cold start
Treat cold start as pod-startup-latency on steroids: it's a multi-GB weight load, not a quick boot. So pair it with warm pools, NVMe weight caches, and asymmetric windows — scale-up-early with slow-scale-down.
Keywords — tap to unfold the plain meaning
num_requests_waiting / concurrency / TTFT, not CPU. (2) Treat cold start as pod-startup-latency on steroids (multi-GB weight load) → warm pools, NVMe weight caches, scale-up-early / slow-scale-down windows. Companion lab: autoscaling-concurrency-cold-starts.ipynb.
Keep HPA or KEDA; change two things — scale on a custom metric (not CPU), and budget for cold start with warm pools, NVMe caches, and scale-up-early / slow-scale-down windows.
On your cluster
num_requests_waiting + TTFT-P99 (not GPU util), budget a 27 GB FP8 weight load for cold start, and cap each replica with --max-num-seqs.Here it all lands on the 4×H100 box: the signal to scale on, the number to budget for cold start, and the per-replica cap that feeds Little's Law.
The signal to wire up
Autoscale your vLLM replicas on num_requests_waiting plus TTFT-P99 (the 99th-percentile time-to-first-token) — not GPU utilization. Those two track real backlog the way CPU and util can't.
The cold-start budget
Budget for a 27 GB FP8 weight load on every cold start. Recall the byte math: FP8 is 1 byte per number; FP16/BF16 is 2 bytes — so the same model at BF16 would be ~54 GB to move into VRAM.
The per-replica cap
Each replica's concurrency cap is --max-num-seqs — on this box, 8 or 64. That number is the numerator in Little's Law: it sets one replica's ceiling, and therefore how many replicas peak QPS demands.
num_requests_waiting + TTFT-P99 (not GPU util). Cold-start budget = 27 GB FP8 weight load (FP8 = 1 byte/number; FP16/BF16 = 2 bytes). Per-replica cap = --max-num-seqs (8 / 64). Companion lab: autoscaling-concurrency-cold-starts.ipynb.
Keywords — tap to unfold the plain meaning
Check yourself
- Autoscaling our LLM is different from our web tier because…?
- Which signals should drive the autoscaler, and why is GPU utilization misleading?
- Using Little's Law, how do you turn
--max-num-seqs, latency, and peak QPS into a replica count?
Request queue/concurrency/TTFT drives scaling (not CPU); GPU util misleads. Cold starts span minutes (tens of GB into VRAM), so keep warm baselines and scale up early. Size with Little's Law: peak QPS ÷ (concurrency cap ÷ latency).