Continuous batching

The previous part established the physics: batch-1 decode achieves roughly 2 FLOPs per byte against an H100 that needs 591 to keep its compute units fed. You are on the memory-bound side of the roofline, using well under 1% of the hardware you are renting, and the only way up is to serve more sequences per pass.

This chapter is about why that is harder than it sounds, and what the gap is worth.

Batching is memory allocation, not scheduling

The naive mental model is that batching is a queueing decision — collect requests, run them together. That model is wrong in the way that matters.

Every sequence you serve holds a KV cache: the attention keys and values for every token it has seen, held in GPU memory for as long as the sequence is alive. The vLLM team measured it at up to 1.7 GB for a single sequence on LLaMA-13B.

So your maximum batch size is not a config value. It is:

(GPU memory − model weights) ÷ KV cache per sequence

Work it for a 13B model at FP16 on an 80 GB H100. Weights are roughly 26 GB, leaving 54 GB. At 1.7 GB per sequence that is **31 concurrent sequences** if the memory packs perfectly.

Thirty-one, against the ~300 the roofline wants. Even perfect packing leaves you memory-bound. That is the real ceiling, and it is set by memory management rather than by your scheduler.

Perfect packing does not happen

It gets worse, and this is the finding that produced PagedAttention. From the vLLM team:

existing systems waste 60% – 80% of memory due to fragmentation and over-reservation

Two distinct failures:

Over-reservation. A sequence's final length is unknown when it starts. Classical implementations reserve for the maximum possible length up front. A request that could have finished in 60 tokens holds an allocation sized for 2,048.

Fragmentation. KV cache was traditionally required to be contiguous. Sequences start and finish at different times, leaving gaps too small for the next request — the same external fragmentation that operating systems solved with paging decades ago.

Apply 60–80% waste to our 31-sequence ceiling and the real batch lands somewhere around 6 to 12. You are now roughly 30× below the roofline crossover, on hardware you are paying full price for.

What the fix is worth, measured

PagedAttention borrows the OS solution: partition each sequence's KV cache into fixed-size blocks that need not be contiguous, and page them. Fragmentation collapses, over-reservation disappears because blocks are allocated as needed, and batch size rises to what the memory can genuinely hold.

The measured results, from vLLM's published benchmarks (LLaMA-7B on an A10G and LLaMA-13B on an A100 40GB, request lengths sampled from ShareGPT):

Up to 24× against the naive baseline. Up to 3.5× against the previous state of the art.

Note which comparison matters for you. If you are serving with plain transformers in a loop — which a surprising number of internal deployments still do — the available win is more than an order of magnitude. If you are already on a modern serving stack, the remaining headroom is 2–3.5×, which is still the largest single number in this book after provider choice.

Why this is the cost chapter, not the performance chapter

Throughput and cost per token are the same quantity viewed from two directions. The accelerator bills by the hour regardless of how many tokens come out of it:

cost per token = $/hour ÷ (tokens/second × 3600)

The hourly rate is fixed by what an H100-hour costs. The denominator is set by where you sit on the roofline. So a 3× throughput improvement is a 3× cost reduction, exactly, with no other change.

At RunPod's H100 rate of $2.99/GPU-hour, a serving stack running 3× slower than it should is paying $2.99 for every $1.00 of work. Nothing about that shows up as a line item — the invoice looks the same either way. It surfaces only as a cost-per-token number you have to compute yourself, which is why instrumentation comes first in this book.

The diagnostic

Three questions, in order:

1. What is your actual concurrent batch size under load? Not the configured maximum — the observed mean. Most serving stacks expose this. If it is in single digits while requests are queueing, memory is your constraint, not compute.
2. What fraction of GPU memory is KV cache, and what fraction of that is live tokens? The gap between allocated and live is your fragmentation, and it is the 60–80% number made specific to you.
3. Are you on a paged-attention stack at all? vLLM, TensorRT-LLM, SGLang, and the managed endpoints built on them all implement this. A hand-rolled transformers loop does not.

Question 3 is worth asking bluntly because the answer is often no, and the fix is a deployment change rather than a research project.

What this does not fix

Continuous batching raises throughput. It does not reduce per-request latency — a batched request can wait for a scheduling slot, and time-to-first-token can get worse under load even as tokens-per-second-per-dollar improves. That trade is usually correct and occasionally is not; it depends on the SLO you actually promised, which is its own chapter.

It also does nothing for a workload that genuinely has no concurrency. One user, one request at a time, is batch size 1 by definition, and the roofline is unforgiving about that. Batch-1 latency-critical serving is a real shape, and for it the levers are quantization, a smaller model, and speculative decoding — not batching.

And it cannot exceed what memory allows. If your KV cache per sequence is enormous because your context is long, the ceiling stays low no matter how well you pack. That is the case where the H200's extra 61 GB stops being a nice-to-have.