Every other lever in this book trades one cost for a smaller cost. This one trades a different resource: it spends arithmetic to buy time. Whether that is a bargain or a disaster depends on a question you have to answer before you enable it, and the answer is not in any vendor's documentation — it is in your own utilisation graph.
What it guarantees, which is unusually strong
Start with the property that makes this worth considering at all. From the original paper's abstract:
an algorithm to sample from autoregressive models faster without any changes to the outputs … Our method can accelerate existing off-the-shelf models without retraining or architecture changes.
DeepMind replicated it independently on a much larger model, with the same guarantee stated differently:
a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics … achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.
Two labs, two model families — T5-XXL at 11B and Chinchilla at 70B — 2–3× and 2–2.5×, with output distributions preserved. This is not quantisation or distillation. Nothing about quality is being traded. That is rare enough to explain why the technique spread so fast.
The mechanism is one sentence: a small draft model proposes γ tokens, the large model scores all γ+1 positions in a single forward pass, and a rejection-sampling rule accepts the longest correct prefix. You get several tokens for the price of one target-model call — when the draft guesses right.
The number that decides everything
That last clause is the whole chapter. The paper names the variable α — the probability that the target model accepts a draft token. Empirically:
approximation models that are a couple of orders of magnitude smaller than the target model tend to produce α values between 0.5 and 0.9.
α is described as "an intrinsic property of the models and the task." You do not set it. You measure it, and it moves with your workload — the same draft model gives you a different α on code than on prose.
Two documented consequences worth knowing before you benchmark:
- Greedy decoding accepts more. The paper observes that "α and walltime improvement are higher for argmax sampling (temp=0)." In their table, T5-small on translation goes from α=0.62 at temperature 1 to α=0.75 at temperature 0 — and 2.6× becomes 3.4×. Your sampling settings change your speedup.
- Even a trivial draft works a little. A bigram model scored α=0.2 on English→German, which the authors note "yields a 1.25X speed improvement, which is surprisingly high for this trivial approximation model."
The bill nobody quotes
Here is the part that belongs in a cost book and is almost always left out of the summary.
The paper is explicit that acceptance is free and rejection is waste:
When we accept the sample from Mq the increased concurrency is "free" and the total number of operations isn't increased. When we reject a guess though, computation is wasted.
So there are two theorems, not one. Theorem 3.8 gives the latency win. Theorem 3.11 gives the arithmetic you paid for it:
| Formula | |
|---|---|
| Walltime improvement | (1 − αγ+1) ⁄ ((1 − α)(γc + 1)) |
| Increase in total operations | (1 − α)(γĉ + γ + 1) ⁄ (1 − αγ+1) |
Setting the draft model's own cost to zero (the paper's own simplifying assumption for this comparison), here is what the trade actually looks like:
| α | γ | Latency | Arithmetic |
|---|---|---|---|
| 0.6 | 3 | 2.18× faster | 1.84× more |
| 0.6 | 7 | 2.46× faster | 3.25× more |
| 0.75 | 7 | 3.60× faster | 2.22× more |
| 0.8 | 7 | 4.16× faster | 1.92× more |
| 0.9 | 3 | 3.44× faster | 1.16× more |
| 0.9 | 7 | 5.70× faster | 1.40× more |
Read the first and last rows together. At α=0.6, γ=7 you burn 3.25× the compute to go 2.46× faster — you are losing arithmetic faster than you are gaining time. At α=0.9, γ=7 you get 5.70× for 1.40×, one of the best trades in this book.
Same technique. Same configuration. Opposite verdicts, decided entirely by α.
When the extra arithmetic is free, and when you pay cash for it
This is the question that determines whether the table above is a cost win or a cost disaster, and it is the same distinction memory bandwidth is the product is built on.
If you are memory-bandwidth-bound — small batch, latency-sensitive, weights streaming from HBM once per step while the compute units sit largely idle — the extra arithmetic is genuinely free. It runs in capacity you already rented and were not using. You rent the accelerator by the hour, so finishing sooner means more tokens per hour: cost per token falls by the full latency multiple.
If you are compute-bound — large batch, throughput-oriented, tensor cores already saturated — there is no idle arithmetic to spend. The extra operations displace real work. Throughput per GPU falls by the operations multiplier, and since the hourly rate does not change, cost per token rises by it:
| α | γ | Latency | Cost per token |
|---|---|---|---|
| 0.6 | 7 | 2.46× faster | +225% |
| 0.75 | 7 | 3.60× faster | +122% |
| 0.9 | 7 | 5.70× faster | +40% |
Even in the best row, a compute-saturated server pays 40% more per token for its speed. This is the only lever in this book that is supposed to raise cost per token. Whether that is a good deal is a product question — sometimes latency is worth 40% — but it must be a decision, not a surprise.
vLLM's documentation states the targeting rule in one line. Speculative decoding is there to
reduce inter-token latency under medium-to-low QPS (query per second), memory-bound workloads.
Their method table carries separate "Low QPS (latency focused)" and "High QPS (throughput focused)" columns for exactly this reason, and notes that the lightweight methods — n-gram and suffix decoding — give more modest speedups "without increasing workload during peak traffic." That is the same trade-off, exposed as a menu.
How far to speculate before it turns on you
γ is yours to choose, and the two theorems pull in opposite directions: more speculation means more tokens per accepted run, and more wasted work per rejected one. There is a point where the arithmetic multiplier overtakes the latency multiplier:
| α | Crossover | Just before it |
|---|---|---|
| 0.5 | γ=3 | γ=2: 1.75× speed, 1.71× ops |
| 0.6 | γ=5 | γ=4: 2.31× speed, 2.17× ops |
| 0.7 | γ=10 | γ=9: 3.24× speed, 3.09× ops |
| 0.8 | γ=24 | γ=23: 4.98× speed, 4.82× ops |
| 0.9 | none below γ=32 | — |
A weak draft model gives you almost no room. At α=0.5 you are underwater by γ=3. At α=0.9 the question does not arise in any practical range. If you are compute-bound, this table is your configuration limit; if you are memory-bound, ignore it and optimise latency directly.
There is a clean way to hold the operations multiplier in your head: it is exactly (γ+1) divided by the expected tokens per target-model run. You always pay for γ+1 positions. You just don't always get γ+1 tokens.
The paradox: the better draft model is slower
The most counterintuitive result in the paper, and the one that will save you a wasted week.
They tried three draft models against the same T5-XXL target at the same γ=7:
| Draft model | α (acceptance) | c (draft cost) | Measured speedup |
|---|---|---|---|
| T5-small | 0.75 | 0.02 | 3.4× |
| T5-base | 0.80 | 0.04 | 2.8× |
| T5-large | 0.82 | 0.11 | 1.7× |
T5-large guesses best and finishes last. Acceptance climbs monotonically with draft size — exactly as intuition says — and the speedup falls by half, because c, the draft's share of a target-model run, went from 2% to 11%. You now run an expensive model to save calls to a slightly more expensive one.
The paper's own conclusion: T5-small "with a good balance of c and α, provides the highest speedup." Not the best α. The best balance.
This is why "use a bigger draft model to improve acceptance" is the most natural and most wrong first optimisation. The paper's c stayed "always less than 0.05 and often negligibly close to 0" only because the drafts were a couple of orders of magnitude smaller. That gap is the design constraint, not an incidental detail.
The diagnostic
- Are you memory-bound or compute-bound right now? Check accelerator utilisation at your real batch size. This single answer decides whether the arithmetic multiplier is free or billed, and nothing else in this chapter matters until you have it.
- What is your α, on your traffic? Not the paper's. Measure acceptance on production-shaped prompts — this is the input to every other decision, and it varies by task.
- Are you sampling at temperature? Greedy decoding accepts more. If you sample, expect the lower end.
- Is your draft model two orders of magnitude smaller than your target? If not, you are likely paying c you cannot see. Try a smaller draft before a bigger one.
- If compute-bound, is γ below your crossover? Use the table above with your measured α.
- Would a draft-free method do? N-gram and suffix decoding need no second model and, per vLLM, avoid adding load at peak. Less upside, far less to go wrong.
What this chapter is not saying
It is not saying speculative decoding is a bad deal. On an interactive product that is memory-bandwidth-bound at small batch — which describes most single-user chat and coding assistants — it is close to free money: real latency improvement, no quality change, no retraining.
It is saying that it is a latency technique that appears in a cost book because of the direction it can move cost. The 2–3× is real and independently replicated. The arithmetic behind it is also real, and it is the one number the summaries drop. If you enable this on a throughput-saturated fleet and your bill goes up while your dashboards go green, nothing has malfunctioned. You bought exactly what the theorem said you were buying.