Prefill/decode disaggregation

Every chapter so far has treated inference as one workload. It is two, and they want different machines.

Memory bandwidth is the product established the physics: prefill is compute-bound, decode is memory-bandwidth-bound. This chapter is about the operational consequence, which is larger than it sounds — you are currently buying one machine type to satisfy two opposite requirements, and paying the maximum of both.

The measurement, from production

Microsoft characterised this on real Azure inference traces from two production services, captured 11 November 2023 and partly released publicly. Their summary of the phase split:

Batching during the prompt phase is compute-bound, whereas the token phase is limited by memory capacity.

And the conclusion they drew from it:

Token generation can be run on less compute-capable hardware for better Perf/W and Perf/$ efficiencies.

That is a direct instruction to stop buying your newest GPU for half your workload.

The headline result, quoted exactly:

Compared to current designs, Splitwise clusters achieve up to 1.4× higher throughput at 20% lower cost. Alternatively, they can deliver 2.35× more throughput under the same power and cost budgets.

Note the or. Those are two different operating points on the same design, not a compounding claim.

The number that reframes the whole problem

Buried in the characterisation is the most useful fact in this chapter. The two production traces have wildly different shapes:

The coding service's median request is 1,500 tokens in and 13 out — a ratio of roughly 115 to 1. That workload is almost entirely prefill. A conversation workload is not.

This is why "what fraction of my fleet should do prefill?" has no universal answer, and the paper's own provisioning proves it. Under the same technique and the same machine type, they provisioned:

The same system, tuned for two real workloads, wants fleet compositions that differ by more than 4×. Any vendor benchmark for disaggregation is therefore a statement about their trace, not yours. Measure your own prompt-to-output ratio before you believe any published split.

Why colocation costs you, precisely

The second paper on this — DistServe, from a different group — names the mechanism:

this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases.

Two distinct costs, and the second is the expensive one.

Interference is the obvious cost: a long prefill lands in the batch and every in-flight decode stalls behind it, so your time-per-output-token spikes for reasons unrelated to those requests.

Coupling is the structural cost. One pool means one machine type, one parallelism strategy, one scaling decision for two workloads with opposite needs. As DistServe puts it, systems must "prioritize one latency over the other, or over-provision compute resources to meet both." That over-provisioning is a permanent line on your bill, and it is invisible because it looks like normal capacity.

The multiple, and what it is actually measuring

DistServe reports:

DistServe can serve 7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.

Read that carefully, because it is easy to over-quote. This is goodput, not throughput — the paper is explicit that it measures "the maximum rate that can be served within both TTFT and TPOT constraints." It is requests-per-GPU that actually met your latency targets.

If you have no latency SLO, this number is not available to you. A batch pipeline with no per-request deadline can already extract full throughput by batching harder; disaggregation would add complexity and a network hop for no gain. The 7.4× is what you recover when latency constraints were forcing you to leave capacity unused — which is a very common situation, and precisely why the technique matters, but it is not a free throughput multiplier.

Same discipline as KV cache management: get the baseline right before you promise anything.

The power argument, which almost nobody makes

This is where the cost case gets genuinely interesting, because it is not about chips at all.

The two phases respond to power caps in opposite ways. Prompt-phase power draw rises with batch size; token-phase draw "does not vary when increasing the number of tokens to process." And on latency under a cap:

• Prompt phase: "highly sensitive to the power cap and the latency increases substantially."
• Token generation: "incurs almost no latency impact when power capping."

So the decode fleet can be power-capped nearly for free. That matters because datacentre cost is provisioned on peak power, not average — a point the paper makes directly about why providers care.

The iso-power comparison makes it concrete: under the power budget of 40 DGX-H100 machines, 70 DGX-A100 machines fit — 75% more machines, for the same power envelope. And on the hardware itself: "the memory-to-compute ratio favors A100 over H100," with A100 showing "better or equal inference cost and energy overall compared to H100" on these workloads.

The older GPU is the better buy for decode. Not a compromise — better on cost and energy. That is the sharpest available rebuttal to buying the newest accelerator for everything, and it puts real evidence under the claim in the cost levers, ranked that chip choice is over-weighted relative to how you schedule work onto chips.

The overhead, which is smaller than the objection

The standard objection is that you now have to ship the KV cache between machines. Measured:

• Total overhead versus prompt computation time: under 7%.
• With per-layer transfer overlapping compute, non-overlapped time is roughly 8 ms on A100, 5 ms on H100 — near-constant rather than growing with prompt size.
• The H100 setup transfers about twice as fast, tracking its doubled interconnect bandwidth (200 vs 400 Gbps).

Naive serialised transfer does grow linearly with prompt size, so the optimisation is doing real work. But the honest summary is that the transfer is not the reason to avoid this — the reason to avoid it is having no latency SLO, or a fleet too small to split.

The diagnostic

1. What is your median prompt-to-output token ratio? 115:1 and 1.7:1 are different businesses. This single number drives your fleet split and you probably already log it.
2. Do you have a real latency SLO, separately for first token and per output token? If not, the headline multiples do not apply to you — they are goodput-under-SLO figures.
3. Are your TPOT spikes correlated with other users' long prompts? That is interference, and it is the symptom disaggregation removes.
4. Is your fleet big enough to split? Two pools of one machine each is worse than one pool of two. This is a technique for fleets, not for a single node.
5. Have you priced your decode pool on last-generation hardware? On these traces the A100 was better on cost and energy for token generation.
6. Do you provision on peak power? If you own or colocate, the decode pool is nearly free to power-cap. If you rent by the hour, this lever belongs to your provider, not you.

What this chapter is not saying

It is not saying disaggregate by default. It adds a network dependency, a second pool to schedule, and a failure mode that did not exist before. For a small deployment, or one with no latency contract, colocation is simply correct.

It is saying that the "one machine type for inference" assumption is a purchasing decision you never consciously made, and that on production traces it costs about 20% of the bill at equal throughput. The multiples here are workload-specific to an unusual degree — more so than anything else in Part 4 — which is why the diagnostic leads with measuring your own ratio rather than adopting anyone's published split.