Tensor, pipeline, and expert parallelism

Splitting a model across devices is how you make it fit, and how you make it faster. It is also how you end up paying for accelerators that contribute almost nothing. This chapter is about the line between those, which is sharper and lower than most teams assume.

The law

Google's inference-scaling paper states the governing asymmetry in one sentence, and everything else in this chapter follows from it:

Both the weight loading part of the memory time and the non-attention compute time are proportional to the model size and inversely proportional to the number of chips. However, for a given partitioning layout, the time needed for chip-to-chip communication decreases less quickly (or not at all) with the number of chips used, so it becomes an increasingly important bottleneck as the chip count grows.

Read it as an accounting identity. The work each chip does halves when you double the chips. The talking does not. Every doubling buys less than the last, and past some count it buys nothing at all while doubling the bill.

There is no configuration that escapes this. There are only configurations that reach the wall later.

MFU is the bill

The metric that makes this legible is model FLOPS utilisation — what fraction of the hardware's theoretical arithmetic you actually convert into model work. It is the closest thing this book has to a direct efficiency-to-dollars conversion, because you rent the whole chip whether or not you use it.

Two reference points from teams with unusual incentive and ability to optimise:

The training figure is the headline of a paper whose entire purpose was composing tensor, pipeline and data parallelism well; they describe it as 502 petaFLOP/s. The inference figure is reached only in a specific regime — high batch, weight-gathered layout, "when the communication overhead is almost negligible."

These are the good numbers. They are what careful, well-resourced teams achieve on purpose. Whatever your deployment is doing, it is unlikely to be doing better.

The result that should change how you size a fleet

Buried in an appendix is the finding that most directly contradicts intuition:

at long-latency decodes, PaLM 62B achieves higher MFU than PaLM 540B, because the former uses 8-way model parallelism and the latter uses 64-way model parallelism.

A model roughly 8.7× smaller is running more efficiently than the large one — and the stated reason is not the model size but the split. Eight ways versus sixty-four. The authors' own next sentence is that they might improve the 540B "by reducing the model parallelism."

More parallelism is not more performance. It is more of a thing that has a cost, applied until the cost dominates. If you have ever added GPUs to a serving tier and watched throughput move less than proportionally, this is the mechanism, and it is expected behaviour rather than a misconfiguration.

Why decode is the expensive half

The paper is blunt about where efficiency goes:

The MFU for decode is typically much lower than for prefill.

Which is the same physics as prefill/decode disaggregation and memory bandwidth is the product, now expressed as a utilisation number. Prefill has large matrices to multiply; decode has one token per sequence and spends its time moving memory.

The scale of that memory movement is worth quoting, because it is easy to underestimate:

for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model's parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle.

Three terabytes of traffic, per token, on a 500B+ model — with the compute units idle throughout. You are renting arithmetic and buying memory bandwidth. Multiquery attention is the documented fix, cutting the KV cache by a factor of the head count and enabling "up to 32× larger context lengths."

Expert parallelism breaks the link between compute and memory

Mixture-of-experts is the third split, and it is different in kind. Tensor and pipeline parallelism divide a model that every token uses in full. MoE divides a model that each token only partly uses.

The Switch Transformer paper describes the trade in its first lines: MoE "selects different parameters for each incoming example," producing "a sparsely-activated model — with outrageous numbers of parameters — but a constant computational cost."

Mixtral 8x7B makes the arithmetic concrete. Eight feedforward experts per layer, a router picking two per token:

each token has access to 47B parameters, but only uses 13B active parameters during inference.

You compute with 13B. You must hold 47B. That is roughly 3.6× more memory than the arithmetic you paid for — and it is the single most important sentence about MoE economics, because of what this book has already established about where inference cost actually goes.

Why the headline oversells it

Prefill/decode disaggregation and the decode-MFU finding above both say the same thing: decode is memory-bandwidth-bound, not compute-bound. MoE reduces FLOPs. FLOPs were not your constraint.

So at small batch — the interactive, latency-sensitive case — a sparse model gives you far less than "13B not 47B" suggests. You still stream weights from memory, and there are still 47B of them to keep resident. The saving is real at large batch, where you genuinely were compute-bound and the arithmetic reduction lands.

MoE is a throughput-regime win that is routinely sold as a general one. The right comparison is not "13B of compute is cheap" but "47B of weights must fit, and be fed, on every device that might need them."

The third cost, which the paper names itself

Switch Transformer is candid that this is not free:

widespread adoption has been hindered by complexity, communication costs and training instability

Expert parallelism places different experts on different devices, so routing a token to its experts means an all-to-all exchange — the collective with the least favourable scaling behaviour of any in common use. The paper's own contribution is partly mitigation: they "simplify the MoE routing algorithm" for "reduced communication and computational costs," routing each token to a single expert rather than several.

Which returns this section to the chapter's opening law. The compute went down. The communication went up. Whether that is a good trade depends on the same variable as everything else here — whether you are compute-bound or not.

There is no best partitioning

The other practical finding is that the optimal layout is not a property of your model. It is a property of your batch size:

the optimal partitioning layout switches from the 2D weight-stationary layouts to the weight-gathered layouts as the batch size increases. The weight-gathered layouts are inefficient at low batch sizes, but eventually they become the most efficient at high batch sizes.

The authors draw the conclusion themselves — this "highlights the importance of flexibility in configuring the inference system with different choices depending on the application setting and goals."

So a partitioning strategy tuned for your offline batch job is the wrong one for your interactive endpoint, on the same model and the same hardware. This is the third chapter in Part 4 to land on the same shape: the right answer is workload-specific, and copying someone's configuration copies their workload's assumptions.

What you are actually buying with latency

The paper states the trade in cost terms directly, which is rare and worth keeping:

Lower latency can often be achieved with smaller batch sizes, but smaller batch sizes also result in worse MFU, resulting in a higher total cost (in terms of chip-seconds or dollars) per token.

That is the whole Pareto frontier in one sentence. There is no configuration that is simultaneously lowest-latency and lowest-cost. You choose a point, and the currency you pay for latency is utilisation.

The pipeline side has its own version of this. The Megatron authors name the two failure modes of naive scaling as "expensive cross-node communication" and "devices spending significant time waiting on other devices to make progress" — that second one is the pipeline bubble, chips idling while holding a stage. Their interleaved schedule recovers 10+% throughput against it, which tells you both that the waste is real and that it is not the dominant term.

The diagnostic

1. What is your MFU? Not your GPU utilisation percentage — that number can read high while doing nothing useful. If you cannot compute MFU, that is the first gap, because it is the only figure that converts directly into cost per token.
2. How many ways is your model split, and does it need to be? The 62B-beats-540B result says the split itself is a cost. Try fewer.
3. Is your tensor-parallel group inside one high-bandwidth domain? Tensor parallelism communicates constantly; crossing a slower boundary is the "expensive cross-node communication" failure mode by name.
4. Have you measured decode MFU separately from prefill MFU? They differ enough that a blended number hides the problem.
5. Did you pick your partitioning for the batch size you actually run? The optimum moves with batch size, and inherited configs rarely move with it.
6. Are you on multihead attention with long contexts and large batches? Then the KV cache may exceed your model by several times, and the fix is architectural.
7. If you run a sparse model, are you sizing memory on total or active parameters? Total. Always total. A 13B-active model that needs 47B resident is a 47B model for every provisioning decision you make.

What this chapter is not saying

It is not saying 52% MFU means you waste half your money. Theoretical peak FLOPS is not reachable — it assumes perfectly shaped, perfectly fed arithmetic that no real workload produces. The recoverable waste is the gap between your MFU and the best MFU achievable for your workload shape, not the gap to 100%.

That is precisely why MFU is useful as a comparative number rather than an absolute one. Measure your configuration, change one thing, measure again. The published figures here are for calibration — if you are at 15% and a well-tuned deployment reaches 50–76%, you have somewhere to go. If you are at 45%, the remaining headroom is small and the next lever is somewhere else in this book.

And it is not saying avoid parallelism. A model that does not fit in memory has no cost per token at all. The point is that parallelism is a tool with a price curve, and almost nobody plots where they are on it.