Chapter 3.44 of 8 in this part

FLOPs, MFU, and the utilisation trap

A single-stream decode on an H100 runs at 0.34% of the chip's arithmetic capacity while nvidia-smi reports 100% utilisation. The ceiling has a closed form — batch size × bandwidth ÷ peak FLOPS — and it collides head-on with the VRAM limit from the last chapter.

7 min read·revised 2026-08-08

nvidia-smi says 100%. Your dashboard is green. The GPU is, in the only sense that tool means, fully busy.

It is also doing about a third of one percent of the arithmetic you are paying for. Both statements are true, and the gap between them is the most expensive misunderstanding in this part of the book.

Two numbers, both called utilisation

GPU utilisation as reported by nvidia-smi is the fraction of sampled time during which at least one kernel was resident. It answers "is the card doing something?" It does not answer "how much?" A kernel that reads memory and computes almost nothing pins it at 100%.

Model FLOPs Utilisation (MFU) is the useful arithmetic actually performed divided by what the chip could perform at peak. It answers the question your invoice asks.

For decode these two diverge by roughly two orders of magnitude, and the reason is entirely mechanical.

The ceiling has a closed form

Memory bandwidth is the product established that decode is bandwidth-bound. Push that one step further and you get a ceiling you can compute in your head.

During autoregressive decode with a batch of B sequences, each step:

  • performs roughly 2N FLOPs per sequence (N = parameters), so 2NB in total;
  • reads the entire weight matrix exactly once, regardless of B.

Time per step is therefore floored by bytes ÷ bandwidth, and the achieved arithmetic rate is:

achieved FLOPS = 2NB ÷ (2N ÷ BW) = B × BW

Divide by the chip's peak and the model cancels out entirely:

MFU_decode = B × BW ÷ F_peak

The model you serve does not appear. Only how many sequences you decode at once, the memory bandwidth, and the peak FLOPS of the silicon.

What that ceiling actually is

NVIDIA's headline tensor-core numbers all carry an asterisk reading "With sparsity" — so the dense figure, which is what dense LLM inference gets, is half the number on the page. The H100 SXM's 1,979 BF16 teraFLOPS is 989.5 dense.

GPU Dense BF16 peak Bandwidth MFU per sequence B for 50% MFU
A100 SXM 312.0 TFLOPS 2.04 TB/s 0.6535% 77
H200 989.5 TFLOPS 4.80 TB/s 0.4851% 104
H100 NVL 835.5 TFLOPS 3.90 TB/s 0.4668% 108
H100 SXM 989.5 TFLOPS 3.35 TB/s 0.3386% 148
L40S 733.0 TFLOPS 0.86 TB/s 0.1179% 425

On an H100 SXM:

Concurrent sequences MFU
1 0.34%
8 2.71%
32 10.83%
64 21.67%
148 50.11%

One user talking to a model on a $2.99/hour H100 uses 0.34% of its arithmetic. The other 99.66% is idle silicon that you are renting, waiting on memory.

Quantisation does not fix this

The obvious move is to quantise: fewer bytes per weight, fewer bytes to read, faster decode. It works — and it does nothing for MFU.

At FP8 the weights halve to N bytes, so achieved FLOPS becomes 2 × B × BW. But the H100's FP8 dense peak is exactly 2× its BF16 dense peak (3,958 vs 1,979 with sparsity; 1,979 vs 989.5 dense). Numerator and denominator both double:

B MFU at FP16 weights MFU at FP8 weights
1 0.3386% 0.3386%
9 3.0470% 3.0470%
148 50.1061% 50.1061%

Identical to fifteen decimal places. Quantisation buys you tokens per second. It does not buy you a better-used chip, because it shrinks the work and the clock in the same proportion. The only lever in B × BW ÷ F_peak that you control at serving time is B.

The collision

Here is where this chapter meets the last one. VRAM sizing computed how many concurrent sequences actually fit: Qwen2.5-72B at FP8, 4K context, weights taking 67.71 GiB off the top.

GPU Sequences needed for 50% MFU Sequences that fit Achieved MFU Shortfall
H100 SXM 148 9 3.05% 16.4×
H100 NVL 108 21 9.80% 5.1×
H200 104 58 28.14% 1.8×
L40S 425 0 — model does not fit

The batch size that would use the chip well is 16.4× larger than the batch size that fits in its memory. These two constraints are not independent problems to optimise separately — they are the same problem pulling in opposite directions, and on an 80 GB card with a 72B model, memory wins decisively.

This also re-explains, from a third direction, why the H200 was the right buy in the previous chapter: not because it is fast, but because capacity is what unlocks the batch that unlocks the arithmetic.

The cheaper chip is easier to use well

Notice the ordering in the ceiling table. The A100 SXM has the best MFU per sequence of any card here — 0.6535%, needing only 77 sequences for 50%. The L40S is the worst at 0.1179%, needing 425.

That is not a mistake. BW ÷ F_peak is the reciprocal of the chip's compute-to-bandwidth ratio, and newer accelerators have raised compute far faster than bandwidth. The A100 is 5.54× easier to saturate than the L40S, because it has proportionally less compute to leave idle.

So a low MFU on a modern chip is not necessarily bad engineering. Often it is the honest consequence of buying a chip whose balance does not match your workload — which is precisely the accelerator ladder's point, arriving again by a different route.

The diagnostic

  1. Stop reading nvidia-smi utilisation as a cost metric. It reports occupancy. It will read ~100% at 0.34% MFU.
  2. Compute B × BW ÷ F_peak for your setup. Three numbers, all public. It takes a minute and it is the ceiling — you cannot beat it by tuning.
  3. Halve every headline TFLOPS figure you read. NVIDIA's tensor-core numbers are quoted with sparsity; dense inference gets half.
  4. Check your ceiling against what fits. If the batch needed for a decent MFU exceeds the batch your VRAM allows, MFU is not your problem — capacity is, and no kernel tuning will help.
  5. Do not expect quantisation to raise MFU. It raises throughput. Those are different claims and only one of them is true.
  6. If MFU matters more than latency, batch harder or go bigger. Both are capacity plays, not compute plays.
  7. Measure achieved tokens per second too. MFU is an efficiency ratio; it says nothing about whether you are fast enough, which benchmarking tokens per second per dollar covers.

What this chapter is not saying

It is not saying low MFU means you are doing something wrong. For interactive single-stream serving, a low MFU is structural — it is what bandwidth-bound decode on a compute-heavy chip looks like, and no amount of engineering changes B × BW ÷ F_peak.

It is saying that "the GPU is at 100%" is not evidence of anything you are paying for, and that the number which is evidence has a two-line derivation almost nobody performs. Do it once for your own stack. If the answer is 3%, you now know that 97% of your GPU bill buys nothing — and, more usefully, exactly which constraint you would have to move to change that.

Sources & methodcaptured 2026-08-08

Sources, captured 2026-08-08. Peak FLOPS and bandwidth are from NVIDIA's own product specification tables: H100 SXM (BFLOAT16 Tensor Core 1,979 teraFLOPS, FP8 3,958 teraFLOPS, 3.35 TB/s, up to 700 W) and H100 NVL (1,671 teraFLOPS, 3.9 TB/s) from the H100 page, whose table carries the footnote "* With sparsity" — every dense figure here is that number halved, which is stated rather than assumed. A100 SXM (624 teraFLOPS BFLOAT16 with sparsity → 312 dense, 2,039 GB/s) is from the A100 page and L40S (1,466 teraFLOPS with sparsity → 733 dense, 864 GB/s) from the L40S page. The H200's dense peak is taken as equal to the H100 SXM's, on the basis established in memory bandwidth is the product that the two parts have identical compute and differ in memory; its 4.8 TB/s bandwidth is NVIDIA's. Computed by me and verified in a separate pass: the MFU = B × BW ÷ F_peak identity and every figure derived from it — all five per-sequence MFU values, all five 50%-MFU batch sizes, the H100 MFU-versus-batch table, the FP16/FP8 invariance (asserted equal to floating-point exactness, not approximately), the 16.4× / 5.1× / 1.8× shortfalls against the fit counts, and the 5.54× A100-versus-L40S ratio. The concurrency counts are carried over from VRAM sizing and inherit its declared assumptions — vendor GB treated as GiB, activations unmodelled, no paged attention — so they are upper bounds, which makes the real shortfalls larger, not smaller. The 2N FLOPs-per-token approximation counts the dominant matrix multiplies and omits attention over the KV cache, normalisation, and activation functions; it is the standard first-order model and is stated as an approximation rather than an identity. The derivation assumes decode is bandwidth-bound and that weights are read once per step, which is the regime this book's Part 4 chapters describe; it does not apply to prefill, which is compute-bound and can reach high MFU at batch size one. No published training-MFU figure appears. I attempted to source one from the Megatron-LM paper to contrast against these decode ceilings and could not resolve a specific percentage from a primary source in this pass, so none is quoted and no number is recalled from memory. The occupancy-versus-MFU framing and the diagnostic are mine.

Want this done on your account rather than by you?

The handbook is the method, written out in full so you can run it yourself — that is the point of publishing it. If you would rather someone else did the first pass, the teardown is free and you keep the findings either way.