Quantization

Quantization is the one lever in this part of the book that changes the answer to "does it fit?" rather than just "how fast?" — and it is the one where the back-of-envelope number is reliably wrong.

The envelope says: FP16 is two bytes per parameter, FP8 is one, INT4 is a half. Divide accordingly. VRAM sizing used exactly that arithmetic and said so. This chapter checks it against checkpoints you can actually download, and the arithmetic loses.

What the published weights actually weigh

Total bytes of the safetensors shards in each repository, against what the divide predicts:

The BF16 rows land exactly on prediction, which is the control: the method is sound and the discrepancy is real. Everything below FP16 misses, and the smaller model misses by three times as much.

The block that never quantizes

Look at the dtype map inside an INT4 checkpoint and it is not one dtype. The 72B AWQ file holds 70,464,307,200 parameters packed into I32 — and 2,493,554,688 left at F16.

That residue is not slack. For the 72B: vocabulary 152,064 × hidden 8,192 = 1,245,708,288 parameters in the embedding table, doubled to 2,491,416,576 because the output head is untied, leaving 2,138,112 for the layer norms. The three numbers add up to the F16 block exactly.

The embedding table, the output head, and the norms stay at full precision in every scheme measured here — AWQ, GPTQ and FP8 alike. They are the parts where rounding does visible damage, so quantizers leave them alone.

That fixed block is the whole story, because it does not scale with depth:

A 7B model carries 43.7% of the 72B's embedding block on 10.5% of its parameters. Vocabulary size is set by the tokenizer, not by the model's depth, so a small model drags a nearly full-size embedding table behind it. At INT4 that block is four times the per-parameter cost of everything around it, which is how +14.4% becomes +46.3%.

What the extra bytes cost you

Recompute the fit table from VRAM sizing with real checkpoint sizes. Qwen2.5-72B, 4K context, 1.25 GiB of KV cache per request, RunPod Secure Cloud rates. n is concurrent requests; the figure beside it is dollars per concurrent-request-hour:

Three things fall out of that table.

The naive number plans for two users who do not exist. On an 80 GB H100 at FP8, dividing by two predicts 12.29 GiB free and nine concurrent requests. The real checkpoint leaves 9.95 GiB and seven — a plan set 28.6% above what the hardware delivers, measured against the seven you actually get.

INT8 fits fewer requests than FP8, 6 against 7, on identical hardware. GPTQ-Int8 is 71.83 GiB where FP8-dynamic is 70.05. Same nominal bit width, different real footprint, and the difference costs you a concurrent user.

INT4 moves the model down two rungs of the ladder. At 38.74 GiB it clears 48 GB for the first time, and an A40 at $0.44/hour serves the same seven concurrent requests as an H100 SXM at FP8 for 6.80× less — $0.0629 against $0.4271 per concurrent-request-hour. That is the largest single cost movement in Part 3, and it is a feasibility change, not a tuning change.

What it costs in quality

Qwen publishes the other half of the trade. Average across MMLU, C-Eval and IFEval:

INT8 is close to free at every size — under 1.1% down on the three larger models. INT4 is free on the 72B and expensive on the 0.5B, a spread of 0.12% against 13.66%: the same model family, the same quantizer, a 111× difference in what it costs.

The asymmetry is the finding

Both measurements point the same way, and it is not the way the technique is usually pitched.

A small model shrinks less than you expect and degrades more than you expect. A large model does neither. Quantization is a large-model technique that gets recommended to people running small models, usually as a way to squeeze onto a cheap card — which is precisely the case where you pay twice and save least.

The correct reading is the opposite one: INT4 is how you run a 72B on hardware priced for a 7B. Going the other way — INT4 on a 7B to fit a smaller card — surrenders 4.19% of the model's quality to recover a fraction of 5 GiB you probably had.

What quantization does not buy

It does not raise MFU. The last chapter showed the numerator and denominator move together, so FP16 and FP8 come out identical to floating-point exactness. Fewer bytes per weight means faster decode; it does not mean a better-used chip.

It does not buy arithmetic on an H100 either. From NVIDIA's own table: FP8 Tensor Core 3,958 teraFLOPS, INT8 Tensor Core 3,958 TOPS, both marked "* With sparsity" — identical peak, so dense figures are 1,979 apiece. Choosing INT8 over FP8 on this generation buys nothing in throughput; it is a decision about tooling and accuracy alone. What both buy over BF16 is the 2× step from 1,979 down to 989.5 dense — and the bandwidth saving, which is the part that actually moves decode.

The diagnostic

1. Read the file sizes, not the parameter count. Every claim in this chapter came from the byte totals of published shards. It takes one API call and it is the only number that is true.
2. Check the dtype map for a full-precision residue. If a repository reports F16 or BF16 alongside I32, that block is your floor. Compute vocab × hidden × 2 and see whether it accounts for it.
3. Expect the overhead to scale inversely with model size. The smaller the model, the larger its vocabulary-to-hidden ratio — 152,064 over 3,584 on the 7B against 152,064 over 8,192 on the 72B — and the more of it sits in the block that will not shrink.
4. Re-derive concurrency from the real weight size. Nine planned against seven delivered on an H100 is a 28.6% capacity error before you have served a request.
5. Take INT8 almost anywhere; take INT4 on large models. Under 1.1% on the three larger models against 4–14% for INT4 below 7B.
6. Do not quantize to raise utilisation. It raises throughput and capacity. Those are the wins; MFU is not among them.

What this chapter is not saying

It is not saying INT4 is safe because it was free on the 72B. One model family on three academic benchmarks is a signal, not a guarantee, and the published averages hide per-task variance that a production eval would surface. Run your own eval on your own task before you quantize anything you charge for.

It is saying that the two numbers everyone quotes about quantization — halve the bytes, lose a little accuracy — are both size-dependent in the same direction, and that nobody checks. The checkpoints are public. The check is one request.