Chapter 4.99 of 9 in this part

Benchmarking tokens per second per dollar

A throughput number is not a number until you know what it was measured under. MLPerf's rulebook is effectively a catalogue of how benchmarks lie — including a rule that bans quoting TDP as power. Use it as a checklist, including on this book.

6 min read·revised 2026-08-08

Every chapter in Part 4 has handed you a multiple. This one is about how to check one — because a throughput figure with no conditions attached is not a measurement, it is a claim.

The most useful artefact here is not a benchmark result. It is MLPerf's rulebook, which exists because vendors kept publishing numbers that could not be compared, and which reads as a systematic list of every way that happens.

Four scenarios, because load pattern is part of the number

MLCommons is explicit that a benchmark is a load pattern plus a metric, not a speed:

MLPerf has defined four different potential test for its benchmarks. A given scenario is evaluated by a standard load generator generating inference requests in a particular pattern and measuring a specific metric.

The four are Single stream, Multiple stream, Server, and Offline — and the split shows up even in how power is reported, with system power for Server and Offline versus energy per stream for the stream scenarios.

The distinction that matters for cost: Offline lets the system see all the work at once. Server delivers requests as they arrive. Those are the batch-job and the live-endpoint cases from tail latency versus cost, and they produce very different numbers on identical hardware.

An Offline throughput figure does not tell you your Server capacity. If a vendor quotes one number, the first question is which one — and if the answer is "throughput," you have not been told.

The quality target is what stops the cheating

This is the rule most people never think about, and it is load-bearing:

Each benchmark is defined by a dataset and quality target.

Without a pinned accuracy floor, throughput is trivially purchasable with worse output. Quantise harder, truncate context, cap output length, decode more greedily — every one of those raises tokens per second, and none of them is free. A benchmark that reports speed without fixing quality is measuring how fast a system can produce something, which is not what anyone is buying.

When someone tells you their stack is faster, the question is not "how much faster." It is "at what quality, on what data."

Closed versus Open, and why the distinction is not pedantry

The Closed division is intended to compare hardware platforms or software frameworks "apples-to-apples" and requires using the same model as the reference implementation.

The Open division is intended to foster innovation and allows using a different model or retraining.

Both are legitimate. They answer different questions. A number from Open cannot be compared with a number from Closed, because in Open the model itself is a variable — and the Open submission format even adds a "Model Used" row precisely because it "may or may not match the Closed Division requirement."

In practice, most vendor blog-post benchmarks are Open-division comparisons presented with Closed-division confidence.

Can you actually buy it?

MLPerf sorts results by availability, which is a category almost no vendor benchmark bothers with:

  • Available — "contain only components that are available for purchase or for rent in the cloud"
  • Preview — "must be submittable as Available in the next submission round"
  • RDI — "experimental, in development, or internal-use hardware or software"

A result on hardware you cannot rent is not a price you can pay. Benchmark numbers from unreleased silicon are real measurements and useless budget inputs, and the distinction is invisible unless someone labels it.

The power rule, which is the sharpest thing in the document

MLPerf measures energy as:

the measured average AC power (energy) consumed by the entire system … the AC power is measured at the wall.

And then explicitly refuses the marketing substitute:

MLPerf Power is only capable of measuring and validating the full system power. Any other references to power in any description (e.g., a TDP configuration, a power supply rating) are [not valid measurements].

TDP is a thermal design specification, not a measurement of what your machine draws. Anyone computing performance-per-watt from a spec sheet is doing arithmetic on a number that was never a power reading. For anyone owning hardware or paying for colocation power, this is the difference between a real operating cost and a plausible-looking one.

Two more disciplines worth stealing: every Closed submission includes a link to code, and MLCommons maintains a public change log because "published results are sometimes modified or invalidated." Even audited, adversarially-reviewed numbers get retracted. Your internal benchmark deserves less confidence than that, not more.

The checklist

Before you believe any throughput number — vendor, colleague, or your own:

  1. Which scenario? Offline or Server. If nobody said, you have a number with no load pattern.
  2. What quality target? Fixed accuracy, or was speed bought with output?
  3. Closed or Open? Same model, or was the model a variable too?
  4. Available, or unreleased? You cannot budget against hardware you cannot rent.
  5. What input and output shape? A 1,500-in/13-out coding trace and a long-form chat load are different workloads, as disaggregation showed with real production data.
  6. Is power measured at the wall or read off a spec sheet? If it is TDP, it is not a measurement.
  7. Can you reproduce it? Is there code, or a claim?

Turning it into cost

Tokens per second is an engineering metric. Tokens per dollar is the one that belongs in a budget, and converting between them requires exactly one more number — the fully-loaded hourly cost of the system under test, which is the subject of what an H100-hour costs and the bill around the GPU.

The conversion is trivial arithmetic. The hard part is that both inputs must describe the same configuration — and the most common error in AI cost work is dividing a benchmark throughput measured in one regime by an hourly rate for a system provisioned in another.

Apply this to this book

Consistency requires saying so plainly: every multiple in Part 4 is a benchmark claim, and the checklist applies to all of them.

The 24× in continuous batching is against a naive transformers loop — a baseline choice, which is why KV cache management exists partly to state the 2–4× figure against real serving systems. The 6.7× in managed endpoints is four vendors on one day at published rates. The 76% MFU in parallelism is TPU-specific.

Every one of those carries its conditions in its provenance note, on purpose. A multiple without conditions is marketing, and that is true of a book as much as a vendor page.

Sources & methodcaptured 2026-08-08

Sources, captured 2026-08-08: all quoted material — the four-scenarios-with-a-standard-load-generator framing, the "each benchmark is defined by a dataset and quality target" rule, the Closed division "apples-to-apples" definition requiring the same model as the reference implementation, the Open division allowing "a different model or retraining," the Available / Preview / RDI availability categories with their definitions, the system-power-measured-at-the-wall methodology and the explicit rejection of TDP configurations and power-supply ratings as substitutes, the Open-division "Model Used" row, the code-link submission requirement, and the note that published results are sometimes modified or invalidated — is quoted from MLCommons' MLPerf Inference: Datacenter benchmark page. The four scenario names are given there in the power-reporting section (system power for Server and Offline, energy per stream for Single stream and Multiple stream); the page links to the full scenario and latency-constraint rules in the MLPerf inference_policies repository, which I did not read in this pass — so no specific latency constraint or per-scenario metric definition is quoted here, and anyone needing those should go to the rules document rather than take this chapter's summary as complete. This chapter deliberately introduces no new cost multiples — it is a methodology chapter, and inventing a figure to illustrate a chapter about figure hygiene would be self-defeating. The bracketed completion in the power quotation marks where the source's sentence continues beyond what was captured; the substance quoted before it is verbatim. The checklist, the tokens-per-second-versus-tokens-per-dollar framing, and the application of the checklist to this book's own claims are my framing.

Want this done on your account rather than by you?

The handbook is the method, written out in full so you can run it yourself — that is the point of publishing it. If you would rather someone else did the first pass, the teardown is free and you keep the findings either way.