Chapter 4.77 of 9 in this part

Tail latency versus cost

Over-provisioning is what you buy to hide your tail. Queueing theory prices it — the last nine points of utilisation cost eleven times the wait — and Google's own conclusion is that tail-tolerance is how you stop paying for it.

6 min read·revised 2026-08-08

Every previous chapter in Part 4 has found the same shape: latency is bought with utilisation. This chapter is about the actual exchange rate, because it is steeper than intuition suggests and it is the reason your fleet is bigger than your traffic.

The law that sets the price

You cannot run a queue at full utilisation and keep it short. That is not an engineering failure, it is arithmetic.

For the simplest queueing model — single server, Poisson arrivals, exponential service times — mean waiting time in queue, expressed in units of one service time, is ρ / (1 − ρ), where ρ is utilisation. That produces this:

Utilisation Mean queue wait (service times)
50% 1.0
70% 2.3
80% 4.0
90% 9.0
95% 19.0
99% 99.0

Read the last two rows against the first. Going from 90% to 99% utilisation buys you nine points of efficiency and costs you eleven times the queue wait. Going the other way — accepting 50% utilisation instead of 90% — cuts waiting by 9× and doubles your fleet.

That is the price of latency, and it is non-linear. Halving your fleet does not double your latency; near the top of the curve it explodes.

Treat this as a model, not a measurement. Real LLM serving is not M/M/1 — service times vary enormously with sequence length, continuous batching changes the service discipline entirely, and continuous batching exists precisely to break the naive queueing assumptions. The shape is right; the constants are not yours. Use it to understand why the curve bends, then measure your own.

What you are actually buying when you over-provision

Google's Dean and Barroso named the cost of the tail directly, and their framing is the most useful sentence in this chapter:

It is challenging to keep the tail of the latency distribution low for interactive services as the size and complexity of the system scales up or as overall utilization increases.

And then the conclusion, which is a cost claim rather than a performance one:

We show that these techniques allow system utilization to be driven higher without lengthening the latency tail, avoiding wasteful over-provisioning.

Over-provisioning is the thing you buy to hide your tail. Idle capacity is not waste in the sense of a mistake — it is a purchase, made deliberately or by default, that converts money into headroom so that a burst does not land on a saturated server. The question is never "why are we at 40% utilisation," it is "is 40% the right price for the tail we promised?"

The paper's other observation matters for anyone running multi-step inference:

Temporary high latency episodes which are unimportant in moderate size systems may come to dominate overall service performance at large scale.

An agent that makes eight sequential model calls does not experience your median latency. It experiences the tail eight times, and the slowest of those eight sets the user's wait. This is why agent loops feel disproportionately slow relative to single-call benchmarks — the arithmetic of repeated sampling is working against you.

The SLO you promised versus the SLO you priced

Here is where the money leaks, and it is an organisational failure rather than a technical one.

Latency targets are usually written once, early, by someone reasoning about user experience — "responses should feel instant." Capacity is then provisioned to hit that number at peak. Nobody returns to ask what the number costs.

Three questions almost nobody has answered about their own service:

  1. Is the target p50, p95 or p99? These are different products at different prices. On the curve above, the same fleet delivers wildly different tail percentiles.
  2. Does it apply at peak or on average? A p99 target "at peak" prices your entire fleet against your worst hour.
  3. What would p95-instead-of-p99 save? This is a real number and it is usually large, because the tail is where the over-provisioning lives.

A latency SLO is a purchasing decision written in engineering language. If nobody has priced it, it was not chosen — it was inherited.

Why this chapter sits at the end of Part 4

Everything else in this part is a way to move the curve rather than move along it.

Continuous batching and KV cache management raise the throughput a given fleet can deliver, so the same latency target sits at a lower utilisation. Disaggregation removes prefill-decode interference, which is a tail-latency source that has nothing to do with load. Speculative decoding buys latency directly with arithmetic. Parallelism sets how efficiently the fleet converts rented hardware into work at all.

Those are the levers. This chapter is the reason to pull them. If your only tool for hitting a latency target is buying more accelerators, you are paying the steepest part of the curve, in the most expensive currency in this book.

The diagnostic

  1. What is your actual utilisation at peak? Not average — peak. If it is above ~80%, your tail is being set by queueing and no amount of per-request optimisation will fix it.
  2. What percentile is your SLO, and has anyone priced the alternative? Compute what p95 would cost versus p99. It is usually the largest single lever on this page.
  3. How many sequential model calls does a user-visible action make? Multiply your tail exposure by that number. Agent workflows are tail-amplifiers.
  4. Is your over-provisioning deliberate or residual? Idle headroom bought on purpose to protect a tail is sound engineering. Idle headroom nobody chose is just spend.
  5. Have you exhausted the throughput levers before buying capacity? Every one of them lets you hold the same SLO at lower utilisation, which is the only way to get both.
  6. Do you shape load, or only absorb it? Queueing, admission control, and moving anything asynchronous to batch all reduce the peak that sets your fleet size.

What this chapter is not saying

It is not saying run hot. Driving utilisation up without tail-tolerance is how you turn a cost problem into an incident, and the curve above shows exactly how fast that happens past 90%.

It is saying that the idle fraction of your fleet is a line item with a purpose, and that most teams can state neither its size nor what it is buying. The useful move is not to eliminate it — it is to name the SLO it protects, price that SLO honestly, and then check whether the throughput levers in this part let you buy the same promise for less.

Sources & methodcaptured 2026-08-08

Sources, captured 2026-08-08: the two quoted passages — that keeping the latency tail low is challenging "as overall utilization increases," and that tail-tolerant techniques "allow system utilization to be driven higher without lengthening the latency tail, avoiding wasteful over-provisioning," along with the observation that high-latency episodes unimportant at moderate size "may come to dominate overall service performance at large scale" — are quoted from the abstract of "The Tail at Scale," Jeffrey Dean and Luiz André Barroso, Communications of the ACM 56 (2013), pp. 74–80. I read the abstract, not the full CACM text, and have quoted only from what I read. The utilisation/wait table is computed by me from the standard M/M/1 result that mean queue wait equals ρ/(1−ρ) service times, verified in a separate pass; it is a textbook identity rather than a measurement, and the chapter says so explicitly because LLM serving violates the model's assumptions in several ways that matter — highly variable service times, and continuous batching changing the service discipline outright. The 11× figure is the ratio between the 99% and 90% rows and is likewise computed. No vendor pricing appears in this chapter, because the trade-off it describes is a property of queues rather than of any provider's rate card; the underlying hourly rates live in what an H100-hour costs. The claim that agent loops amplify tail exposure follows from the Dean and Barroso observation applied to sequential model calls — that application is mine, not something the paper states about LLMs, which did not exist in their present form when it was written. The three SLO questions and the diagnostic are my framing.

Want this done on your account rather than by you?

The handbook is the method, written out in full so you can run it yourself — that is the point of publishing it. If you would rather someone else did the first pass, the teardown is free and you keep the findings either way.