essays · guides · the handbook

Blog

Essays on AI costs, model choices and building useful systems. More writing in the newsletter.

EssayNovember 3, 20265 min read

xAI retired Grok Imagine Quality overnight — your bill fell a cent an image, and so did your quality

Effective November 2, 2026, xAI retired the grok-imagine-image-quality slug: requests now resolve silently to grok-imagine-image-2.0 at low quality, $0.01 per image cheaper at every resolution — $0.05 to $0.04 at 1K, $0.06 to $0.05 at 1.5K, $0.07 to $0.06 at 2K. Nothing errors, so the only way to know is the model field in your responses. Restoring the old quality on medium costs a cent more than you were paying.

EssayNovember 2, 20266 min read

One pool for ChatGPT and Codex: OpenAI merges your allowance in December — and hasn't told you how big it is

Starting in December, Plus, Pro, and Business plans draw from one token-based allowance shared across ChatGPT and Codex. Instant chat is exempt; at the limit you wait, upgrade, or buy credits. OpenAI has published no per-plan token figure and no exact start date — so the size of the pool, and whether it is bigger or smaller than the two limits it replaces, is unknowable until it lands.

EssayOctober 30, 20266 min read

OpenAI cancelled the Pro cut — and doubled Pro instead: the $200 plan keeps its 20x

On October 29, one day before the scheduled Pro 200 cut, OpenAI cancelled it and doubled Work and Codex allowances on Pro 100 and Pro 500. Prices are unchanged, Plus is untouched, and Ultrafast stays exclusive to Pro 500. The cut that was supposed to land today didn't — and the math now favours Pro over Plus again.

EssayOctober 28, 20267 min read

Anthropic gives government 40% off Claude. The $1 era is over — the metered bill is here.

Anthropic and the GSA expanded the OneGov agreement on October 27, 2026: a 40% discount on Claude token usage for federal, state, local, tribal and territorial governments, with no minimum order, replacing the $1 OneGov offer that ended October 31. At list rates, Sonnet 5.5 falls from $2/$10 to $1.20/$6 per million tokens — but only if you buy through the government channel.

EssayOctober 15, 20263 min read

When AI picks 7: why random doesn’t mean equal odds

A model can sample randomly without giving every number an equal chance. What a preference for seven does—and does not—tell builders about randomness, model costs and creative work.

EssayOctober 14, 20263 min read

Gemini 4 Argon: API prices are set to double December 1

Gemini 4 Argon launched with introductory API rates of $2/$10 per million tokens through November 30, 2026. Budget for the published December 1 rates, then evaluate it on your own workload.

EssayOctober 6, 20262 min read

The model got 33% cheaper. The bill fell 12%.

A teardown of the Sonnet 4.6 to 5 switch shows why token prices lie: the new tokenizer emits roughly 30% more tokens for the same text, so a third off the price becomes an eighth off the bill. Price the task, not the token.

EssayOctober 6, 20262 min read

Your free Gemini is now a Lite Gemini

On October 9, Google cuts free Gemini accounts to Flash-Lite only and pulls Pro from the $4.99 AI Plus plan. The free tier becomes a Lite-only demo — the squeeze goes downmarket.

ReadingAugust 4, 20261 min read

The bookshelf — five books worth owning for AI and cloud cost

The five physical books worth owning in a field that doesn't have its own book yet — one per layer of the stack, from FinOps canon to serving systems — plus the sixth you assemble yourself from the primary literature.

EssayAugust 4, 20263 min read

Why I don't do pay-from-savings — and what I do instead

Contingency pricing rewards the easy 10% of cloud savings and structurally ignores the hard 30%. Why a fixed fee, agreed before the work starts, is the only model that keeps an advisor independent — and what the engagement includes instead.

Cost HandbookAugust 11, 20266 min read

Attribution at scale

Attribution fails at the join, not the tag. The three clouds disagree about case, length and count, so the only portable key is the intersection — 63 characters, lowercase — and on the trace side the GenAI token convention is still marked Development.

Cost HandbookAugust 11, 20266 min read

Capacity planning and forecasting

Forecasting dollars for AI is forecasting a moving target. Forecast capacity instead — and note that a commitment discount is really a utilisation threshold. At AWS's published 62.4% three-year discount, the committed rate is 37.6% of on-demand, so the machine has to be busy 37.6% of the time to break even.

Cost HandbookAugust 11, 20265 min read

Cost in CI

Budget alerts cannot be the control, because the vendors document how slow they are — AWS Budgets updates up to three times a day, 8–12 hours apart, and Google states that an alerts-only budget is not a hard cap. The gate has to move to the pull request.

Cost HandbookAugust 11, 20266 min read

Incident economics

A cost incident is not detected by the billing alert — AWS documents updates up to three times a day, 8–12 hours apart. On one p5.48xlarge at $55.04/hour, a 12-hour blind window is $660.48 spent before anyone is told. The control has to be a ceiling, not an alarm.

Cost HandbookAugust 11, 20266 min read

Showback, chargeback, and ownership

The choice is not showback versus chargeback, it is whether the number lands on someone who can change it. Azure states the constraint that decides your start date — tags are not applied to historical data, so allocation can only ever begin the day you tagged.

Cost HandbookAugust 11, 20265 min read

The operating cadence

The right review rhythm is not a management preference — it is set by how fast the data arrives. AWS updates budgets up to three times a day, Azure needs up to 48 hours on a new subscription and never backdates a tag. Reviewing faster than the pipeline reports is reviewing noise.

Cost HandbookAugust 11, 20266 min read

Unit economics for AI features

Three layers of the same industry publish three different denominators for the same cost. NVIDIA sells cost per token, Microsoft warns shareholders about margin, and Doximity buries inference inside a cost-of-revenue line whose growth outran revenue by almost exactly two to one.

Cost HandbookAugust 11, 20266 min read

Vendor management and negotiation

No vendor publishes what it concedes, so this chapter carries no negotiated figures and never will. What it does instead is turn the public record into leverage — the published discount ladder is your floor, the switching cost from lock-in is your credibility, and a listed vendor's own filings tell you how badly it needs you.

Cost HandbookAugust 10, 202610 min read

Owning the hardware

Buying converts a variable cost into a fixed one, and the bill arrives whether the machine is busy or idle. CoreWeave's filings put numbers on it — $25.83 million of capex per megawatt of live capacity, against $6.04 million of revenue per megawatt-year, which means depreciation alone consumes 71.3% of revenue at the six-year life the company itself uses.

Cost HandbookAugust 9, 202610 min read

Interconnect and topology

Inside a p5 the GPUs talk at 900 GB/s each. Outside it they talk at 50 GB/s each — an 18× cliff at the edge of the box, on the fastest network AWS sells. The fabric itself is cheap: doubling it on a Trainium instance costs 15.26%, and the marginal gigabit costs a sixth of the average one.

Cost HandbookAugust 9, 20269 min read

Quantization

Halving the precision does not halve the checkpoint. Measured on published INT4 weights, a 72B model comes out 14.4% larger than the arithmetic says and a 7B model 46.3% larger — because one block never quantizes. The same asymmetry runs through the quality loss, and it points the technique at the opposite end of the model range from where most people apply it.

Cost HandbookAugust 9, 20268 min read

TPU, Trainium, and AMD

A Trainium instance costs 60.9% less per hour than an H100 instance from the same vendor on the same day — and 6.8% more per TB/s of memory bandwidth, which is what decode actually consumes. The inference-specific chip is the real bargain, and it is not the one being marketed at you.

Cost HandbookAugust 8, 202610 min read

Autoscaling accelerators

The cold start is not a latency problem you tolerate. It is a billed line item you pay for on every scale-up — and AWS's serverless inference does not offer GPUs at all.

Cost HandbookAugust 8, 20266 min read

Benchmarking tokens per second per dollar

A throughput number is not a number until you know what it was measured under. MLPerf's rulebook is effectively a catalogue of how benchmarks lie — including a rule that bans quoting TDP as power. Use it as a checklist, including on this book.

Cost HandbookAugust 8, 20267 min read

FLOPs, MFU, and the utilisation trap

A single-stream decode on an H100 runs at 0.34% of the chip's arithmetic capacity while nvidia-smi reports 100% utilisation. The ceiling has a closed form — batch size × bandwidth ÷ peak FLOPS — and it collides head-on with the VRAM limit from the last chapter.

Cost HandbookAugust 8, 202611 min read

Lock-in is a cost, and it is largest for the smallest buyer

AWS will refund a Savings Plan in full — if the hourly commitment is $100 or less, you ask within seven days, and the calendar month hasn't ended. Every exit in this market is an absolute dollar threshold, which means they are generous to small commitments and meaningless to large ones. The small buyer still carries the larger cost.

Cost HandbookAugust 8, 20266 min read

Multi-tenancy and co-location

A model that needs 20 GB on an 80 GB card wastes three quarters of what you rented. Hardware partitioning fixes that with a guarantee; time-slicing fixes it with a hope. NVIDIA describes the failure mode precisely, and it is the exact resource this book says you are short of.

Cost HandbookAugust 8, 20268 min read

Prefill/decode disaggregation

Your inference server runs two workloads with opposite hardware appetites and bills them on one machine type. Splitting them delivered 1.4× throughput at 20% lower cost on Azure production traces — and made the older GPU the better buy for half the job.

Cost HandbookAugust 8, 20269 min read

Regions, residency, and sovereignty

AWS runs 106 regions. The H100 p5 instance exists in 15 of them, and exactly one is in the European Union. Before you price residency, check whether the hardware is there at all — and know that an "EU (…)" region label is not a statement about the European Union.

Cost HandbookAugust 8, 202610 min read

Reservations, savings plans, and commitment maths

AWS advertises "up to 72% off." On a p5 the best real number is 62.4%, and the flexible plan gives you 33.4%. The premium you pay for flexibility scales with how scarce the silicon is — 1.20× on a c5, 1.77× on a p5.

Cost HandbookAugust 8, 20269 min read

Serverless GPU versus dedicated

The crossover is not set by how much traffic you have. It is set by how long your jobs are — because a 2-second job on RunPod's defaults bills 17 seconds. And Modal's headline rate is the preemptible one; guaranteed execution is 3× that.

Cost HandbookAugust 8, 202610 min read

Speculative decoding

The one lever in this book that can make your product faster and your bill bigger at the same time. It buys latency with arithmetic, and the exchange rate is set by a single number most teams never measure.

Cost HandbookAugust 8, 20267 min read

Spot, preemptible, and the economics of interruption

AWS gives you two minutes. Azure gives you thirty seconds. Google's default is zero — and Google says outright that if you want warning on a GPU, don't use Spot. None of those windows is long enough to save a loaded model.

Cost HandbookAugust 8, 20266 min read

Tail latency versus cost

Over-provisioning is what you buy to hide your tail. Queueing theory prices it — the last nine points of utilisation cost eleven times the wait — and Google's own conclusion is that tail-tolerance is how you stop paying for it.

Cost HandbookAugust 8, 202610 min read

Tensor, pipeline, and expert parallelism

Compute per chip falls as you add chips. Communication does not. That asymmetry is the entire economics of splitting a model — it is why a 62B model can beat a 540B one on efficiency, and why a mixture-of-experts model computes 13B parameters but must store 47B.

Cost HandbookAugust 8, 20268 min read

The accelerator ladder

Fifteen rungs spanning 13.9× in price. Rank them by dollars per unit of memory bandwidth instead and the order scrambles — the cheapest card on the ladder is also the cheapest bandwidth, and the mid-range card everyone buys is the most expensive thing on it.

Cost HandbookAugust 8, 202610 min read

The neocloud landscape

Two of these companies file with the SEC, and one discloses that a single customer is 67% of its revenue. The other five disclose nothing, because they don't have to. That is the tier boundary — not price.

Cost HandbookAugust 8, 202610 min read

The small-business path

Fifty thousand a year is $5.71 an hour. That single conversion decides everything — it buys between 0.48 and 2.21 continuously-running H100s depending only on where you buy, and it puts you at 5.71% of the one refund threshold AWS offers. The small buyer's advantages are real and almost nobody uses them.

Cost HandbookAugust 8, 20269 min read

The total cost of self-hosting

Amazon shortened its server depreciation from six years to five and told the SEC why — AI. A DGX H100's nameplate says 19.8 kW and its own heat spec says 11.30. Both numbers change the buy-versus-rent answer, and neither appears in the spreadsheet people actually build.

Cost HandbookAugust 8, 20268 min read

VRAM sizing

Qwen2.5-72B's KV cache costs exactly 320 KiB per token — so one request at its full 32K context needs 10 GiB, on top of the weights. Work that through and an H200 serves 6.4× the concurrency of an H100 for 1.47× the price, which inverts the per-gigabyte ranking entirely.

Cost HandbookAugust 5, 20265 min read

Continuous batching

Batch size is where you sit on the roofline, and KV cache fragmentation is what caps it. The measured gap between naive and good serving is 24×.

Cost HandbookAugust 5, 20266 min read

Egress, storage, and the bill around the GPU

GCP charges 33% more than AWS to move a byte out, and gives you a free tier 100× smaller. At 10 TB a month that gap alone is worth two weeks of a dedicated H100.

Cost HandbookAugust 5, 20266 min read

Hyperscaler versus neocloud

Run one H100 continuously with 10 TB of egress and the spread is 4.5×. The neocloud discount is real and it is not free — this is what it costs you, itemised.

Cost HandbookAugust 5, 20265 min read

KV cache management

The 24× and the 2–4× are different baselines, and conflating them will make you buy the wrong thing. Paging fixes waste; sharing fixes duplication. Only one of them helps a chatbot.

Cost HandbookAugust 5, 20265 min read

Making it stick

Savings that depend on someone remembering decay. Anthropic's Spend Limits API makes the cap a data structure with inheritance and an approval loop — and the gotchas in it are instructive about what durable cost control requires.

Cost HandbookAugust 5, 20268 min read

Managed inference endpoints

One open-weights model, four vendors, a 6.7× spread. AWS and Google price it identically to the cent — and the independent that undercuts them both is not the one you would guess.

Cost HandbookAugust 5, 20265 min read

Memory bandwidth is the product

H100 and H200 have identical compute. The H200 serves 1.9× the inference throughput. Once you understand why, most accelerator decisions answer themselves.

Cost HandbookAugust 5, 20265 min read

Reading your first bill

Anthropic exposes usage at minute granularity and cost only at daily. Each endpoint has a blind spot the other covers, and two attribution holes return null. Here is how to read what you are actually spending.

Cost HandbookAugust 5, 20265 min read

The cost levers, ranked

Nine levers, ordered by the multiple each one has actually been measured at in this book — not by how much attention each gets. The largest is 24×. Most teams pull the 1.4× one first.

Cost HandbookAugust 5, 20265 min read

The output premium

The 5× output premium is constant across a 10× price ladder. Changing model tier scales your bill; it never changes your exposure. Only fewer output tokens does that — and reasoning models bill their thinking as output.

Cost HandbookAugust 5, 20265 min read

What an H100-hour actually costs

The same 8 GPUs cost $55, $88, or $98 an hour depending only on which logo is on the invoice. The full spread is 4.7×, and the break-even maths that decides commitments.

Cost HandbookAugust 5, 20264 min read

What you are actually paying for

Output costs 5× input. Cache reads cost a tenth of input. Cache writes cost more than input. Three published ratios, and almost every app-layer decision follows from them.

In the pipeline

Next
The AI bill, from electrons to your P&L — the cost cascade in one essay (Curriculum M0 artifact)
Soon
What a GPU actually costs per hour — the spreadsheet nobody publishes — with the open TCO model (M1)
Soon
I measured what a million tokens actually costs — a weekend, an H100, and vLLM (M4)
Soon
The actuarial problem inside your AI agent — 30x run-to-run cost variance, measured (M6)