Open any model provider's pricing page and you will find two numbers: a rate for input tokens and a higher rate for output tokens. Across the live catalogue the price index tracks, the median ratio sits around 4x, and plenty of models price output considerably higher than that.
Most people notice this, shrug, and move on. It is the most actionable fact in model economics.
Two different computations
Serving a request has two phases, and they are not variations on a theme. They stress different parts of the machine.
Prefill processes your prompt. Every token in the input can be handled at once, because they are all already known. This is a large matrix multiplication across the whole sequence, and it saturates the GPU's compute units. Prefill is compute-bound, and it is embarrassingly parallel.
Decode generates the response, one token at a time. Each new token depends on every token before it — the model cannot write token 50 before it has written token 49. So decode is inherently serial. Worse, generating each single token requires streaming the model's weights out of memory and through the compute units. For one token. The GPU spends most of decode waiting on memory bandwidth, not computing. Decode is memory-bandwidth-bound.
That asymmetry is physical. It is a property of transformer inference on current hardware, not a pricing choice, and it will not be negotiated away. Providers price it because it costs them.
The practical consequence: a thousand tokens of input and a thousand tokens of output are not remotely the same amount of work. Your bill reflects that whether or not your architecture does.
What follows from it
Long outputs are the expensive thing. If you are choosing between sending more context and generating more text, sending context is usually the cheaper side of the trade. This inverts the instinct most engineers arrive with, which is that prompts are the thing to trim.
"Be concise" is a cost control. An instruction that reliably shortens responses is a direct multiplier on your largest per-token rate. Not a rounding error — on a 5x-premium model, cutting average output length by a third takes real money off the bill. Test it as you would any other change; measure the output token count before and after, and check quality did not move with it.
Structured output beats prose when you only need the data. If a downstream system parses the answer, asking for prose and then parsing it means paying decode rates for words no human reads. Ask for the fields. Keep the field names short — you pay for those too, on every call.
Streaming does not make it cheaper. It makes latency feel better, which is worth doing, but every token still bills. Do not confuse a perceived-performance win with a cost win.
Reasoning tokens bill as output. Models that produce extended internal reasoning before answering are charging you decode rates for that reasoning. This can be excellent value when the task genuinely needs it and a large, silent, recurring waste when it does not. Whether a step needs a reasoning model is a question worth asking per call site, not once per application.
Where caching enters
Cached input reads are the other side of this. Where a provider offers prompt caching, a repeated prefix can often be re-read at a substantial discount to the normal input rate — the index shows the published discounts by model, and they are frequently large.
Caching attacks the prefill side. It is most valuable exactly where you have a long, stable prefix — a system prompt, a tool schema, a document being asked about repeatedly — followed by a short variable suffix. Ordering matters: the stable part must come first, because the cache matches on prefix. A request that puts the user's question before the system prompt gets no benefit at all, and this is a real and common bug.
Caching does nothing for your output costs. If your bill is decode-heavy, caching is not your lever.
The diagnostic
Take your top call sites and compute the ratio of input tokens to output tokens for each.
Input-heavy (say 10:1 or higher) — a RAG pipeline, a document Q&A, a long-system-prompt agent. Your money is in prefill. Look at caching, at whether the retrieved context is actually being used, at whether the system prompt has accumulated instructions nobody has read in a year.
Output-heavy (approaching or below 1:1) — content generation, code generation, long-form summarization. Your money is in decode. Look at length limits, at output format, at whether a reasoning model is earning its cost, at whether you are generating text a human will never read.
The two cases have almost nothing in common, and a team optimizing the wrong one can work hard for weeks and move nothing. Which is why the price index publishes blended cost at both a 3:1 and a 10:1 mix: the cheapest model at one ratio is often not the cheapest at the other, and the ranking changes more than people expect.
Find your ratio first. Then optimize.