GenAI services bill by the token, a token being roughly three quarters of a word, and split the price into input tokens you send and output tokens the model generates. Output tokens typically cost several times more than input tokens, so a verbose model that writes long answers is expensive in a way the headline per million rate hides. The four levers that move the bill are model choice, input context length, output length, and pricing mode, on demand versus a provisioned throughput commitment. A buyer who governs those four controls the spend; one who only negotiates the rate does not.
Here is how token pricing actually works and where the money goes.
Why do input and output tokens cost different amounts?
Generating an output token is more compute intensive than reading an input token, so providers price them separately, with output often three to five times the input rate depending on the model. This asymmetry matters because it inverts intuition: a long document you summarise once is cheaper than a short prompt that produces a long answer repeatedly. It also means the cost of a feature depends on its output shape. A classification call that returns one word is cheap regardless of context; a drafting feature that writes paragraphs is where output cost compounds. Read every per million figure as indicative and confirm against the current provider pricing page, because model rates change often.
What actually drives the bill?
Four levers, in rough order of how often they are mispriced.
- Model choice. Frontier models can cost an order of magnitude more per token than smaller models that handle routine tasks well. Routing simple calls to a smaller model is usually the largest single saving.
- Input context. Every token of retrieved context, system prompt, and chat history is billed on every call. Bloated context and uncapped history quietly multiply cost.
- Output length. Because output tokens are the expensive ones, capping response length and asking for concise output cuts the costly half of the bill directly.
- Pricing mode. On demand per token suits variable or early workloads; provisioned throughput, a committed capacity reservation, lowers the effective rate for steady high volume but carries utilisation risk like any commitment.
How do caching, batching, and provisioned throughput change the math?
Three mechanisms cut the rate without changing the feature. Prompt caching lets you reuse a large static context, a system prompt or a knowledge base, at a reduced rate on repeat calls, which suits assistants that send the same preamble every time. Batch processing trades latency for a discount on work that does not need an immediate answer, such as overnight enrichment or evaluation. Provisioned throughput, sold as committed capacity units on the major clouds, lowers the effective per token cost for predictable high volume in exchange for a commitment you carry whether you use it or not, the same risk adjusted decision as any cloud commitment. The buyer rule is to exhaust the free levers, model routing, context discipline, and output caps, before committing to provisioned capacity.
| Lever | What it cuts | Tradeoff |
|---|---|---|
| Route to a smaller model | Per token rate, often an order of magnitude | Quality on hard tasks, needs evaluation |
| Cap output length | The expensive output half of the bill | Less verbose answers |
| Trim and cache context | Input tokens billed every call | Engineering effort, cache design |
| Batch non urgent work | Per token rate on deferred jobs | Higher latency |
| Provisioned throughput | Effective rate at steady volume | Utilisation risk on the commitment |
How should an enterprise buyer govern token spend?
Govern it like any unit cost: pick a denominator and watch the cost per unit, not just the total. For a support assistant that is cost per resolved ticket; for a drafting feature it is cost per document. That framing exposes a feature whose token cost outruns its value before it scales.
A European SaaS company ran a customer support assistant entirely on a frontier model with uncapped chat history, at a cost per resolved conversation that made the feature hard to justify. We routed routine intents to a smaller model, capped output length, trimmed and cached the system context, and reserved the frontier model for escalations only. Token cost per resolved conversation fell by roughly 60 percent with no measured drop in resolution rate, and only then did a small provisioned throughput commitment on the steady base load make sense. Figures are verified against billing data and anonymised.
This sits inside the cross cloud cost optimization guide and pairs with AI capacity reservations and commitments and why GenAI broke the cloud budget.
Frequently asked questions
Govern your GenAI token spend before it scales
We break your token bill into model, context, output, and pricing mode, cut the cost per unit with the free levers first, and size any provisioned throughput commitment on a defensible base load. It is the same discipline behind the 31 percent median reduction we see in the first 90 days, with zero provider commissions. Our guarantee: we reduce your cloud spend or we reimburse our service fee.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.