AI is the fastest growing line on the enterprise cloud bill, and it does not respond to the rules that work on the rest of the cloud. Classic rightsizing assumes a server you can shrink; AI cost is driven by tokens consumed by managed model APIs, by GPU hours and capacity reservations for training and self hosted inference, and by provisioned throughput commitments for predictable serving. Those drivers need their own discipline. The buyer side approach is to measure cost per AI outcome rather than raw spend, to choose the right serving mode for each workload among on demand tokens, provisioned throughput, and your own GPUs, and to govern the things that quietly inflate the bill: oversized context, retries, idle GPU reservations, and shadow AI that no one is tracking.
This guide covers why AI spend behaves differently, the three serving modes, what drives token cost, GPU economics, and governance.
Why does AI spend behave differently?
Three things make AI cost its own category. First, managed model APIs bill per token, counting both the input you send and the output you receive, so cost scales with prompt size and response length rather than with server time. Second, the GPUs that train and serve models are scarce and expensive, so capacity, not just utilization, is the constraint, and an idle reserved GPU is a large standing cost. Third, training and inference have opposite profiles: training is a bursty, capacity hungry project cost, while inference is a steady, per request running cost that grows with adoption. A budget built for general compute will misread all three, which is why AI spend so often surprises the finance team.
What are the three serving modes and when does each pay?
Most AI cost decisions come down to how you serve inference. There are three modes, and the right one depends on volume and steadiness.
| Serving mode | How it bills | Best for |
|---|---|---|
| On demand tokens | Per input and output token, no commitment | Spiky or low volume workloads and early experimentation |
| Provisioned throughput | Reserved model capacity for a period at predictable price | Steady, high volume serving that keeps the capacity busy |
| Self hosted on your GPUs | GPU hours plus operating overhead | Very high steady volume, or control and data residency needs |
The decision is a utilization question. On demand tokens are cheapest when volume is low or unpredictable, because you pay only for what you use. Provisioned throughput wins when volume is high and steady enough to keep the reserved capacity well used. Running your own GPUs pays at very high steady scale or where control matters, but only if you keep the GPUs busy, because idle accelerators erase the saving fast. Picking the mode per workload, rather than defaulting the whole estate to one, is where the structural saving lives.
What drives token cost, and how do you cut it?
On managed APIs, the bill is the number of tokens times the model's price, so the levers are the token count and the model choice. Trimming prompt and context size removes tokens you are paying for on every call, and large retrieval contexts are a common, quiet inflator. Caching repeated or near duplicate responses avoids paying twice for the same answer. Batching inference where latency allows raises efficiency. And model right sizing matters most of all: paying frontier model prices for a task a smaller model handles well is the most expensive habit in AI spend, and choosing the smallest model that meets the quality bar can change the unit cost by an order of magnitude. Where a workload runs at scale on a narrow task, fine tuning a smaller model can beat prompting a large one on cost.
For your highest volume AI feature, compute cost per outcome: total spend divided by the units the business cares about, such as resolved tickets or generated documents. If you cannot produce that number, you cannot tell whether a model or mode change is worth it, and measuring it is the first move.
What governs GPU economics?
For training and self hosted inference, the GPU is the cost. The choices are on demand GPUs, reserved capacity or capacity blocks, and spot where the workload tolerates interruption. On demand suits short, unpredictable bursts; reserved capacity and capacity blocks suit planned training runs and steady serving; spot suits interruptible training and batch inference at a deep discount. The dominant waste is idle GPU: a reserved accelerator that sits unused bills at full rate, so utilization governs whether a reservation was wise. Capacity reservations and commitments for AI follow the same discipline as any commitment, sized to a defensible forecast of steady demand, not to the largest discount.
A Fortune 500 enterprise had routed every AI feature through a single frontier model on on demand tokens, with large retrieval contexts and no caching, and held a block of reserved GPUs that ran well below full utilization. We measured cost per outcome per feature, moved the high volume narrow tasks to a smaller model, trimmed context and added response caching, shifted steady high volume serving to provisioned throughput, and resized the GPU reservation to genuine steady demand with spot used for interruptible batch. Effective AI cost fell materially while quality held, and the spend became forecastable. Figures are verified against billing data and anonymized.
Where to go next
Understand how this line broke the budget in why GenAI broke the cloud budget, put a number on it the board trusts in AI unit economics the board understands, and get the commitment side right in AI capacity reservations and commitments. The way AI fits the wider program sits in the cross cloud cost optimization guide.
Frequently asked questions
Why is AI so expensive on the cloud?
How do you reduce AI inference costs?
What is provisioned throughput?
Take the next step on your cloud spend
We are an independent buyer side advisory that cuts public cloud spend across AWS, Azure, GCP, and OCI. We hold over $2.4B in annual cloud spend under management, take zero provider commissions, and deliver a 31 percent median reduction in the first 90 days. Our guarantee is simple: we reduce your cloud spend or we reimburse our service fee. Pricing is either a Fixed Fee scoped up front or Gainshare, a share of verified savings with no retainer and no risk to you.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.