Running open weight models on your own GPUs beats a per token API only when inference volume is high enough and steady enough to keep reserved accelerators busy. The economics turn on utilization: a managed API charges per token and costs nothing when idle, while a reserved or on demand GPU costs the same at 20 percent load as at 80 percent. So the real comparison is the per token API rate against your fully loaded GPU cost divided by the useful tokens you actually serve. Self hosting wins at scale and steady demand; the API wins for spiky, low volume, or experimental traffic. Decide on the workload, not on a preference for owning the stack.
Here is the break even, the hidden costs, and the levers that move utilization in your favour.
Where is the break even?
Compare two numbers. The first is the per token price of a comparable managed API for your traffic. The second is your fully loaded GPU hourly cost divided by the tokens that GPU serves per hour at your real utilization. As long as the API rate per token sits above your self hosted cost per useful token, self hosting saves money. As utilization falls, your self hosted cost per token climbs until the API is cheaper.
This is why the same decision lands differently for two teams running the same model. The one with steady, high volume traffic keeps the cards busy and beats the API comfortably. The one with bursty daytime traffic and idle nights pays for capacity it never uses.
What hides the true cost?
Three costs are routinely left out of the self hosting case and each one bites.
- Idle GPU time. Accelerators reserved for peak sit idle off peak, and you pay for every idle hour. Utilization across the full day, not at peak, is the number that matters.
- Operating effort. Someone runs the serving stack, the autoscaling, the model updates, and the on call. That engineering cost is real and recurring, and it does not appear on the cloud bill.
- Capacity premium. GPU capacity is scarce, so on demand accelerators carry a premium and committed capacity reservations carry utilization risk. A reservation only pays off if you keep it busy, exactly like any other commitment.
How do you raise utilization?
Since utilization decides the outcome, the optimization is mostly about packing more useful work onto each card. Batch requests so the GPU processes many at once. Quantize the model to fit more throughput per card and serve more tokens per hour. Route low complexity traffic to a smaller model and reserve the large model for genuinely hard generation. Consolidate scattered workloads onto fewer, well used accelerators rather than spreading them thin. Each move lifts useful tokens per GPU hour, which is the denominator that makes a capacity reservation worth holding.
An AI platform team ran a frontier scale open weight model across on demand GPUs at roughly a quarter average utilization. Cost per useful token was well above the managed API they had moved off. We consolidated onto fewer cards, batched and quantized to raise throughput, routed the simple intents to a smaller model, and committed a capacity reservation sized only to the steady base load while bursts overflowed to on demand. Average utilization roughly doubled, the per useful token cost fell below the API rate, and the spiky overflow stayed flexible. Figures are verified against billing data and anonymised.
So which should you choose?
Choose the per token API for low volume, spiky, or experimental work where reserving capacity would mean paying for idle accelerators. Choose self hosting on your own GPUs when volume is high and steady, when you can keep utilization up, and when data control or latency justify operating the stack. Many estates run both: a managed API for the long tail and self hosted capacity for the steady, high volume core. The right split follows a defensible forecast of your traffic, which is the same discipline that governs every capacity reservation.
Frequently asked questions
When are self hosted open weight models cheaper than APIs?
What hides the true cost of self hosting?
How do you raise GPU utilization?
Model the decision against your own traffic
We build the break even on your actual inference volume and decide self hosting against managed APIs on the numbers, with zero provider commissions. Our guarantee: we reduce your cloud spend or we reimburse our service fee. Pricing is either a Fixed Fee scoped up front or Gainshare, a share of verified savings with no retainer and no risk.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.