Scheduling GPUs for cost comes down to five moves: partition each accelerator with multi instance GPU so several light workloads share one card, time slice where partitioning does not fit, separate training from inference onto different node pools with different rate strategies, run checkpointed training on spot or preemptible GPUs for the discount, and reserve only the stable floor of demand while the variable tail rides on demand. The default of one whole GPU per pod is what makes AI bills balloon, because a card that costs many times a CPU node sits half idle behind a single process. The same mechanics apply on EKS, AKS, GKE, and OKE, and commitments such as Savings Plans, Reservations, Committed Use Discounts, and Universal Credits apply only to the GPU hours that remain after scheduling has packed the cards.
GPUs do not behave like CPU and memory. They are coarse, scarce, and expensive, so the scheduling decisions that barely matter for general compute dominate the bill for AI. Here is how to make each one earn its money.
Why do GPU bills run away?
By default Kubernetes treats a GPU as an indivisible resource: a pod requests one GPU and gets the whole card for its lifetime, busy or not. Inference services, notebooks, and small fine tuning jobs rarely saturate a modern accelerator, so the cluster fills with cards running at a fraction of capacity while the meter charges for the whole thing. Because one GPU can cost more per hour than a dozen CPU nodes, a handful of half used accelerators outweighs every other line in the cluster. The waste is concentrated and expensive, which is good news: a few scheduling changes move real money.
How do you pack more work onto each GPU?
Two mechanisms raise GPU utilization. Multi instance GPU partitions a physical accelerator into several smaller, hardware isolated instances, each scheduled to a different pod with its own memory and compute slice. This suits inference and light workloads that never need a whole card, turning one expensive GPU into several billable units of useful work without the cards contending for each other. Time slicing instead lets multiple pods share a card by interleaving in time, which raises utilization for bursty or low duty cycle workloads but offers no isolation, so it fits trusted internal work better than strict production isolation. Choose partitioning when workloads are small and steady, time slicing when they are bursty and tolerant of contention.
Should training and inference share capacity?
No. Training and serving have opposite cost profiles and belong on separate node pools. Training is throughput bound, batchable, and usually fault tolerant if it checkpoints, which makes it an ideal candidate for spot or preemptible GPU capacity at a deep discount against on demand. Inference is latency sensitive and steady, so it wants reliable capacity sized to a forecast and is a poor fit for interruptible nodes. Putting both on one undifferentiated pool forces you to either overpay for training on reliable capacity or risk serving on interruptible nodes. Split them and each gets the rate strategy that fits.
| Workload | Profile | Scheduling move | Rate strategy |
|---|---|---|---|
| Training | Throughput bound, checkpointed | Dedicated pool, full cards | Spot or preemptible GPUs |
| Batch fine tuning | Periodic, tolerant of delay | Scale to zero between runs | Spot, on demand floor |
| Inference | Latency sensitive, steady | MIG partitions, packed | Reserved or committed floor |
| Notebooks and dev | Bursty, low duty cycle | Time slicing, out of hours off | On demand, shut down idle |
Table: matching each AI workload to a GPU scheduling move and a rate strategy.
How much should you commit and how much leave on demand?
Reserve the floor, not the peak. Once partitioning and time slicing have packed the cards and training has moved to spot, the cluster has a stable baseline of GPU hours that runs every day and a variable tail that spikes with experiments and launches. Cover the baseline with commitments and let the tail ride on demand or spot. Committing to the current run rate of an unpacked, unoptimised GPU estate locks the waste in for the length of the term, so the order matters: pack first, measure the new floor, then commit to it. This keeps the discount on the steady part without carrying utilization risk on the part that moves.
A worked example
A scaling fintech ran model serving on a GKE estate where every inference pod held a whole accelerator and training shared the same on demand node pool. Most serving cards sat well below half utilization, and training jobs that checkpointed cleanly were paying on demand rates for capacity they could lose and resume. Partitioning the serving GPUs so several inference pods shared each card, moving checkpointed training to preemptible GPU nodes, and scaling fine tuning pools to zero between runs raised utilization sharply and dropped the rate on the interruptible work. Only then was the remaining steady floor covered by commitments. The AI estate came out materially lighter with no change to serving latency. Figures are verified against billing data and anonymised.
Where this fits the wider review
GPU scheduling is the AI specific layer on top of the general container cadence in the Kubernetes cost review playbook. Make the utilization visible on the platform team cost scorecard, and size the steady floor with the method in how Kubernetes commitments map to the cloud. The full container picture across EKS, AKS, GKE, and OKE lives in the Kubernetes cost guide, which links up to the cross cloud cost optimization guide.
Frequently asked questions
Why are GPU costs so high in Kubernetes?
What is MIG and how does it save money?
Should AI training run on spot GPUs?
Schedule your GPUs with us
We pack and schedule GPU workloads across your EKS, AKS, GKE, and OKE estate and leave your platform team able to keep the cards full, with zero provider commissions. Our guarantee: we reduce your cloud spend or we reimburse our service fee. Pricing is either a Fixed Fee scoped up front or Gainshare, a share of verified savings with no retainer and no risk.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.