Vertex AI spend on GCP is driven by two distinct cost models that need separate governance. Generative models charge per input and output token or per character under pay as you go, with optional Provisioned Throughput that reserves capacity for a committed term. Custom training and online prediction bill by node hour on the machine type and accelerators you choose, and a prediction endpoint charges for its nodes whether or not requests arrive. The largest savings come from right sizing the model to the task, trimming prompt and output length, tuning prediction autoscaling, and undeploying idle endpoints. Treat Vertex AI as its own governance line, because it grows faster than the rest of the data estate and the usual rightsizing playbook only covers half of it.
This sits alongside BigQuery and the rest of the GCP data platform, but the AI line behaves differently. Here is how to keep both axes under control.
What drives the generative side of the bill?
For generative models, three variables set the cost of every call: the model, the input token count, and the output token count. Output usually costs more than input, and a larger model multiplies the rate on both. The most expensive pattern is a verbose prompt sent to an oversized model that returns a long answer, repeated at scale. The levers follow directly: right size the model to the task, because a smaller model often handles classification, extraction, and routing at a fraction of the cost of a frontier model reserved for hard generation; trim prompts and cap output length; and cache deterministic or repeated calls so you are not paying tokens twice. Retrieval pipelines that re embed unchanged documents inflate the count quietly, so the retrieval layer deserves the same scrutiny as the generation layer.
Why do prediction endpoints get expensive?
The node hour side is where Vertex AI surprises teams. An online prediction endpoint keeps nodes running to serve traffic, and it bills for those nodes whether or not requests arrive. An endpoint deployed at a high minimum replica count, or left running after an experiment ended, charges node hours around the clock for capacity no one is using. The same applies to custom training jobs that request large accelerator backed machines and to notebooks left running overnight.
- Right size the machine type. Match the prediction machine type and accelerator to the model's real serving needs rather than the largest option that fits.
- Tune autoscaling. Set a sensible minimum replica count so idle periods scale down, and let the endpoint grow under load rather than provisioning peak capacity permanently.
- Undeploy idle endpoints. Sweep for endpoints serving little or no traffic and undeploy them. An experiment that shipped an endpoint and was never cleaned up is a common silent cost.
GPU and TPU accelerators carry the heaviest node hour rates, so the discipline that governs them is the same that governs accelerated training, covered in GPU and TPU cost control on GCP.
Pay as you go tokens or Provisioned Throughput?
For generative traffic, Vertex AI offers a commitment style option that mirrors the decision everywhere else on GCP. Pay as you go per token means you pay only for what you call, ideal for spiky, experimental, or low volume workloads. Provisioned Throughput reserves model capacity for a committed term, buying predictable throughput and a lower effective rate once volume is high and steady. As with Committed Use Discounts on compute, it pays off only against a defensible forecast of sustained usage, so reserve capacity once traffic is steady enough that the committed capacity beats accumulated token charges, and keep the spiky top on pay as you go.
A European SaaS company ran a Vertex AI feature that defaulted every generative call to a frontier model and kept three prediction endpoints deployed from earlier experiments at a high minimum replica count. Routing simple intents to a smaller model, capping output, undeploying the two idle endpoints, and tuning autoscaling on the remaining one cut both the token line and the node hour line materially. With generative volume then steady, part of the traffic moved to Provisioned Throughput. Figures are verified against billing data and anonymised.
How do you govern it like a budget?
Vertex AI spend escapes control when no one owns it. Tag AI workloads and deploy them into projects that map to a team, so cost allocation attributes every token and node hour to an owner. Set budgets and alerts per application and per environment, because a forgotten endpoint or a looping agent in a development project can outspend production. Review Vertex AI as its own line in the monthly cadence, split into the generative token side and the training and prediction node hour side, and put a recurring rhythm on it with observability for AI cost. The full data and platform picture lives in the GCP cost optimization guide, and the cross cloud view sits in the cross cloud cost optimization guide.
Frequently asked questions
How is Vertex AI priced?
Why do Vertex AI prediction endpoints get expensive?
When does Provisioned Throughput pay off on Vertex AI?
Put governance on your Vertex AI spend
We help enterprises govern both sides of Vertex AI, the generative token line and the training and prediction node hours, before either becomes an unmanaged line on the bill. Our guarantee: we reduce your cloud spend or we reimburse our service fee. Pricing is either a Fixed Fee scoped up front or Gainshare, a share of verified savings with no retainer and no risk. We take zero provider commissions and answer only to you.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.