GenAI token costs on GCP, controlled.

Token cost is one of the few cloud lines that engineering controls directly, on every single request.

How does Vertex AI actually bill generative models?

The unit is the token, roughly a few characters of text, and Vertex AI counts input and output separately. Input tokens are everything you send, the system prompt, the conversation history, retrieved context, and the user message. Output tokens are what the model returns. Output is typically priced higher than input, so a verbose response costs more than a long prompt of the same token count. Model choice multiplies all of it: the largest, most capable models carry the highest per token rate, while smaller and faster models cost a fraction. Add ons matter too, because grounding against search or your own data, very long context windows, and image or multimodal inputs each have their own pricing on top of the text tokens.

The practical consequence is that two systems serving the same number of requests can have wildly different bills depending on prompt length, output length, and model. None of that shows up until you measure tokens per request and attribute them to a feature.

When does provisioned throughput beat pay as you go?

Vertex AI offers provisioned throughput, where you reserve dedicated model serving capacity at a fixed cost instead of paying per token. The logic mirrors any reserved capacity decision: reserved throughput is cheaper than pay as you go above a break even utilization, the point where the fixed cost of the capacity you reserved equals what the same traffic would cost per token, and it gives predictable latency as a bonus. Steady, high volume production traffic that would keep the reserved capacity busy clears that bar and benefits. Spiky or low volume traffic does not, because reserved capacity sits idle between bursts while you still pay for it. The usual answer is a blend: provisioned throughput on the steady base, pay as you go on the spikes, sized to a real demand forecast rather than to the busiest minute.

Which levers cut the token bill fastest?

Four moves do most of the work, and all of them are engineering decisions. First, shrink every call: tighten the system prompt, prune conversation history to what the model needs, and cap output length, because input and output tokens both bill and most prompts carry dead weight. Second, route by difficulty: send simple classification or extraction to a small cheap model and reserve the large model for genuinely hard requests, which often halves cost with no quality loss on the easy majority. Third, cache: use context caching for stable, repeated context such as a long system prompt or a fixed knowledge block so it is not re sent and re billed on every request, and cache full responses for identical queries. Fourth, for steady high volume, move the base onto provisioned throughput. Each lever either lowers tokens billed or lowers the rate per token, and they compound.

Where this sits in the wider GCP picture

Token spend is the newest line in a GCP estate but it follows the same disciplines as the rest. The GPU and TPU side of AI cost, for training and self hosted inference, is covered in GPU and TPU cost control on GCP, and the monitoring you need to keep any of this attributable is in monitoring and observability costs on GCP. The full estate playbook lives in the GCP cost optimization guide, and the cross cloud view, including how Azure OpenAI and AWS Bedrock price the same tokens, is in the cross cloud cost optimization guide.

Put a number on your token spend

We measure tokens per request, attribute them to features, and apply the prompt, routing, caching, and capacity levers that bring the bill down without degrading the product. Our guarantee: we reduce your cloud spend or we reimburse our service fee. Pricing is either a Fixed Fee scoped up front or Gainshare, a share of verified savings with no retainer and no risk.