Observability for AI cost means instrumenting spend at three layers so it is attributable, not just visible. The provider bill gives you aggregate tokens, GPU hours, and provisioned throughput. Application telemetry records, per request, the model called, input and output tokens, the feature, and the customer or tenant. A derived cost per request joins the two so you can see which feature, prompt, or customer drives the bill. Most teams instrument the first and third layers and skip the middle one, which is exactly the layer that turns an opaque AI bill into something a buyer can act on this week.
This article sits in the AI and GenAI cloud costs cluster, and links up to the cross cloud cloud cost optimization guide. AI workloads are the fastest growing line on most cloud bills, and the hardest to see.
Why is AI cost so hard to see?
A managed model bill tells you how many tokens you consumed in a month. It does not tell you that one new feature accounts for most of them, that a verbose system prompt doubled input tokens, or that a single customer's integration is looping. The unit on the bill, tokens, is real money, but it is disconnected from the application behaviour that produced it. GPU based workloads have the mirror problem: you see GPU hours, but not which training run or inference path consumed them. Until the bill and the behaviour are joined, every cost conversation is guesswork, and guesswork is how AI budgets overrun.
What should you instrument?
Capture three layers and join them.
Layer one, the provider bill. Pull the cost data: token counts and rates from the managed model service, GPU hours from compute, and provisioned throughput charges. On AWS the Cost and Usage Report is the source of truth, with Bedrock usage broken out; Azure surfaces Azure OpenAI usage; GCP surfaces Vertex AI; OCI surfaces its GPU shapes. This is the ground truth dollar figure.
Layer two, request level telemetry. This is the layer that matters and the one teams skip. For every model call, log the model, input tokens, output tokens, latency, the feature or endpoint, and the tenant or customer. This is application instrumentation, not a billing export, and it lives in your tracing or logging stack. Without it, attribution is impossible.
Layer three, derived cost per request. Multiply tokens by the model's rate, or amortise GPU and provisioned throughput cost across the requests served, to get a cost figure per call. Aggregate it by feature, tenant, and model. Now the bill has a shape: you can see the expensive feature, the costly customer, and the prompt that drifted.
What does good AI cost observability surface?
Once the three layers are joined, specific actions appear that the raw bill hides.
- Prompt and context bloat. Cost per request rising while volume is flat usually means input tokens are growing, an oversized system prompt or unbounded context. You see it the day it starts, not at month end.
- Model mismatch. A high volume, low complexity feature running on a premium model is a cost per request you can cut by routing it to a cheaper model with no quality loss.
- Provisioned throughput utilization. If you reserved throughput, layer one and layer three together show whether you are using what you committed to, the same utilization discipline a reservation demands.
- Shadow and runaway spend. A tenant or integration consuming far more than its share stands out immediately when cost is attributed per customer.
A Fortune 500 retailer with a customer facing assistant on a managed model watched its AI bill climb month over month with no clear cause; the provider invoice only showed rising token totals. Adding request level telemetry, model, input tokens, output tokens, and feature per call, and deriving cost per request revealed that a single internal feature on a premium model drove most of the growth, that its prompt had grown to several times its original length, and that it could run on a cheaper model. Trimming the prompt and routing the feature cut the cost per request sharply while volume kept growing. Figures are verified against billing data and anonymised.
How does this connect to governance and commitments?
AI cost observability is the data foundation under two other disciplines. It feeds AI unit economics, the cost per outcome a board can read, and it feeds capacity decisions, because you cannot size a provisioned throughput reservation or a GPU commitment without knowing your real consumption pattern. Treat observability as step one: instrument first, then optimise prompts and model choice, then commit to capacity once the demand pattern is clear. The cross cloud governance context is in the cloud cost optimization guide, and the rhythm for acting on what you see is in the AI cost review cadence.
Frequently asked questions
Make your AI spend visible with us
We instrument the three layers, join the bill to request level behaviour, and hand you a cost per request your product owners can act on, then optimise the prompts, models, and commitments it reveals. Our guarantee: we reduce your cloud spend or we reimburse our service fee.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.