A RAG pipeline carries five recurring cost components: embedding the documents and each query, storing and indexing the resulting vectors, running the retrieval query, the generation tokens sent to the model, and the orchestration compute that connects the steps. Generation tokens are usually the largest single line, but the costs that most often run away are embedding churn, where unchanged documents get re embedded, and oversized vector storage held at peak when actual recall needs are smaller. The good news is that the same trims that cut the bill, retrieving fewer and better chunks and passing a tighter prompt, tend to improve answer quality rather than hurt it.
Here is each component, what drives it, and where the realistic savings sit.
Where does a RAG bill actually go?
It helps to see the pipeline as a sequence of billed events rather than a single AI cost. The table below lists the five components, the unit you pay on, and the lever that moves each one.
| Component | What you pay for | Main cost lever |
|---|---|---|
| Embedding | Tokens embedded for documents and queries | Embed only changed content; cache query embeddings |
| Vector storage and index | Stored vectors and index memory or capacity | Right size the index; prune stale vectors |
| Retrieval | Query operations against the index | Cap chunks retrieved; cache frequent queries |
| Generation | Prompt and output tokens to the model | Trim context; right size the model |
| Orchestration | Compute running the pipeline and agents | Avoid loops; batch where latency allows |
Read across the table and a pattern emerges: every lever either removes work that did not need doing or shrinks the payload that flows to the most expensive step. That is the whole discipline of RAG cost control in one sentence.
Why does the embedding line creep?
Embedding feels cheap per token, so teams rarely watch it, and that is exactly why it creeps. The common failure is re embedding the entire corpus on a schedule when only a fraction of documents changed. If your knowledge base has tens of thousands of documents and a nightly job re embeds all of them, you are paying daily for work that a change detection step would reduce to the handful that actually moved. Embedding only new and modified content, and caching the embedding of repeated queries, often removes most of this line without touching answer quality.
Why is vector storage usually oversized?
Vector indexes are frequently provisioned for a worst case recall scenario and then left there. Two things inflate the storage line: keeping vectors for documents that are stale or never retrieved, and holding more index capacity or memory than the real query volume needs. Pruning vectors for retired content and sizing the index to observed retrieval patterns rather than the theoretical maximum brings this line back in proportion. On managed vector services the index capacity tier is a direct dial; on self hosted indexes it is the instance and memory footprint.
Why generation tokens dominate, and how to shrink them
Generation is usually the largest line because you pay for both the context you send in and the answer you get back, and RAG systems tend to over stuff context. Retrieving the top twenty chunks when the top three answer the question multiplies the prompt token cost for no quality gain, and often hurts quality by burying the relevant passage. Capping the number of retrieved chunks, compressing or summarising retrieved context before it reaches the model, and routing requests to the smallest model that meets the quality bar are the three levers that move this line the most. Caching full responses for repeated questions removes the generation cost entirely for those calls.
How does this map across AWS, Azure, GCP, and OCI?
The components are the same on every cloud; the billing surfaces differ. On AWS, embedding and generation run through managed model endpoints and the underlying compute can sit under Savings Plans, with the Cost and Usage Report attributing tagged RAG workloads. On Azure, Azure OpenAI usage needs its own token and provisioned throughput discipline, and Log Analytics costs can creep if the pipeline logs verbosely. On GCP, generation and embedding draw on managed model services with committed use discounts on supporting compute, and BigQuery usage appears if retrieval analytics run there. On OCI, flexible compute shapes size the orchestration tier precisely and Universal Credits draw down across the stack. In every case, tag the pipeline so each component is attributable, because you cannot trim a line you cannot see.
A worked example
A business services company ran an internal knowledge assistant on RAG and treated the whole thing as a model token bill. Breaking it into components showed three quarters of the spend in generation, driven by retrieving fifteen chunks per query and passing them all to a frontier model. A nightly full re embedding job was the second largest line. The team capped retrieval to the most relevant chunks, added a summarisation step before generation, switched a change detection embedding job in for the full re embed, and routed routine lookups to a smaller model. Answer quality held or improved on internal evaluation while the monthly bill fell sharply, with the largest saving coming from the generation token reduction. Figures are verified against billing data and anonymised.
Frequently asked questions
What are the cost components of a RAG pipeline?
Why does my RAG bill keep growing?
How do you reduce RAG retrieval cost?
Break down and cut your RAG bill with us
We help enterprises attribute and optimise every component of a RAG pipeline across AWS, Azure, GCP, and OCI, so the architecture scales without the bill scaling with it. Our guarantee: we reduce your cloud spend or we reimburse our service fee, on either a Fixed Fee or a no risk Gainshare basis. Start with the guide, then send us your billing data.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.