Batch inference submits a large set of requests to be processed asynchronously and returns results within a service window, typically up to twenty four hours, at around a fifty percent discount to synchronous on demand pricing. The discount is real and large, so the economics favour batch for every workload that tolerates delay: embeddings backfills, document classification, bulk summarization, evaluation runs, and offline enrichment, while interactive features stay on synchronous endpoints.
Here is how batch pricing works, which workloads qualify, the tradeoffs to weigh, and a worked example of the saving when a pipeline moves from synchronous to batch.
How does batch pricing work?
The major model platforms all offer a batch path. AWS Bedrock batch inference, Azure OpenAI Batch, Google Vertex batch prediction, and the batch APIs on other providers accept a file of many requests, process them when capacity is available, and deliver results to storage within a stated window. Because the provider can schedule the work into spare capacity rather than holding resources ready for an instant response, it passes back a discount that sits around half the synchronous per token rate across the major platforms. You trade immediacy for price, and for the right workloads that trade is heavily in your favour.
Which workloads should move to batch?
The test is simple: does a human or a live system need the answer right now? If not, the work belongs in batch.
| Workload | Batch or synchronous | Reason |
|---|---|---|
| Interactive chat and copilots | Synchronous | A user is waiting on each response |
| Embeddings backfill and reindexing | Batch | Runs offline, no user waiting |
| Bulk classification and tagging | Batch | Large volume, tolerant of a delivery window |
| Nightly summarization and digests | Batch | Scheduled, results consumed later |
| Evaluation and regression runs | Batch | Internal pipeline, not latency sensitive |
A common mistake is running large offline jobs through the synchronous endpoint simply because that is the integration the team already built. Re pointing those jobs at the batch path is often a half price cut with no change to the model or the output quality, only to the delivery time.
What are the tradeoffs?
The headline tradeoff is latency: batch results arrive within a window, not instantly, so anything user facing stays synchronous. The second is operational: batch is a submit and collect pattern, so your pipeline needs to write request files, poll or subscribe for completion, and handle partial failures within a job, which is a different shape from a request and response call. The third is forecasting: very large batches still consume real capacity, so a sudden enormous job can move the monthly bill even at the discounted rate, which is why batch volume belongs in the same spend forecast and governance as the rest of AI usage. None of these outweigh a near fifty percent discount for genuinely offline work, but they decide how you build the pipeline.
A worked example
A retailer enriched its catalogue by generating product descriptions and embeddings, and ran the whole pipeline through the synchronous endpoint because that was how the prototype was wired. The job was nightly and nothing waited on it. Moving it to the provider's batch API cut the cost of that pipeline by close to half, with results landing well inside the overnight window and no change to output quality. The only engineering was a submit and collect wrapper around the existing calls. Figures are verified against billing data and anonymised; the discount is indicative and depends on provider and model.
Frequently asked questions
How much does batch inference save?
What is the catch with batch inference?
Which AI workloads are good batch candidates?
Find the half price work in your AI bill
We audit AI pipelines to move every offline workload onto batch, size the discount, and fold batch volume into a spend forecast you can govern, as an independent advisory that takes zero provider commissions and answers only to you. Our guarantee: we reduce your cloud spend or we reimburse our service fee, on a Fixed Fee or a no risk Gainshare basis. Download the cloud cost optimization playbook, read the cross cloud cost optimization guide, and on profiles see training versus inference cost profiles. For monthly buyer side analysis, subscribe to The Cloud Spend Navigator.
Put a defensible number on your cloud spend.
No provider in the room, no published price list. Tell us your footprint and we will scope the savings against your billing data — we reduce your cloud spend or we reimburse our service fee.
The Cloud Spend Navigator: what changed in cloud pricing, commitments, and FinOps — no vendor spin.