How to Reduce LLM API Token Costs for AI Agents

You reduce LLM API token costs for agents by treating cost as an architecture decision, not a pricing problem. Measure cost per completed task, cache static context correctly, replace raw tool calls with code execution, route work to the cheapest model that can do the job, and self-host inference once volume justifies it. Teams that do this cut agent bills 40 to 90 percent. Teams that just bolt a SaaS wrapper on top watch their invoice grow faster than the model's own price drops.

Why is your agent bill exploding even though tokens got cheaper?

Here's the paradox: per-token prices have fallen sharply over the past two years, and your bill went up anyway. Bain & Company found that price per token fell by roughly half over one year, while token consumption over the same period grew 4.5x, more than erasing the savings.¹

The reason is structural, not incidental. Agentic workflows don't send one prompt and get one answer back. They re-read history, call tools, wait on results, and loop. Gartner's March 2026 analysis found agentic models require 5 to 30 times more tokens per task than a standard chatbot query.² Stanford's Digital Economy Lab went further: agentic coding tasks can consume up to 1,000x more tokens than simple code chat, and the same task run twice can vary in cost by up to 30x depending on how the agent loops.³⁴

Uber is the case study every engineering leader already knows. Claude Code adoption inside its 5,000-engineer org jumped from 32% to 84% between December 2025 and March 2026. By April, the entire annual AI budget was gone, with per-engineer monthly API costs running $500 to $2,000.² Nearly 70% of U.S. companies report at least partial AI budget overruns in the past year.¹ EY notes that most organizations budget for tokens, subscriptions, and infrastructure, while governance, failure recovery, and organizational change costs stay invisible until they hit the bill.⁵ Gartner projects more than 40% of agentic AI projects will be canceled by the end of 2027 over exactly this kind of cost surprise.⁵

Where do agent tokens actually go?

The single biggest driver isn't output length. It's re-sent input. Every turn an agent takes, it re-processes the system prompt, the tool schemas, and the full conversation history up to that point. Stanford's attribution research found this re-sent context accounts for roughly 62% of total agent inference bills.² The cost sits in input tokens, not output, because agents must re-read the accumulating context before every subsequent action.³

Retries make it worse. A ten-step agent with 95% reliability per step has only about a 60% chance of finishing a run cleanly, and it burns roughly 40% more tokens on retries than a perfectly reliable system, because a failed step in a context-accumulating workflow re-pays for everything sent before the failure, not just the retried step.¹ If you haven't mapped where your context goes turn by turn, start there. It's worth understanding what agentic context management actually means before you touch a line of routing logic.

Step 1: Measure before you optimize

Don't guess. Instrument first. EY's agentic FinOps framework and Cockroach Labs both point to the same starting metrics.²⁵

• Tokens per completed task, P50 and P95. Averages hide the outliers actually draining your budget.
• Cache hit rate. A low hit rate means you're paying full price for context you already sent.
• Cost per completed task, not cost per API call. A cheap call that fails and retries three times isn't cheap.
• Retry rate per workflow step. This tells you exactly where the 40% retry tax is landing.¹

Until you have these numbers, every architecture change is a guess. With them, you know which of the next four steps will actually move your bill.

Step 2: Turn on prompt caching correctly

Prompt caching is the highest-leverage change most teams haven't fully implemented. Anthropic prices cache reads at roughly 10% of the base input token rate, a 90% discount on anything you can keep in a stable prefix.²⁶ Break-even lands around 2.3 reuses of the same cached prefix within the cache TTL window.

The catch is that caching requires exact prefix matches. Put your system prompt, tool definitions, and any static instructions first, in a fixed order, before anything that changes turn to turn. Keep dynamic content, like the current user query, at the end. One team learned this the expensive way: a timestamp baked into their system prompt dropped their discount from 90% to 1%, because a timestamp or session ID anywhere in the cached prefix invalidates the match on every request.²⁶ If your bill has a caching line item that looks suspiciously flat month over month, this is almost always why. It's worth understanding the mechanics of cache pricing before you assume caching is already saving you money.

Step 3: Replace direct tool-calling with code execution

Most agent frameworks load every tool's full schema into context before the first call, then pass every intermediate result back through the model. That's expensive by design. Anthropic and Cloudflare both found the same fix: have the agent write code that calls tools through an MCP server, filtering and transforming results inside a sandbox before anything reaches the model's context.

In Anthropic's sample workflow, this dropped token usage from 150,000 to 2,000, a 98.7% reduction.⁷ The mechanism is progressive disclosure: the agent loads only the tool definitions it needs for the current step, instead of every tool the system knows about, and large intermediate outputs get filtered in the sandbox instead of round-tripping through the model. MindStudio's survey of production multi-agent workflows found that implementing even three or four techniques like this, including compact structured-data encoding, typically cuts total token usage by 50 to 70%.⁸

This is the step teams skip, because it means touching orchestration code instead of a pricing dashboard. It's also the step with the biggest single number attached to it.

Step 4: Route tasks to the cheapest model that can do the job

Not every step in an agentic workflow needs your most expensive model. Planning and judgment calls do. Data extraction, formatting, and simple classification usually don't.

One team cut its monthly API costs from $40,000 to $24,000, a 40% reduction, purely by routing simpler subtasks to cheaper models, with no product changes.² Industry estimates generally put model-routing savings in the 40 to 70% range. Pair this with layered tool access, where a small coordinator agent holds the high-level tools and delegates to specialized sub-agents that each carry a narrower tool schema, plus just-in-time retrieval that keeps working context under roughly 8,000 tokens.² Both techniques shrink the context each call has to carry, which compounds with routing savings instead of competing with them. If you haven't built a routing layer yet, the same mechanics used to reduce LLM API costs in development apply directly to production agents too.

Step 5: Own the orchestration layer instead of renting it

The platform you run agents on matters as much as the model you call. SaaS agent platforms and per-token billing tiers scale their price with your usage, so your most successful workflows become your most expensive ones. GitHub's June 2026 move from flat-rate Copilot premium requests to usage-based token billing is the clearest recent example: some users saw bills jump 10x or more, with one developer's estimated monthly cost rising from $39 to nearly $1,800 under the new calculator.⁹

Self-hosted, open orchestration avoids this entirely. One team running n8n on a $6-a-month VPS replaced Zapier plans priced at $19.99 to $49 a month, with overage charges kicking in around 2,000 tasks. Over 12 months they saved roughly $320 to $500 and got unlimited executions, at the cost of one to two hours a month of server maintenance.¹⁰ The workflow logic itself is an asset. Every task you run through someone else's billing meter is a task you're paying for twice: once in engineering time, once in invoice.

Step 6: Bring inference in-house where volume justifies it

Self-hosting an open-weight model through tools like Ollama or vLLM becomes cost-competitive with API pricing at roughly 2 million tokens a day of volume. Below that, idle infrastructure costs make APIs cheaper.¹¹ Above it, the math flips fast.

• A fintech company cut monthly AI spend from $47,000 on GPT-4o mini to $8,000 with a hybrid self-hosted approach, an 83% reduction.¹¹
• A telehealth client cut costs from $48,000 to $32,000 a month by moving chat triage to a self-hosted model.¹¹

The practical pattern is hybrid, not all-or-nothing: route high-volume, low-complexity traffic like chat triage or classification to a self-hosted model, and keep frontier APIs for the planning and reasoning steps that actually need them. Walk through the full break-even math, including where local hardware starts beating a metered bill outright, in the crossover math between cloud APIs and owned hardware.

Guardrails: cost caps, retry limits, and context rot

A bigger context window doesn't solve this. Chroma tested 18 frontier models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found every one degrades in accuracy as input length grows, even on simple tasks. A 200K-token context window can show significant degradation by 50,000 tokens.¹² Feeding an agent more context to fix a cost problem often makes both the cost and the reliability worse at once.

Set mechanical limits instead:

1. Retry caps per step. Retries alone add roughly 40% in token overhead. Cap retries per workflow step and fail loudly instead of looping silently.¹
2. Context compaction between turns. Summarize or discard completed sub-tasks instead of carrying full history forward indefinitely.
3. Just-in-time retrieval. Pull context only when a step needs it, keeping working context under roughly 8,000 tokens where possible.²
4. A hard dollar ceiling per run. Treat this the same way you treat a cloud spend alert, not an afterthought.

A token-cost checklist for builders

Starting from a bill you don't like? Work through this in order:

1. Instrument cost per completed task and cache hit rate before changing anything.
2. Restructure prompts so static content sits first, with no timestamps or session IDs in the cached prefix.
3. Move tool calls behind code execution so the model loads only the schemas and results it needs.
4. Add a routing layer that sends planning to a frontier model and execution to something cheaper.
5. Check whether your orchestration platform bills you per task, and whether owning that layer instead would pay for itself in months, not years.
6. Run the 2-million-token-a-day math on self-hosting for your highest-volume, lowest-complexity traffic.
7. Cap retries and context size as a permanent guardrail, not a one-time fix.

None of this requires giving up capability. It requires treating the agent's token consumption as an engineering budget with real levers, the same way you already treat compute and storage. The teams still surprised by their bill next quarter are the ones who skipped step one.