The Hidden Costs of AI Agents: Why You Are Overpaying for Proprietary APIs

The short answer

Running an AI agent on a proprietary API like GPT-4o or Claude Sonnet typically costs 5 to 50 times more per token than serving an equivalent open-weight model on your own GPU infrastructure, once you clear roughly 100 million tokens of monthly volume. Below that volume, the API almost always wins because you are not paying for idle hardware. The right question is not "agents or APIs," it's "at what volume does my fixed infrastructure cost stop mattering."

Where the markup actually lives

Frontier proprietary APIs price on convenience and model quality, not on the cost of the electrons. OpenAI's GPT-4o runs $2.50 per million input tokens and $10 per million output tokens. Anthropic's Claude 3.5 Sonnet runs $3 and $15. Claude Opus 4.6 sits even higher, at $5 in and $25 out per million tokens.

Compare that to the raw compute floor. A well-optimized 70B-class open-weight model served with vLLM on an H100 at high utilization lands around $0.45 to $0.58 per million tokens. Some benchmarks that account for INT8 quantization and KV-cache pruning push that as low as $0.47 per million tokens including hardware amortization. That is not a 20% gap. That is a 5x to 30x gap depending on the model tier and how efficiently you run it.

The pattern holds at industry scale. AI.cc's 2026 infrastructure report, drawn from 2.4 billion API calls across more than 8,000 enterprise accounts, found enterprise token costs fell 67% year over year, from $18.40 per million tokens in Q1 2025 to $6.07 in Q1 2026, almost entirely because open-source and open-weight models jumped from 11% to 38% of enterprise token volume in the same window. Enterprises that fully adopted a tiered model strategy, routing routine work to open models and reserving frontier models for hard reasoning, reported median blended costs of $2.31 per million tokens against $18.40 for frontier-only deployments. That is an 87% reduction.

Fixed cost versus marginal cost, the real divide

Managed APIs have zero fixed cost and a real marginal cost per token. Self-hosted inference flips that: high fixed cost, near-zero marginal cost. That is the entire economic argument in one sentence.

A concrete break-even example: self-hosting Llama 4 Maverick on two H100 GPUs costs about $3,986 a month in fixed infrastructure. Against GPT-4o's blended rate of $5.50 per million tokens, the break-even point lands at roughly 724 million tokens per month. Below that, the API is cheaper. Above it, self-hosting pulls ahead fast: at 2 billion tokens a month, self-hosting is 64% cheaper; at 5 billion tokens, it's 86% cheaper.

A separate industry rule of thumb from Digital Applied's cost analysis: under 8 million tokens a day, stick with the proprietary API. Between 8 and 30 million tokens a day, run a hybrid. Above 100 million tokens a day, self-hosting delivers 60% to 70% cost reduction.

Cost per million tokens, by approach

That last row matters more than most cost calculators admit. Self-hosting only beats the API if you keep the GPU busy. Research on production deployments puts the crossover at roughly 60% average GPU load. Below that, self-hosted inference actually costs more per token than a managed API tier, because you are paying for idle silicon.¹

Why agents make this worse, not better

Agents are not single API calls. They chain calls: planning, tool use, retries, verification. That multiplies token consumption 3 to 10 times over a simple chatbot interaction. Model API spend across the industry doubled from $3.5 billion to $8.4 billion between late 2024 and mid-2025, and enterprises now average $85,521 a month in AI operating costs. The scary part: a single agent conversation averaging $0.14 in token cost looks trivial until you multiply it by 3,000 employees firing the agent 10 times a day. That is $4,200 a day, or $1.5 million a year, from what looked like a rounding error.

The routing layer that exposes the markup

This is where open-source routing tools matter. Gateways like OpenRouter and self-hosted proxies like LiteLLM and Portkey do not reduce the price a model charges, they expose the spread between providers hosting the same model and let you route dynamically to the cheapest one that meets your quality bar.

OpenRouter itself charges no markup on provider token prices. Its only fee sits on credit purchases: 5.5% on card payments, 5% in crypto, or free if you bring your own provider key up to a monthly allowance. LiteLLM and Portkey are MIT-licensed and free to self-host entirely.

The real savings, though, come from routing logic, not gateway access. AT&T reportedly cut coding-AI costs by up to 56% by routing simple tasks to cheaper models and reserving premium models for requests that actually needed them. McKinsey's research on enterprise AI cost management backs this up structurally: reusing static prompt context can cut repeated input-token costs by up to 90%, and routing each task to the lowest-cost model that can still deliver the required quality is the single highest-leverage lever available to a CIO managing AI spend. Stacked together, caching plus routing plus batch discounts routinely deliver 70% to 85% cost reduction against an unoptimized baseline.

One useful distinction: a router that just gives you access to more models, without task-aware selection, does not lower your bill. If you were manually picking Claude Opus for every request before, you will still be paying Opus rates after adding a router. The savings come specifically from automated complexity classification, sending 70% of routine traffic to a cheap model and reserving the expensive one for the 10% of requests that actually need it.²

What this means for how you build

None of this is an argument that self-hosting is always right. For most teams below a few million tokens a day, the proprietary API is genuinely the cheaper, saner choice once you count engineering time. The point is that the sticker price on your AI vendor's dashboard is not a law of physics. It is a margin decision, and that margin compresses hard once your volume crosses the break-even line or once you introduce routing logic that treats model choice as a cost variable instead of a default.

This is the same logic behind Remy, which treats the tools a team runs as infrastructure you own and optimize, not a subscription you accept at face value. The teams paying the least per token are not the ones with the best negotiating leverage with OpenAI. They are the ones who measured their own usage pattern and built a system that only pays frontier prices when the task actually demands frontier intelligence.