Why LLM Evaluation Costs More Than Inference (And Nobody Budgets For It)

LLM evaluation costs swing 4x to 33x for the exact same task, depending on which scaffold you run, which model you use as judge, and how many times you rerun the test for statistical confidence.¹ That variance, not the price of the model itself, is what's quietly inflating AI budgets right now.

Inference is collapsing. Evals are not.

Inference prices have fallen fast, but unevenly. Epoch AI found the cost to hit a fixed performance bar dropped anywhere from 9x to 900x per year depending on the benchmark, with one PhD-level science task falling 40x annually.² Model providers compete on price weekly. Tokens are a commodity now.

Evaluation is the opposite story. It isn't a commodity, because it isn't one thing. Every eval run bundles a scaffold (the harness that orchestrates the agent), a judge model (usually a second LLM grading the first one's output), a token budget, and however many repeated trials you need to trust the number. Each variable moves independently, and each one multiplies the cost rather than adding to it. That's why two teams testing the same agent on the same benchmark land on bills an order of magnitude apart.

What drives the 4x-33x spread?

Exgentic ran a $22,000 sweep across different agent configurations on identical tasks and found a 33x cost spread driven almost entirely by scaffold choice, not the underlying model.¹ The harness wrapped around a model, how it retries, how it chains tool calls, how verbose its intermediate reasoning gets, matters more to your bill than which model you picked.

The Holistic Agent Leaderboard (HAL) spent about $40,000 running 21,730 agent rollouts across 9 models and 9 benchmarks.¹ Agent behavior is noisy: the same prompt against the same model can produce different tool calls, different token counts, different outcomes on different days. A statistically credible version of that same sweep, with 8 reruns per cell to average out the noise, would cost roughly $320,000.¹ Reliability isn't a nice-to-have here. It's an 8x multiplier bolted onto every number you were planning to trust.

Judge models add another layer. A single full PaperBench evaluation, judge included, runs about $9,500.¹ Scale that to the three-seed, six-model comparison you'd need for a defensible published result, and you're past $150,000.¹ None of that is model inference in the traditional sense. It's the cost of checking the model's work, and it belongs in the same conversation as the markups baked into proprietary agent APIs.

Does higher eval spend buy better accuracy?

No, and that should worry budget owners.

On Online Mind2Web, one agent configuration cost $1,577 to hit 40% accuracy. A different configuration hit 42%, a full two points better, for $171.¹ That's a 9x cost gap for a rounding-error accuracy difference. On GAIA, one setup cost $2,828 for 28.5% accuracy, while another reached 57.6%, roughly double the score, for $1,686, less money.¹ CLEAR's analysis of six state-of-the-art agents across 300 enterprise tasks found the accuracy-optimal configuration for a given task costs 4.4x to 10.8x more than a Pareto-efficient alternative delivering comparable real-world performance.¹

In plain terms: teams routinely pay for expensive evaluation setups that don't outperform cheap ones. Spend and signal aren't correlated the way anyone assumes.

Why DIY agent teams get blindsided

Teams buying an eval product see a price tag up front. Teams building their own agent evaluation in-house, which is most teams building agents at all, tend to discover the real cost after the fact. Three things catch them off guard:

1. Non-determinism forces reruns. Agents don't produce the same output twice. A single test run tells you almost nothing; a credible answer requires repeated trials, and each rerun is a full-price repeat of the original spend.¹
2. Judge costs exceed agent costs. Using a second LLM to grade the first one's output is standard practice, but that judge call often costs more than the agent inference it's grading. Observability vendor Monte Carlo left an LLM-powered eval running for days and came away with a five-figure bill.³
3. The budget line doesn't exist yet. Nearly 80% of enterprises have already deployed AI agents, but most don't understand what training and evaluating them actually costs.³ Evaluation for a small, well-scoped agent can run into the thousands of dollars; for a complex agent, tens of thousands.³ Neither number shows up in most AI budgets until the invoice does.

A basic cost model makes the trap obvious. Evaluation cost compounds multiplicatively: dataset size times test scenarios times iterations times cost per call. A modest 10,000-example suite across 10 scenarios costs $100 at $0.001 per call. Add five prompt iterations across three models, standard practice during development, and the same suite costs $1,500.⁴ Nobody budgeted for the 15x.

Why doesn't eval spend show up as its own line item?

It's scattered across LLM gateway invoices, GPU compute for self-hosted judges, and whatever observability tool the team adopted. That fragmentation is structural, not accidental.

LLM pricing itself has hidden dimensions that make cost tracking hard even before you get to evaluation. Portkey, which tracks $180 million in LLM spend across more than 3,500 models, found that systems counting only visible output tokens undercount agentic workloads by 30% to 40%, because thinking tokens get billed at output rates without ever appearing in the response.⁵ The same task can cost $0.04 per million tokens on one provider and $25 on another, a 625x spread for comparable work.⁶

That opacity is why generic cloud FinOps tools miss AI evaluation spend entirely. AI cost visibility tooling has split into four separate categories, gateway proxies, trace-level observability, billing-based visibility, and full-stack FinOps, because no single layer captures model API cost, GPU infrastructure, and agent-workflow token spend together.⁷ Evaluation spend falls into the gaps between them.

And the sticker price for evaluation infrastructure itself isn't small. Private evaluation platforms run from roughly $249 a month at the entry tier to $10,000-plus monthly for enterprise.⁸ Self-hosting your own evaluation stack with "free" open-source tools costs $125,000 to $190,000 a year at minimal scale, and $500,000 to $820,000 a year at moderate scale once you count GPU compute, engineering time, and dataset curation.⁸ Enterprises consistently underestimate this by 40% to 60%.⁸ It's the same dynamic covered in the true cost of renting tokens versus owning quantized models: the sticker price on the API is never the whole story, and the missing piece is usually infrastructure you didn't know you were buying.

How do you control eval cost variance?

Teams that get this under control tend to do the same handful of things:

• Right-size the judge model. Use a smaller, cheaper model for grading whenever the task doesn't demand a frontier judge. Reserve the expensive judge for ambiguous or high-stakes cases.
• Tier the eval set by difficulty. Run the full, expensive suite only on the subset of tasks that actually discriminate between good and bad agent behavior. Most benchmarks have a long tail of easy cases that don't need repeated, judge-heavy scoring.
• Target the Pareto frontier, not the accuracy ceiling. CLEAR's own data shows configurations near the cost-efficient frontier match the accuracy of far pricier setups.¹ Chasing the last percentage point of accuracy is usually where the multiplier lives.
• Compress before you scale reruns. Cut redundant prompt tokens and dataset size before adding reruns for statistical confidence. Reruns are the multiplier; don't multiply a bloated baseline.
• Track judge cost separately from agent cost. If the judge bill is bigger than the agent bill, that's a signal to swap models, not to run the eval less often.

Build vs buy, applied to evals

This is, at bottom, the same build-vs-buy question Remy's readers already ask about the rest of the stack, and it comes with the same lock-in tradeoffs as any other vendor decision. A managed eval platform like Braintrust, Langfuse, or LangSmith gives you a fixed, visible monthly number, at the cost of paying someone else's margin on top of your usage. A self-hosted eval harness, following the same playbook as self-hosting the rest of your AI stack, gives you full cost control and ownership of the data, but only if someone on your team is actually tracking GPU hours, judge-model spend, and rerun counts against a real budget, not just running it until Finance asks questions.

The teams getting burned aren't the ones who chose wrong between build and buy. They're the ones who never treated evaluation as infrastructure at all, and so never gave it a budget owner, a cost model, or a dashboard. Whichever side of that line you land on, the fix is the same: know your per-run cost, your judge-to-agent cost ratio, and your rerun multiplier before you scale, not after the invoice shows up. Tools like Remy exist for exactly that gap: giving teams visibility into what their AI infrastructure, including the eval layer, is actually costing them, instead of discovering it as a line item nobody planned for.