How to Choose the Cheapest AI Model That Passes Your Evals
How companies pick the cheapest AI model that passes their own evals: realistic tasks, edge cases, full review costs, routers, and what survives upgrades.
For Procurement, Engineering and Executives

How do you choose the cheapest AI model that still does the job?
Write down the work you need done, build a set of real examples (including the hard ones), and set a standard for an acceptable result. Run your current model and cheaper ones against that set. Pick the cheapest model that passes. Count the human review and correction time in the cost, not just the price per token. Then rerun the same tests whenever a new model arrives. That set of tests is called an eval, and it replaces opinion with evidence.
TL;DR
- An eval is a test of your actual work, judged against your own standard for "good enough," not a public leaderboard score.
- Many tasks are already saturated: several models handle them well, so paying for a smarter one adds little or nothing.
- Test realistic assignments plus hard edge cases, because cheap models usually fail on the odd ones, not the typical ones.
- The real cost is token price plus review and repair time, so a cheap model that needs heavy correction can cost more.
- A router can send routine volume (support, coding) to a cheap model and the difficult cases to a frontier model.
- Your eval set is the asset that survives the next model release, because you can rerun it and get an answer in hours.
Why doesn't every task need the most expensive model?
Because of what's called task saturation. It means several models can already do a particular job well enough that buying more intelligence at a higher price adds very little. The models keep improving. The work you ask them to do often doesn't need the improvement.
A strategy document is a good example. You'd expect to want the best reasoning available. Yet people in one AI community have spent hundreds of dollars in API costs on frontier models, looked at the output, and wondered whether it was worth it. The model was impressive, but it wasn't improving the decision enough to justify the bill.
This shows up in company spending data. Ramp, which tracks business spending, described two months (August and September) of almost no growth in AI token spend even as token volumes kept rising. Its November 11th update said AI spending rose just 1.65% in October. Businesses are using more AI. They are not paying proportionally more for it.
The arithmetic is simple. A company uses a billion tokens at an average of $5 per million. That is a $5,000 bill. It moves routine work to cheaper models and the average falls to $1 per million. Now $5,000 buys 5 billion tokens. Usage grew fivefold and spending stayed flat.
What is an AI eval, exactly?
An eval is a test of the work you actually need done, judged against standards for an acceptable result. It isn't a general intelligence score. It's closer to a work sample you'd give a job candidate.
The useful thing about an eval is that it doesn't ask whether a model is smart. It asks whether the model meets your requirement. A model doesn't have to be equally capable in every situation. It only has to pass your tests. If a cheaper model passes the tests for the work you're buying, you have evidence it's a viable substitute, and you can say so in a budget meeting or a contract negotiation.
How do you build an eval that means something?
You don't need special software to start. A spreadsheet works. Follow these steps.
- Pick one task. "Draft replies to billing questions" is testable. "Help with customer service" is not.
- Collect 20 to 50 real examples. Use actual past tickets, documents, or code changes, with sensitive details removed as your policy requires.
- Include the hard cases on purpose. Add the angry customer, the ambiguous request, the messy file, the policy exception. Cheap models tend to look fine on easy inputs and fail on odd ones. If your test set is all easy cases, every model passes and you learn nothing.
- Write the pass standard before you look at outputs. Decide what "acceptable" means in plain terms, such as accurate, follows policy, right tone, no invented facts.
- Run the same inputs through each model. Start with the model you use now, then try cheaper options. You can stay with the same vendor and move down to a smaller model in their lineup.
- Score blind if you can. Have a reviewer grade outputs without knowing which model wrote them.
- Iterate toward the cheapest passer. If a model fails, the failure tells you why the expensive option may be worth paying for. Or it tells you that a bit of prompt tuning gets the cheaper model over the line.
Here is a simple template for each test case:
Task: [what the model is asked to do]
Input: [the real example]
Difficulty: [routine / edge case]
Pass standard: [what an acceptable result looks like]
Result by model: [pass / fail / pass with edits]
Review time: [minutes a person spent checking or fixing]
Notes: [what went wrong, if anything]
How do you count the full cost, not just the token price?
Token price is only one line. The number that matters is the total cost of getting an acceptable result.
| Cost item | What to measure |
|---|---|
| Model usage | Tokens used, at that model's price |
| Review | Minutes a person spends checking each output |
| Repair | Minutes spent fixing outputs that fail |
| Retries | Extra runs needed to get a usable answer |
| Failure impact | What an undetected error costs you |
This matters most in coding. The questions a useful eval answers: Does the model produce the right change? Does it work in the codebase? How much human review and repair will it need? If a cheaper model meets the standard at a lower total cost, there's every reason to use it. If it takes two or three tries to reach the result you need and that's still cheaper than the premium option, that's a legitimate win too.
It also settles arguments. A manager may worry a developer is overspending on a frontier model. A developer may feel forced onto a weaker one. You can argue about which model is smarter forever. An eval asks whether either one meets the requirement for the work in front of you.
When does a model router make sense?
A router is software that looks at each request and sends it to a particular model. Routers are now commonly available.
Customer support is the classic case. For a routine question, the customer wants a correct answer fast, and the business wants to resolve it reliably at an acceptable cost. A cheaper model can handle most of the volume well. The most capable model can be reserved for the really difficult cases. Ramp's data shows how fast this area is growing: customer success and support went from under 1% of classified spending in June to 13% in October, which suggests these use cases have become well understood enough for ordinary companies to adopt.
Coding works the same way. Routine edits go to the cheaper model. Tricky architecture problems go to the stronger one. Your eval tells you where the line sits, using your codebase rather than a guess.
Are expensive models ever worth it?
Yes. Ramp's November report said frontier models accounted for almost 70% of spending, with their share rising slightly in October. Some tasks demand more intelligence and justify the cost.
But a large share of the dollars isn't the same as a large share of the work. Many tasks are already well within what cheaper models can do. The point of evals is to find which of your tasks are which, rather than paying premium prices across the board "to be safe."
What should you invest in that survives the next model?
Models change every few months. Your eval set doesn't have to.
- The test set and pass standards. When a new model launches, rerun your tests and you'll know in a short time whether it's better, worse, or just more expensive. You stop waiting for someone to tell you where it belongs.
- Routing rules and workflows. The logic for which work goes where carries over when you swap models.
- The process around the output. How the result gets used, reviewed, and tied to a business outcome matters more than which model sits in the middle.
Buyers also gain bargaining power. When several models pass your eval, suppliers have to compete for work you know you can move. Even if you stay with one provider, evidence that a cheaper tier is enough can support a lower price in negotiation.
What does this mean for you?
If you manage a team or budget: Don't approve "the best model" by default. Ask for the eval. Ask what the cheapest passing option is and what review time it adds.
If you use AI daily: Keep a log of tasks where the cheap model was fine and where it fell short. That log is the start of an eval.
If you sell services: Expect your own customers to apply the same logic to you. One source described Infosys facing pressure from clients who want AI productivity gains passed through in pricing.
Start small. One task, 20 examples, two models, one afternoon.
Could you build your own model-testing tracker?
A spreadsheet works for a first eval, but it gets clumsy once several people score outputs and you want to rerun tests on every new model. This is the kind of small internal tool you can describe in plain English to Remy and have built as a working app: a place to store test cases, run each against several models, record pass or fail and review minutes, and show cost per acceptable result. Remy gives apps access to nearly 400 AI models without managing API keys, so comparing models doesn't require separate accounts. A typical initial build takes about an hour.
Frequently Asked Questions
How many test cases do I need?
Start with 20 to 50 real examples for one task. That's enough to see clear differences between models and to catch obvious failures. Add more edge cases over time whenever a model gets something wrong in real use.
Do I have to switch vendors to save money?
No. You can often keep the same vendor and move down to a less expensive model in their lineup. Vendors offer a range of models at different prices, and an eval tells you which tier your work needs.
What if the cheap model fails my eval?
The failure is useful. It shows where the expensive model earns its price, or it shows you what to fix. Sometimes two or three iterations with a cheaper model reach the result you need at a lower total cost than the premium option.
Why not just trust public benchmarks?
Benchmarks measure general performance on someone else's tasks. Your eval measures your work against your standard. A model can rank higher on a leaderboard and still add nothing to your particular job.
How often should I rerun my evals?
Rerun them whenever a new model or pricing tier appears that you might use. Because the test set already exists, this takes little effort, and it tells you whether to switch, stay, or negotiate.
How we source, write, and correct our articles. Editorial standards
Related articles

Will AI Token Usage Grow 1,000x? Agents and Budgeting

Do You Need the Most Expensive AI Model? Task Saturation

Why Is Nvidia So Dominant in AI Chips? The CUDA Moat Explained
