Ignore the Benchmarkpocalypse: How to Choose an AI Model for Internal Tools

The short answer

Pick an AI model by testing it on your own real tasks with your own team, not by reading leaderboard scores. Benchmarks measure narrow, often contaminated academic exercises that have little connection to how a model performs on your invoices, your support tickets, or your internal wiki. The model that wins on MMLU or HumanEval is frequently not the model your team will actually use, trust, or stick with.

This matters more now than it did two years ago, because so many companies are letting individual employees pick their own tools. The choice used to be made once, by IT. Now it is made dozens of times, by whoever is building the internal app, script, or workflow. Getting the decision framework right is a governance question as much as a technical one.

Why benchmarks are a bad map for this decision

The case against today's benchmarks is not a fringe complaint. It is close to a consensus among the researchers who study evaluation itself.

Maarten Sap, an assistant professor at Carnegie Mellon and a co-creator of one widely used benchmark, put it bluntly: "The yardsticks are, like, pretty fundamentally broken."¹ Emily Bender, a linguistics professor at the University of Washington, has gone further, arguing that most popular benchmarks lack basic construct validity, meaning nobody has actually shown that a high score means what people assume it means.¹ Even Rowan Zellers, the lead author of HellaSwag, one of the most cited benchmarks in the field, has admitted the whole exercise was somewhat improvised: "It's sort of like we kind of just made these benchmarks up."¹

Three structural problems explain why this happens.

The tests are built from junk data. Many popular benchmarks were assembled by scraping WikiHow articles, Reddit threads, and gig-worker submissions rather than working with domain experts.¹ A 2025 review of roughly 100 academic papers on benchmark quality found that more than 70 percent of benchmark datasets used in prominent computer vision papers had simply been reused from other, unrelated tasks.² That is not a foundation you want under a business decision.

The scores get gamed. Benchmarks are also commercial marketing tools. The same 2025 review notes that private companies' share of the largest AI models rose from 11 percent in 2010 to 96 percent in 2021, giving labs both the means and the incentive to optimize for specific leaderboard numbers rather than general usefulness.² One widely cited example: a lab reportedly spent hundreds of thousands of dollars in compute specifically to post a high score on the ARC-AGI benchmark.²

They measure the wrong thing entirely. Arvind Narayanan, a Princeton computer science professor, has written plainly that "these benchmarks might not be good indicators of how a system will behave in the real world."¹ A model that passes a bar exam question set is not the same as a model that is useful for your legal team's contract review workflow, any more than a chef who aces a written food-safety exam is guaranteed to run a good kitchen.

This is not a new industry doing calisthenics before it matures. Foundation model releases nearly doubled in a single year, reaching 149 major models in 2023, according to Stanford's AI Index.¹ The pace of releases is outrunning the pace at which evaluation methods can be fixed, so the gap between what benchmarks measure and what businesses need is widening, not closing.

What actually predicts whether a model will work for you

If leaderboard rank is not the signal, what is? The practitioners who evaluate models for a living converge on a small set of factors that matter far more for internal tools than any academic score.

1. Performance on your actual task, not a proxy task

A recent AWS engineering guide frames the alternative to "vibes-based" model selection as structured testing against your own prompts and your own expected outputs, scored on dimensions like correctness, completeness, relevance, and instruction-following.³ In one worked example from that guide, a company building a database-modeling tool ran four candidate models against its own text-to-SQL prompts. There was no universal winner. The most accurate model was also the slowest and most expensive, the cheapest model scored lowest on correctness, and the company ended up assigning different models to different customer tiers based on which tradeoff mattered for each segment.³ That is the real shape of model selection: multiple right answers depending on the job, not one best model.

2. Cost and latency for your workload, at your volume

A model that is three times more expensive per call adds up fast once it is embedded in a workflow that runs thousands of times a day. Benchmarks almost never account for this, because academic evaluation was never designed to answer whether you can afford to run a model on every support ticket.

3. Team comfort and adoption

A model your team trusts and understands gets used correctly. A model that produces technically superior but unfamiliar output gets second-guessed, rechecked by hand, or quietly ignored. This is not a soft consideration. It is the difference between a tool that saves time and one that becomes shelfware. If your team is already comfortable inside a particular ecosystem or interface, that familiarity often outweighs a few points of benchmark difference.

4. Failure mode, not just success rate

A model that fails obviously, with a garbled answer or a clear refusal, is safer in production than one that fails confidently with a plausible-sounding wrong answer. Researchers reviewing benchmark literature have pointed out that most tests report a single success percentage and tell you almost nothing about how or why a model fails, even though that is often the more operationally important question.² For internal tools touching real business decisions, ask about failure behavior specifically, not just pass rate.

A simple process that works for internal tools

You do not need a research lab to do this well. A workable process looks like this:

1. Write down 10 to 20 real examples of the task the tool needs to do, pulled from actual work, not hypotheticals.
2. Run two to three candidate models against those examples, with the same prompt structure for each.
3. Score them on the outcomes that matter to you: correctness against a known right answer, time to get that answer, and cost per run at your expected volume.
4. Let the people who will use the tool weigh in. If your support team dislikes the tone of one model's output, that is a real data point, not noise.
5. Re-test quarterly. Models change fast enough that a six-month-old comparison is close to worthless.

This is a smaller version of what teams building production AI systems already do formally. The AWS framework described above essentially automates these same steps at scale, using an LLM as a judge to score outputs so the process can run on hundreds of prompts instead of ten.³ For most internal tools, you do not need that much rigor. You need the same instinct: test against your own work, not someone else's leaderboard.

Why this matters for shadow AI and ownership

This connects to a bigger shift happening inside companies right now. Employees are no longer waiting for IT to bless a model. They are picking whichever one plugs into the tool they are building themselves, often without anyone else in the company knowing which model, or which version, is doing the work. When the criteria for that choice is whichever one topped a benchmark someone saw online, you end up with a patchwork of tools chosen for the wrong reasons and no consistent way to compare or replace them later.

A better standard, evaluate for the task, the cost, and the team, travels well across that shadow AI landscape. It gives non-technical builders a way to make a defensible choice, and it gives the company a shared vocabulary for auditing those choices later. Platforms like Remy that help teams turn ad hoc internal tools into owned, maintained software make this discipline easier to enforce, because the model choice becomes a documented decision attached to a real piece of software rather than a one-off preference buried in someone's personal workflow.

FAQ

Do benchmarks matter at all? They are useful for narrowing a very long list of options down to a shortlist, especially for basic capability checks like language support or context window size. They are a poor tool for the final decision because they do not reflect your data, your workflow, or your cost constraints.

How many models should I actually test? Two or three is usually enough. Testing more than that adds evaluation overhead without meaningfully improving your decision, since the marginal difference between the fourth and fifth candidate is rarely worth the time spent.

Should I switch models once I've made a choice? Only if a new test on your own tasks shows a real improvement in accuracy, cost, or speed for your specific workload. Model churn has a cost: prompts, integrations, and team habits all have to be rebuilt. Re-test quarterly rather than chasing every new release.

Is a bigger, more expensive model always safer for business use? No. The AWS case study on database-schema generation found that the most expensive model was also the slowest, and it was worth it only for the premium tier where accuracy mattered most. For lower-stakes internal tasks, a faster, cheaper model that scores reasonably well is often the better business decision.³

What is construct validity, and why should a non-technical person care? It is the idea that a test should actually measure the thing it claims to measure. Multiple benchmark researchers have found this basic property missing from popular AI benchmarks, meaning a high score does not reliably indicate the capability being marketed.¹ That is the core reason to distrust benchmark rankings as a purchasing signal.