Ignore the Benchmarkpocalypse: How to Choose an AI Model for Internal Tools

How should you choose an AI model for internal tools?

Pick an AI model by testing it on your own real tasks with your own team, not by reading leaderboard scores. Benchmarks measure narrow, often contaminated academic exercises that have little connection to how a model performs on your invoices, your support tickets, or your internal wiki. The model that wins on MMLU or HumanEval is frequently not the model your team will actually use, trust, or stick with.

This matters more now than it did two years ago, because so many companies are letting individual employees pick their own tools. The choice used to be made once, by IT. Now it is made dozens of times, by whoever is building the internal app, script, or workflow. Getting the decision framework right is a governance question as much as a technical one.

Why are benchmarks a bad map for this decision?

The case against today's benchmarks is not a fringe complaint. It is close to a consensus among the researchers who study evaluation itself.

Maarten Sap, an assistant professor at Carnegie Mellon and a co-creator of one widely used benchmark, put it bluntly: "The yardsticks are, like, pretty fundamentally broken." Emily Bender, a linguistics professor at the University of Washington, has gone further, arguing that most popular benchmarks lack basic construct validity, meaning nobody has actually shown that a high score means what people assume it means. Even Rowan Zellers, the lead author of HellaSwag, one of the most cited benchmarks in the field, has admitted the whole exercise was somewhat improvised: "It's sort of like we kind of just made these benchmarks up."

Three structural problems explain why this happens.

The tests are built from junk data. Many popular benchmarks were assembled by scraping WikiHow articles, Reddit threads, and gig-worker submissions rather than working with domain experts. A 2025 review of roughly 100 academic papers on benchmark quality found that more than 70 percent of benchmark datasets used in prominent computer vision papers had simply been reused from other, unrelated tasks. That is not a foundation you want under a business decision.

The scores get gamed. Benchmarks are also commercial marketing tools. The same 2025 review notes that private companies' share of the largest AI models rose from 11 percent in 2010 to 96 percent in 2021, giving labs both the means and the incentive to optimize for specific leaderboard numbers rather than general usefulness. One widely cited example: a lab reportedly spent hundreds of thousands of dollars in compute specifically to post a high score on the ARC-AGI benchmark.

They measure the wrong thing entirely. Arvind Narayanan, a Princeton computer science professor, has written plainly that "these benchmarks might not be good indicators of how a system will behave in the real world." A model that passes a bar exam question set is not the same as a model that is useful for your legal team's contract review workflow, any more than a chef who aces a written food-safety exam is guaranteed to run a good kitchen.

This is not a new industry doing calisthenics before it matures. Foundation model releases nearly doubled in a single year, reaching 149 major models in 2023, according to Stanford's AI Index. The pace of releases is outrunning the pace at which evaluation methods can be fixed, so the gap between what benchmarks measure and what businesses need is widening, not closing.

What actually predicts whether a model will work for you?

If leaderboard rank is not the signal, what is? The practitioners who evaluate models for a living converge on a small set of factors that matter far more for internal tools than any academic score.

How does the model perform on your actual task, not a proxy task?

A recent AWS engineering guide frames the alternative to "vibes-based" model selection as structured testing against your own prompts and your own expected outputs, scored on dimensions like correctness, completeness, relevance, and instruction-following. In one worked example from that guide, a company building a database-modeling tool ran four candidate models against its own text-to-SQL prompts. There was no universal winner. The most accurate model was also the slowest and most expensive, the cheapest model scored lowest on correctness, and the company ended up assigning different models to different customer tiers based on which tradeoff mattered for each segment. That is the real shape of model selection: multiple right answers depending on the job, not one best model.

What do cost and latency look like for your workload, at your volume?

A model that is three times more expensive per call adds up fast once it is embedded in a workflow that runs thousands of times a day. Benchmarks almost never account for this, because academic evaluation was never designed to answer whether you can afford to run a model on every support ticket.

Will your team be comfortable with it and adopt it?

A model your team trusts and understands gets used correctly. A model that produces technically superior but unfamiliar output gets second-guessed, rechecked by hand, or quietly ignored. This is not a soft consideration. It is the difference between a tool that saves time and one that becomes shelfware. If your team is already comfortable inside a particular ecosystem or interface, that familiarity often outweighs a few points of benchmark difference.

How does it fail, not just how often does it succeed?

A model that fails obviously, with a garbled answer or a clear refusal, is safer in production than one that fails confidently with a plausible-sounding wrong answer. Researchers reviewing benchmark literature have pointed out that most tests report a single success percentage and tell you almost nothing about how or why a model fails, even though that is often the more operationally important question. For internal tools touching real business decisions, ask about failure behavior specifically, not just pass rate.

What is a simple process that works for internal tools?

You do not need a research lab to do this well. A workable process looks like this:

1. Write down 10 to 20 real examples of the task the tool needs to do, pulled from actual work, not hypotheticals.
2. Run two to three candidate models against those examples, with the same prompt structure for each.
3. Score them on the outcomes that matter to you: correctness against a known right answer, time to get that answer, and cost per run at your expected volume.
4. Let the people who will use the tool weigh in. If your support team dislikes the tone of one model's output, that is a real data point, not noise.
5. Re-test quarterly. Models change fast enough that a six-month-old comparison is close to worthless.

This is a smaller version of what teams building production AI systems already do formally. The AWS framework described above essentially automates these same steps at scale, using an LLM as a judge to score outputs so the process can run on hundreds of prompts instead of ten. For most internal tools, you do not need that much rigor. You need the same instinct: test against your own work, not someone else's leaderboard.

Why does this matter for shadow AI and ownership?

This connects to a bigger shift happening inside companies right now. Employees are no longer waiting for IT to bless a model. They are picking whichever one plugs into the tool they are building themselves, often without anyone else in the company knowing which model, or which version, is doing the work. When the criteria for that choice is whichever one topped a benchmark someone saw online, you end up with a patchwork of tools chosen for the wrong reasons and no consistent way to compare or replace them later.

A better standard, evaluate for the task, the cost, and the team, travels well across that shadow AI landscape. It gives non-technical builders a way to make a defensible choice, and it gives the company a shared vocabulary for auditing those choices later. Platforms like Remy that help teams turn ad hoc internal tools into owned, maintained software make this discipline easier to enforce, because the model choice becomes a documented decision attached to a real piece of software rather than a one-off preference buried in someone's personal workflow.