Blog
Live Nerf benchmarkMargin Lab Claude CodeAI model tracker

Live Nerf vs Margin Lab: Do AI Models Really Get Worse?

Live Nerf saw no drop in 50 days of daily tests. Margin Lab caught a 65% to 50% fall tied to a Claude Code update. Here is what each tracker shows.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Engineering, Managers and Executives

A flat geometric laptop whose screen is divided into a dark half and a lighter half, with one small bright dot on the lighter side.

What do AI nerf trackers like Live Nerf and Margin Lab actually prove?

They prove that the tool you use can get worse without the underlying model changing. Live Nerf ran the same 78 questions through a model every day and, after its first 50 days, detected no drop. Margin Lab caught a real fall from 65% to about 50%, but traced it to an update to the Claude Code app, not the model. Neither tracker has shown that a company swapped in a weaker model after launch.

TL;DR

  • Live Nerf found no model drop. It reran the same 78 questions daily, and after 50 days it saw no decline.
  • Margin Lab found a real drop. Scores fell from 65% to about 50%, and the cause was an update to the Claude Code app.
  • Both results can be true at once. A model can stay the same while the software around it (the "harness") makes it perform worse.
  • Nobody has proved secret model downgrades. Quantization (shrinking a model to run cheaper) is a popular theory, but no one has admitted it or leaked it, and the trackers haven't caught it.
  • Companies have admitted other kinds of degradation. Anthropic published two postmortems saying Claude got worse for weeks at a time.
  • Even the trackers say they miss things. Live Nerf itself says small changes can slip past it, so we know less than it feels like we do.
  • You can check your own tools. A small set of fixed prompts, rerun on a schedule, tells you more than a feeling does.

Why does everyone think AI models get worse after launch?

The pattern is familiar. A new model launches, demos go viral, and a week or two later people ask whether it got dumber. One YouTube creator who looked into this described rerunning the same prompt on the same model two weeks after launch and getting a noticeably worse picture.

Part of this is human. If a model fumbles one answer in 20, you will eventually see a few failures, and those are the ones you remember. Launch week is also when people test with flashy one-off demos. A few weeks later, you throw real work at it, with long context and hard reasoning, and the cracks show.

But the complaints are not only perception. Anthropic has published two postmortems (public write-ups of what went wrong) admitting Claude got worse for weeks at a time. In one, they wrote that the evaluations they ran "simply didn't capture the degradation that users were reporting." Their own tests said things were fine while users were right. OpenAI has had similar problems. After GPT-5 launched, Sam Altman said it seemed dumber because something broke behind the scenes.

What is Live Nerf and what did it find?

Live Nerf is an independent benchmark that runs the exact same 78 questions through a model every day. The idea is simple. If the model's weights (the billions of numbers that make up its "brain") were quietly changed or rounded off, the scores on a fixed test should move.

After its first 50 days, it detected no drop. That is useful evidence against the theory that a model is secretly swapped for a smaller one a few weeks after launch.

It is not proof of the opposite, though. Live Nerf says small changes can slip past it. A 78-question test has limited resolution, and it measures the model, not necessarily the full experience of using the model inside a coding app.

What is Margin Lab and what did it find?

Margin Lab caught a real drop. Its measured performance fell from 65% to about 50%. The cause was not the model. It was an update to the Claude Code app, the software that wraps the model and feeds it instructions and tools.

This is the more practical finding for most people. If your results got worse, the model may be identical while the product around it changed. The trackers were built by independent people. The companies themselves did not publish daily scores like these.

Why can one tracker see nothing while another sees a 15-point drop?

They are looking at different layers. Think of a chef (the model) and a restaurant (the app). You can test the chef with the same recipe every day and get the same dish. The restaurant can still get worse if it changes the menu, rushes the kitchen, or sends your order to a different cook.

The video's creator described four "dials" around the model that can change without touching its weights:

  1. Effort. This controls how hard the model thinks before answering. Anthropic once lowered the default effort in Claude Code from high to medium to make it faster, did not announce it, and later admitted it was the wrong tradeoff.
  2. Instructions. Apps wrap the model in a hidden system prompt. At one point Anthropic added a line telling Claude to keep its text between steps to 25 words or less, and its own testing showed that made Claude worse. OpenAI had a similar issue where old skills fired too often and got in the model's way.
  3. Routing. This decides which model actually answers you. When one model came back online, a benchmark showed its debugging score falling from 86 to 26. But nine of the 12 tasks never reached that model because a safety filter sent them to an Opus model instead. In the creator's own test, 14 of 63 questions were silently routed that way.
  4. The app itself. Anthropic once shipped a bug that wiped Claude's thinking on every turn, so it kept forgetting what it was doing. OpenAI found misconfigured servers that hurt part of its traffic.

Margin Lab's finding fits dial four. Live Nerf's flat line fits the idea that the model itself held steady.

Does the launch-day score match what you actually get?

Often not. The creator pointed to Anthropic's own launch page for Opus 5.5: a coding score of 57.8% in the headline benchmark, versus about 52% for the version people typically use. The gap comes from effort. The benchmark used max effort, while the everyday default is medium.

You can turn effort up yourself, but it costs more time and money. When you see a demo or a benchmark, ask what effort setting it used. A result at max effort tells you what the model can do, not what your default settings will do.

Do the companies have to tell you when something changes?

As far as the creator could find, no. Connecticut passed what he described as the closest thing, a law requiring companies to disclose usage limits before you sign up or renew. It says nothing about whether a model got worse.

There is also a business incentive worth noting, though it is opinion, not proof. Launches bring a surge in demand (Altman said GPT-5 API traffic roughly doubled in 24 hours), and compute is expensive. The creator argued that incentives point toward giving users a bit less over time. That is a hypothesis. No tracker has demonstrated it.

What should you take from this?

Treat the "nerf" as real but usually not about the model. Here is what that means in practice.

If you use AI for work every day:

  • Keep three to five fixed test prompts from your real tasks. Save the outputs from a good week.
  • Rerun them when your tool feels off. Compare against your saved outputs, not your memory.
  • Check the effort or reasoning setting before blaming the model. Turn it up for hard tasks.
  • If you use a coding tool, note when it auto-updates. A drop right after an update points to the app, not the model.
  • Try the same prompt in a different app that uses the same model. If the result differs, the harness is the likely cause.

If you manage a team or budget for AI tools:

  • Do not rely on vendor benchmark numbers alone. Ask what settings produced them.
  • Log complaints with dates. Patterns that line up with app updates are useful evidence for a vendor.
  • Remember the postmortem lesson: vendor tests missed what users noticed. Your own checks matter.

A simple template for your test log:

Prompt name:
Date:
Tool and version:
Model and effort setting:
Score (1-5) on: accuracy / completeness / format
Notes on what changed:

Can you build your own AI quality tracker?

You can, and it does not need to be a spreadsheet you forget to update. A small internal app could store your fixed prompts, run them on a schedule, save each output, and chart your scores by week and by tool version. Remy builds full-stack apps from a plain-English description, with a database and scheduled jobs included, and it gives apps access to nearly 400 AI models without API key management. So you could describe a tracker like this and have a live version your team can use, rather than paying for another subscription or relying on memory.

Frequently Asked Questions

5 questions

Has anyone proved AI companies secretly make models worse after launch?

No. The video's creator searched and found no proof that the model on launch day differs from the one weeks later. Quantization is a popular theory, but nobody has admitted it, leaked it, or caught it with a tracker. What has been admitted are changes to effort, instructions, routing and app bugs.

What is a harness, and why does it matter?

A harness is the software wrapped around a model: the app, its hidden instructions, its tools, and its settings. Claude Code is an example. It matters because most of what you experience comes from the harness plus the model, so a harness change can make results worse even when the model is untouched.

Does Live Nerf's result mean the model never gets worse?

No. It means that over its first 50 days of daily testing on 78 questions, it detected no drop. Live Nerf itself says small changes can slip past it, and it does not cover every way a tool can degrade.

What caused the drop Margin Lab found?

Margin Lab saw performance fall from 65% to about 50%, and the cause was an update to the Claude Code app, not the model.

How can I tell if my AI tool got worse?

Save a few fixed prompts and good outputs, then rerun them when quality seems to slip. Check the effort setting, note any recent app updates, and compare against another app using the same model.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.