Do AI Models Get Nerfed After Launch? What Labs Admitted
Anthropic and OpenAI have admitted quality dips after launch. Here are the four dials (effort, system prompt, routing, app bugs) and how to test for yourself.
For Engineering, IT and Executives

Do AI models get worse after launch?
Sometimes the experience does, but there is no public proof that the model itself changes. Anthropic and OpenAI have both admitted in writing that quality dipped for users after launches. The causes they described were settings and software around the model: how hard it thinks, the hidden instructions it receives, which model answers you, and bugs in the app. Trackers that run the same questions daily have not caught the model's core numbers (its "weights") changing.
TL;DR
- Anthropic published two postmortems admitting Claude got worse for weeks at a time, and said its own evaluations missed the problem users were reporting.
- Sam Altman said GPT-5 seemed "way dumber" the day after launch because something broke behind the scenes, according to the video source reviewed for this article.
- No one has shown the underlying model changing after launch. What changes is the setup around it.
- Four dials matter: effort level, the system prompt, routing between models, and bugs in the app you use.
- Launch-day benchmark scores often use max effort, while the default setting you get can score noticeably lower on the same page.
- You can test this yourself with five prompts from your real work, rerun every couple of weeks.
What did Anthropic and OpenAI actually admit?
Anthropic has published two postmortems, which are public write-ups of what went wrong. They acknowledged that Claude got worse for weeks at a time. In one, users had reported that Claude Code felt less intelligent. In the other, Anthropic wrote that the evaluations it ran "simply didn't capture the degradation that users were reporting." In other words, the company's own tests said things were fine while users saw otherwise.
OpenAI has had similar moments. The day after GPT-5 launched, Sam Altman posted that it seemed way dumber because something broke behind the scenes. When problems like this happen, a common remedy is a usage reset, which gives you extra allowance but doesn't explain much.
So the people complaining on social media were often right. What they usually got wrong was the cause.
What does "nerfed" mean if the weights don't change?
An AI model is a huge file of numbers, billions of them, called weights. Training means tuning those numbers until good answers come out. Once training is done, the weights are the model's brain.
A common theory is that labs quietly swap in a cheaper version of the model after launch. The usual mechanism proposed is quantization, which rounds those numbers off so the model is smaller and cheaper to run. Think of streaming a movie in 720p instead of 4K: same movie, less detail.
Nobody has admitted to doing this, and nobody has leaked evidence of it. It could still happen, but it hasn't been demonstrated. Meanwhile, the evidence for changes around the model is solid, and it comes in four forms.
What are the four dials that change how a model behaves?
Dial 1: Effort
Effort is how hard the model is allowed to think before answering. Anthropic once lowered the default effort in Claude Code from high to medium to make it faster, and didn't announce it. It later admitted that was the wrong tradeoff.
This shows up in launch pages. The Opus 5.5 launch page, as described by the creator whose videos informed this article, lists a coding score of 57.8% at max effort and about 52% at the default effort, on the same page. If you only see the headline number, you may be comparing your default experience to a max-effort result.
In Claude Code you can set effort yourself by typing /effort or using the button. There's a tradeoff: more effort costs more time and money, and too little gives weaker results.
Dial 2: The system prompt
Most AI apps wrap the model in a hidden set of instructions called the system prompt. On top of that, you may add your own files, such as a Claude.md. The lab controls the hidden layer, and changes there can make the model behave differently.
Anthropic once added a line telling Claude to keep its text between steps to 25 words or less. Its own testing then showed that made Claude worse. OpenAI had a comparable issue with one of its products, where old "skills" fired far too often and got in the model's way.
Dial 3: Routing
Routing decides which model actually answers you. Altman's GPT-5 explanation was a routing-type failure: something behind the scenes broke, and answers came from somewhere they shouldn't have.
A second example from the source involves a newer model called Fable 5. One benchmark showed its debugging score falling from 86 to 26, and people called it nerfed. But nine of the 12 tasks never reached it. A safety filter had routed those requests to Opus 4.8, and at first users weren't told. The creator ran his own test of 63 questions: Fable refused 14 and silently passed them to Opus 4.8. You may think you're reading one model's output when it came from another.
Dial 4: The app itself
The software around the model has bugs too. Anthropic once shipped a bug that wiped Claude's thinking on every turn, so it kept forgetting what it was doing. OpenAI found misconfigured servers that made one of its products worse for part of its traffic. None of this touches the weights, but you feel all of it.
What do the independent trackers show?
Volunteers, not the labs, built these trackers. Two examples from the source:
| Tracker | What it does | What it found |
|---|---|---|
| Live Nerf | Runs the same 78 questions through a model daily | No drop after its first 50 days (it notes small changes can slip past) |
| Margin Lab | Tracks performance over time | A real drop from 65% to about 50%, traced to an update to the Claude Code app, not the model |
The pattern fits the four dials. The model held steady, and the product around it moved. The labs themselves don't publish daily scores like these, so we know less than it seems.
Why might labs have a reason to turn the dials down?
This part is opinion, not proof. Compute is expensive: every token costs real money and real GPUs. After a launch, demand jumps. When GPT-5 launched, Altman said API traffic roughly doubled in 24 hours. A lab under that pressure has an incentive to give each user a bit less, such as lower effort or a cheaper route.
We have seen at least one case of that: the effort cut in Claude Code, made so it would run faster. That doesn't prove a pattern of launching strong and quietly dialing back. It does show the incentive is real and the changes can happen without notice.
Is some of it just in our heads?
Partly, probably. Two things skew perception:
- Launch week is a stress test. Everyone pushes the model to produce impressive demos. Weeks later, you bring real work with long context and hard reasoning, which is harder.
- Hype raises the bar. If a model fails one time in 20, you'll hit a few failures by prompt 100, and those are the ones you remember.
The creator also reports repeating the same prompt on the same model two weeks after launch and getting a clearly worse picture. One person's retest isn't a controlled study, but it's the kind of check anyone can run.
Is it legal for AI companies to change what you get?
The source's research found nothing that requires a company to tell you when it turns these dials down. The closest thing found was a Connecticut law that requires disclosure of usage limits before you sign up or renew. It says nothing about whether the model got worse. Rules are likely still catching up, so for now the person checking is you.
What this means for you
If you pay for an AI tool at work, or rely on one for a workflow, treat quality as something to monitor, not assume.
- Set effort deliberately. If your tool has an effort setting, don't accept the default without checking how it affects your results.
- Read benchmarks with the settings in mind. Ask what effort level a demo or score used.
- Build a small nerf test. Here is a template:
Pick 5 prompts from your actual work (a summary, a spreadsheet
formula, an email draft, a piece of code, a reasoning task).
Save each prompt exactly as written.
Run all 5 on launch day with the same settings.
Save the outputs and note the model and effort level.
Rerun every 2 weeks. Compare against the originals.
Score each 1-5 on whether you'd use it as is.
- Keep notes on which app version you use. If results drop, the cause may be an app update, not the model.
- Have a fallback. If one tool gets worse, another may serve you better that month.
- Check for silent routing. If your tool says which model answered, look at it.
Could you build your own AI quality tracker?
A nerf test works best when it runs on a schedule and stores results, which is awkward in a spreadsheet. With Remy, you could describe a small tracker in plain English: a page where you save your five prompts, run them on a recurring schedule against several models, store each answer, and show a score history over time. Remy apps can include a database, scheduled jobs, and access to nearly 400 AI models, so one app could compare the same prompt across models. You'd own the result instead of paying for another monitoring subscription.
Frequently Asked Questions
Did Anthropic and OpenAI really admit their models got worse?
Yes, in terms of user experience. Anthropic published two postmortems acknowledging Claude got worse for weeks at a time, and said its own evaluations missed it. Sam Altman said GPT-5 seemed way dumber the day after launch because something broke behind the scenes.
Do companies secretly swap in a cheaper, quantized model?
There is no proof of it. Quantization, which rounds a model's numbers to make it cheaper to run, is a popular theory. Nobody has admitted or leaked it, and daily trackers haven't detected the weights changing. Small changes can still slip past those trackers.
Why does the same model score differently on launch pages and in my app?
Effort level is one reason. The Opus 5.5 launch page, as cited in the source, showed 57.8% on a coding test at max effort and about 52% at the default effort. Your tool's default may be lower than the setting used for the headline number.
How can I check if my AI tool got worse?
Save five prompts from your real work, run them on day one, and rerun them every couple of weeks with the same settings. Compare the outputs side by side. If results drop, check whether the tool updated, since an app update caused one of the biggest drops the trackers found.
How we source, write, and correct our articles. Editorial standards
Related articles

What Does Jensen Huang Leaving Nvidia Mean for Your AI?
Live Nerf vs Margin Lab: Do AI Models Really Get Worse?

Why Is Nvidia So Dominant in AI Chips? The CUDA Moat Explained
