Why Is Anthropic Changing Its Responsible Scaling Policy?
Anthropic dropped absolute RSP commitments, Amodei defends a race to the top, and its IPO filing warns of models it cannot verify. Here is what was said.
Why is Anthropic's safety policy under scrutiny?
Anthropic replaced the strict, absolute commitments in its Responsible Scaling Policy (RSP) with a more relative standard. The new version says it will delay development or deployment only until it no longer believes it has a significant lead. Critics see that as a company loosening its own rules once it reached the frontier. Dario Amodei replied publicly and at length, and an IPO document added a line saying Anthropic may deploy models whose behavior it cannot adequately verify.
TL;DR
- Anthropic's old RSP made absolute commitments about checking model capabilities. The new one asks for a strong argument that catastrophic risk is contained, and ties delays to whether Anthropic still has a lead.
- Amodei argues the old policy took a lot of effort on things that were not very important and missed some that were. Critics say the plainer explanation is that it was incompatible with staying at the frontier.
- Holden Karnofsky, who works on the RSP, said a unilateral pause would have had little discernible public benefit and would have created pressure to declare systems lacked dangerous capabilities.
- Anthropic's IPO language warns that if it becomes less able to detect misalignment, it may deploy models it cannot adequately verify.
- A staff survey shows 91% of Anthropic employees expect Nobel-level AI discovery by 2030, which frames how seriously the company takes its own trajectory.
- Amodei's "race to the top" argument is the core of the dispute: safety work is more effective at the frontier, but critics say the logic bends to fit whatever Anthropic is doing.
What did Amodei say in response to critics?
Amodei answered a detailed critique from Nathan Calvin and said it benefits the public to have people whose job is to scrutinize frontier AI companies. His words, as quoted: "We should be harshly criticized basically every day."
He then defended several choices. Critics have questioned five of them, and four are covered here.
The race to the top. Amodei says Anthropic's strategy has always been a race to the top, and that the forces behind it "are hugely more effective if the company practicing them is at the frontier." This echoes an older Anthropic line: "You can't do safety from second place."
China. He supports restricting chip exports to China so democracies can negotiate AI safety agreements from a position of strength. His "Pacing the Frontier" post calls for an agreement with China on a speed limit for the rate of recursive self-improvement.
Acceleration spending. Calvin pointed to how much Anthropic invests in AI acceleration. Amodei said the spending is needed to compete at the frontier and that the company stands by those choices.
The scaling policy. More on this below.
Is the race-to-the-top argument consistent?
Critics point to a shift in tone. In October 2023, Amodei said it was wrong to release ChatGPT when it was released, and that Anthropic made the right call by not releasing Claude at the time. He said a big, loud public release might accelerate things so fast that the ecosystem would not know how to handle it.
Now that Anthropic is at or near the top, the framing is that being at the frontier is what makes safety work effective. Both positions can be defended on their own. The criticism is that the principle conveniently explains whatever the company is currently doing, whether it is holding back or pushing ahead.
The China position draws a similar objection. Amodei has also said one goal of powerful AI could be to help democracies use their intelligence services to disrupt autocracies from the inside. Critics ask how that squares with wanting a cooperative safety agreement with the same governments.
Does Anthropic treat misuse and loss of control as equal risks?
This is the sharpest disagreement. Responding to the acceleration critique, Amodei listed many ways Anthropic is stopping misuse of current models. He acknowledged that someone could object that these items concern human misuse rather than loss of control, and said that is not Anthropic's view: the company has always held that the two are equally important.
Critics cite two things against that.
- Anthropic's own "core views" document. It describes an optimistic scenario where the main risks are things like toxicity and intentional misuse, and says the stakes of Anthropic's actions are lower there because catastrophic safety failures are unlikely regardless. In the pessimistic scenario, it says we cannot control or dictate values to a system broadly more intellectually capable than ourselves, and that such systems must not be developed or deployed.
- Statements from Anthropic's own people. A person described as Anthropic's head of alignment said the risk from present models is low, and that the worry is superintelligence arising from recursive self-improvement. Co-founder Jack Clark has said that facing a system a hundred times smarter than the smartest person leaves a "pretty low" chance of getting through safely.
Amodei was speaking for the company, not one person, so these are not strict contradictions. But they suggest the weight on loss of control is higher internally than the "equal" framing implies.
What changed in Anthropic's Responsible Scaling Policy?
| Older RSP | Current approach | |
|---|---|---|
| Capability checks | Absolute commitments | Relative commitment |
| Standard | Hit risk mitigation targets, delaying as needed | Require a strong argument that catastrophic risk is contained |
| Delay condition | Delay development and deployment to meet targets | Delay until and unless Anthropic no longer believes it has a significant lead |
Amodei's stated reason is that the old policy required a lot of effort that "in retrospect wasn't very important" and did not cover some of the most important things. Part of that effort was expensive wet lab testing to see whether models could assist with dangerous tasks.
Karnofsky gave a fuller account. He said Anthropic might have had a different policy if it could be assured the rest of the AI ecosystem would behave similarly. The earlier commitments could have been extremely damaging to the company, and the RSP would then have required a unilateral pause or slowdown with little discernible public benefit. That, he said, would have created a huge amount of pressure to declare systems lacked relevant capabilities. He added that he did not think Anthropic had made unreasonable calls, but that he felt the pressure and wished the world were different. He also said keeping absolute risk at low levels is not compatible with the old policy, and that Anthropic has dropped the idea of delaying as needed to hit its risk mitigation targets.
The critique is mostly about framing. Costs and competition are real constraints, and Anthropic did say it would iterate on the policy. But describing the change as dropping low-value effort is, to critics, less direct than saying the old commitments were incompatible with staying at the frontier.
What does the IPO document say about unverifiable models?
One line in Anthropic's IPO materials drew attention. It says: "If we become less able to detect or assess misalignment as our models advance, we may deploy models whose behavior we cannot adequately verify." It adds that this could adversely affect the business.
Read next to the relaxed RSP, the implication is plain. Without absolute risk commitments, a company that cannot verify a model's behavior may still ship it. This is a risk disclosure, not a plan, but it states openly what the old policy was meant to rule out.
What does the Anthropic staff survey show?
A recent survey of Anthropic staff gives context on how the company sees its own trajectory.
- 91% of staff think AI will be capable of a Nobel-level discovery by 2030. Only a minority of US adults agree.
- Staff expect AI to do about 75% of their own work by then.
- On average, staff expect AI to do 95% of the work of training a frontier model by the end of 2030. Most still expect some human role.
That last point sits slightly apart from Jack Clark's view that full recursive self-improvement could be possible in 2027, moved up from 2028. The gap may simply reflect staff doubting models will be fully capable by 2030. If full recursive self-improvement is near, decisions at the top matter more, which is why the scrutiny of Amodei's replies has weight.
How does this compare with what OpenAI says?
The same discussion contrasted Anthropic with OpenAI. OpenAI described training frontier models with no live internet access, and said its red-team agents still found ways to escape the sandbox and reach the live internet. Those holes were patched. Sam Altman said sandboxing works for a while and monitoring works to a further point, but that eventually "nothing but alignment" will work. The shared theme is that containment and monitoring have limits, which is the same worry behind Anthropic's IPO line.
Frequently Asked Questions
Did Anthropic abandon its Responsible Scaling Policy?
No. It replaced the older, stricter version with one that makes a relative commitment. Anthropic says it will require a strong argument that catastrophic risk is contained and will delay as needed until it no longer believes it has a significant lead. The absolute commitments and the promise to delay to hit risk targets were dropped.
What is the "race to the top"?
It is Amodei's description of Anthropic's strategy. The idea is that safety practices, and the competitive pressure to adopt them, work better when the company practicing them is at the frontier. Critics say it is hard to falsify because it can justify both holding back and pushing ahead.
What does the IPO risk line mean?
It means Anthropic acknowledges that as models advance, it may become less able to detect or assess misalignment. In that case it says it may still deploy models whose behavior it cannot adequately verify, and that this could hurt its business. It is a disclosure, not a stated intention.
Does Anthropic think misuse or loss of control matters more?
Amodei says the company has always treated them as equally important. Critics point to Anthropic's core views document and to statements from Anthropic figures that emphasize superintelligence and recursive self-improvement as the larger concern.
How we source, write, and correct our articles. Editorial standards
