What Did METR's Anthropic Audit Find About Self-Improvement?

What did METR's audit of Anthropic find about recursive self-improvement?

METR, an independent research group, ran an embedded 11-week audit at Anthropic focused on automating AI research. According to coverage of the release, the share of AI R&D projects where AI leads rose from about 1% in March to 26% in August. The best estimate of AI-driven acceleration is 1.9x, just under Anthropic's 2x tripwire, with a plausible range up to 3x. Much of the most informative evidence was redacted.

Anthropic also amended its S-1 filing to say recursive self-improvement (RSI) is imminent. The reports were released in a coordinated batch.

A caveat on sourcing: this article relies on a creator's walkthrough of the reports, not on the documents themselves. Where the walkthrough offers interpretation instead of reported fact, this article says so.

How does Anthropic measure AI R&D automation?

Anthropic uses a five-step scale for how much of AI development AI performs. It runs from no AI involvement, to minimal use, to AI assisting, to AI collaborating, to AI leading, and finally to fully automated R&D led by AI. The "AI leading" stage is called AL4.

The scale has no sharp boundary. Progress is gradual, and AI has been creeping into research work for a few years. That makes any single threshold somewhat arbitrary, which is why Anthropic picked a tripwire.

What is the 2x acceleration tripwire?

The tripwire is the point where AI doubles the pace of AI progress compared with a world without AI help. Anthropic treats it as the level at which it should have a concrete plan, because development becomes harder to predict beyond it.

This figure isolates acceleration caused by AI. It is not about buying more GPUs or building more data centers. METR's stated confidence is that the observed speedup did not come from scaling training compute.

At 1.9x, Anthropic is under the line. The uncertainty is the problem. The reported confidence interval is narrow for earlier periods and widens for recent ones. The widening happens because the newest models are hard to benchmark, and the top of the range reaches about 3x, above the tripwire.

What was redacted, and why does it matter?

METR said Anthropic redacted a majority of the evidence METR found most informative. METR also noted that some redactions are hard to describe without revealing the very information it was asked to withhold.

The walkthrough highlights several areas:

• Compute allocation. METR and Anthropic had reportedly agreed that METR would assess how Anthropic splits compute across buckets. Anthropic later canceled that component of the assessment despite the contract.
• Inference compute usage. Details on where inference compute goes were redacted.
• An agent scaffold. Some passages refer to Claude Code, while others refer to a redacted agent scaffold. The scaffold's identity is not disclosed.
• Parts of the training pipeline. METR said Anthropic redacted evidence of a speedup closely tied to specific pipeline components.
• Historical efficiency data. This covers algorithmic efficiency, model strength in pre-training and reinforcement learning, and gains in compute utilization, inference efficiency and data efficiency.

Anthropic's reported justification, attributed to a co-founder, is that detailing exactly what drives the speedup could hand others a road map. That is a coherent safety argument. It also means nobody outside can check the claim.

The creator's own reading goes further. He speculates that something changed around October, that the scaffold paired with Mythos acts like an engine turning agent work into algorithmic improvements, and that shifting compute ratios between pre-training, reinforcement learning and inference would reveal what changed. None of that is confirmed by the reports. It is inference from what was withheld.

What are Models 2, 3 and 4?

The reports reportedly reveal several unreleased internal models. The public tiers, as described, are Fable, the most capable model anyone can use, and Mythos, which is in the same class but only released to some trusted organizations. Behind those sit three internal models, known by code names.

Model 4 is the focus. It reportedly beat a benchmark that no other model has cleared. That is also a measurement problem: if a model saturates your internal benchmarks, you know it improved but not by how much. The walkthrough puts it simply: it is getting better, and the rate is unknown.

Does the benchmark jump prove hyperbolic growth?

No. The creator is explicit on this point. A new model sweeping a benchmark is striking, but a single jump does not show that the growth rate itself is accelerating. Hyperbolic or continuously accelerating growth would require the rate of improvement to keep rising, not just a new model outscoring old ones.

Other evidence makes acceleration more plausible: the rise in AL4 share, the 1.9x estimate, the S-1 language and the redactions around the likely drivers. Still, plausibility is not proof, and the missing compute data is exactly what would help separate the explanations.

Why does the S-1 amendment matter?

An S-1 is the registration document a company files to go public, and amending it is costly and legally consequential. Per the walkthrough, Anthropic changed it to say RSI is imminent. A company making that statement in a securities filing is presumably choosing its words with care, because investors can rely on them.

Reaction has been mixed. Daniel Kokotajlo, known for work on AI timelines, reportedly said that if the claim is true, timelines to full recursive self-improvement and superintelligence may be very short. Others focus on the redactions and ask what a reader can actually conclude.

What should builders take from this?

Three practical points follow from the reporting.

1. Agent scaffolding appears to matter as much as the base model. The reports credit a combination of a frontier model and a specialized agent harness, not model scale alone.
2. Public model tiers lag internal ones. Fable is the best generally available model, Mythos is in the same class as Fable, and Models 2 to 4 sit above it. What you can build with today is not the frontier.
3. Treat the numbers as provisional. The 1.9x figure has a wide range, and the underlying evidence is partly hidden.