METR's Anthropic audit explained: AI now speeds up its own R&D ~2x

An independent evaluator has spent eleven weeks inside Anthropic and come back with a number that sounds abstract but is not: the lab's AI models are now improving about twice as fast as they were a year or two ago, and the best guess is that most of that speed-up comes from AI helping to build the next AI. The twist is that the people who did the measuring say you cannot fully check their work, because the company they audited removed most of the evidence they found most informative.

The plain-English version

Imagine a bakery that makes better bread every year. For two years the owners improved the recipe at a steady pace. Then, about a year ago, the improvement rate doubled. Why? Maybe they hired more bakers, bought better ovens, or got lucky. Or maybe the new ovens themselves started suggesting recipe tweaks, so the bakery now improves partly because of its own bread-making machines.

That is the question METR (an independent nonprofit that evaluates frontier AI) set out to answer about Anthropic, the company that makes Claude. From 12 August to 29 October 2026, a small METR team worked inside Anthropic with access to internal data and about 15 staff interviews. They asked three things: has the pace of improvement sped up, how much of that is because AI is helping, and is anything likely to slow it down?

Their answers, with plenty of uncertainty attached:

• Faster: on tasks with clear right answers, Anthropic is improving its models at roughly twice its March 2024–October 2025 rate. METR "would not be surprised" by 1.3x or 3x.
• Mostly AI: METR guesses about 80% of the current acceleration is due to AI assisting the research. It stresses this is a judgment, not a measurement.
• No obvious brake: it found no convincing bottleneck over the next 12 months in ideas, computing power, experiments or data, though its view of this changed several times.

Importantly, AI is not running the lab. Researchers describe it as mostly assisting humans; there are "signs of life" of more autonomous contributions in some areas. Think of a very fast research assistant, not a self-directing scientist.

The second half of the story is trust. Anthropic had the right to cut anything from the public report, and METR says it removed most of the evidence METR found most informative. So you are asked to trust METR's judgment. METR itself says that is "much more than would be ideal."

How it actually works

Measuring "AI speeding up AI" is hard because many things change at once: more staff, more chips, better data, and better AI tools. METR's approach was to triangulate with several independent lines of evidence and see whether they agree.

1. A trend line with a kink. Anthropic tracks its models on AECI, its internal version of the Epoch Capabilities Index, a composite of 15 benchmarks. METR fitted a trend that bends at a fixed date, 7 October 2025. Before that, the index rose about 15 points a year; after, about 29. The ratio is the headline 1.9x (95% confidence interval 1.5–2.8x). Put differently, Anthropic made about as much benchmark progress in the last year as in the two years from October 2023 to October 2025.

2. A "without AI" baseline. Acceleration is only interesting if it is caused by AI. METR separates three growth rates: the historical one, the current one, and a counterfactual one (what growth would be today if AI weren't helping). The counterfactual can only be estimated. METR sketches an example: if the historical pace is 2x per year, the current pace is 9x, and the no-AI pace is 3x, then the AI-driven acceleration is $\log 9 / \log 3 = 2$.

3. Checking the pipeline. METR used a simple accounting picture: better algorithms, better data and more compute feed pretraining strength and reinforcement-learning strength, which together produce capabilities. It requested data at each step: efficiency histories, compute use, headcount, lines of code, pull requests, spending on AI agents. Anthropic shared more than first scoped. Most of the details of how this accounting was used are redacted, but METR says it is confident the acceleration was not driven by faster growth of training compute.

4. Asking the people. Internal Slack polls of self-reported AI uplift rose from about 1.2x (Claude 3.7 Sonnet, January 2025) to about 12x (Model 4, October 2026). METR calls that "very loose evidence": respondents were an unrepresentative subset giving low-effort answers, and personal speed-up is not the same as faster lab-level progress. In the interviews, Claude classified 11 interviewees' statements about whether their area is accelerating by more than 2x: 2 yes, 3 leans yes, 4 borderline, 0 leans no, 2 no.

No single line is decisive. The case rests on them pointing the same way, and on METR's confidence that non-AI explanations (compute above all) do not account for the jump.

The deep dive

The benchmark fit and its fragility

AECI is a latent-ability model: each model gets a strength score, each benchmark a difficulty, and pass rates are linked to ability through a link function (logistic in the main fit). The "acceleration" is the slope after the breakpoint divided by the slope before, about 29/15. The choice of link function matters a lot, and METR publishes the sensitivity:

Across all variations the range is 1.1x–2.9x. One variant, complementary log-log, essentially erases the effect. That is why the authors keep a wide band around "about 2x".

Two other checks matter. First, slope versus jump: a permanent change in slope fits recent models better than a one-time jump at Mythos Preview (held-out RMSE 1.9 vs 3.9 AECI points; slope change won in 81% of bootstrap draws). Second, saturation: Model 4 scores above 50% on all 15 benchmarks. Dropping three poorly linear benchmarks moved its score only from 178.6 to 179.7, so METR sees little sign that saturation is hiding or inflating the trend.

The estimate also depends on which models are included. Anthropic's Haiku 5.5 system card reported 1.7x; METR's 1.9x is higher mainly because it includes newer, internal-only models. In METR's review of the Risk Report, the estimate was about 1.5x on the original data and about 1.9x after more benchmarks were added.

Is it Anthropic, or the whole field?

Using all frontier developers, METR finds the trend rising from about +17 to +24 points a year, i.e. 1.4x. The authors give two readings: Anthropic's pre-2025 progress may have been understated (so 1.4x is closer to the truth), or earlier points may be more overfit to benchmarks (so 1.9x is closer). Jessica Taylor, a former MIRI staffer, has noted that Anthropic's internal index remains close to linear, per BigGo's roundup.

Attribution: the 80% number

The hardest part is splitting the acceleration. METR says non-AI factors likely drove much of the late-2025 acceleration, while AI's share has grown since. Its "about 80%" for October 2026 is a subjective guess. The interviews are a good illustration of the spread: one interviewee put AI's share at "15–20%" and did not trust self-reported productivity gains, while another is quoted in The Decoder as saying they are "now only bottlenecked on ideas & compute, no longer bottlenecked on code."

Inference cost per task

METR also reports how many tokens the benchmark attempts consume: internal benchmarks use about 2.5M input and 72k output tokens per attempt; agentic coding about 10M input and 94k output. For comparison, Epoch AI's typical task is about 0.5M input and 10k output. Heavy token use is a reminder that these scores come from long agent runs, not single answers.

The threshold dispute

Anthropic's Responsible Scaling Policy (v3.4) has two triggers for the "Automated R&D in key domains" threat model: a model that could substitute for Anthropic's entire research staff within a factor of 5 in cost, or progress at double the baseline rate that is substantially attributable to AI R&D automation. The August Risk Report (coverage date 15 July) judged risk "Low" and said acceleration was "by less than a factor of 2." In its companion review, METR computed that the private report's central estimate implies 1.97, argues "around 2" is the fairer phrase, and says crossing the 2x threshold by 15 July was plausible. It agrees that catastrophic risk from the July models was low and that full substitution had likely not occurred. Anthropic's amended S-1 filing of 2 November says it "may imminently meet" the threshold, according to coverage of the filing; no current model has been designated as exceeding it.

Process facts

METR took no money and worked on Anthropic devices; the engagement started as a four-week contract and grew to eleven weeks at about 2.5 full-time equivalents. A planned verification of compute allocation between capabilities and safety was cancelled midway, with Anthropic saying it wanted to keep the work focused. Most analysis and drafting used Claude; METR says it cross-checked against human interviews and found no significant discrepancies.

Why it matters

For everyday users: the faster models improve, the shorter the gap between "impressive demo" and "default feature". If the 2x holds on practical tasks, expect product changes to arrive in months rather than years. It also strengthens the case that tools you use today, including AI that writes code, are already part of the machine that produces tomorrow's models.

For developers and builders: a faster release cadence means shorter shelf lives for prompts, workflows and model-specific tooling. METR's evidence suggests the speed-up is concentrated in well-specified, checkable work, which is exactly the type of work agent tooling targets. Building on evals you control matters more when the underlying model changes quickly.

For companies and investors: Anthropic's own IPO filing flags the threshold, and the risk of acceleration now sits alongside the financial risk. Our earlier piece on Anthropic's S-1 covers the compute commitments that this trajectory has to pay for. Reported AI stock jitters over whether infrastructure spending can keep its pace make the question of how fast models really improve a financial one too.

For the field and for governance: this is a template for how frontier risk may be assessed: a third party gets high access, writes an assessment, and the lab decides what the public sees. METR scores itself against its own transparency checklist and marks several items as not fulfilled, including methodological transparency, disclosure rights and limits on redaction. Palisade Research's Jeffrey Ladish put the problem plainly: "I trust METR, but I don't think that the world should have to trust METR on this one." Daniel Kokotajlo, per The Decoder, criticised dropping the compute audit because slowdown commitments need to be verifiable.

A second-order effect: if AI-driven acceleration is the dominant driver, the usual levers (hiring, chips) matter less and the quality of the AI tools matters more, which favours whoever is already ahead.

What to be skeptical of

• It is a judgment call. METR's "about 2x" and "about 80%" are expert estimates. The only formal interval (1.5–2.8x) is for one benchmark index, and robustness checks span 1.1x–2.9x.
• The best evidence is unseen. Anthropic redacted a majority of the evidence METR found most informative, reportedly including forecasts, breakdowns by training stage and the bottleneck analysis. Anthropic co-founder Sam McCandlish told AP the details are commercially sensitive and could help other developers accelerate, and acknowledged the redactions make METR's work harder to check (CryptoSlate).
• Narrow tasks. The result applies to well-specified, automatically gradable tasks. Transfer to messier research is more uncertain, which could make real acceleration lower.
• Baseline problems. A slow pre-2025 Anthropic baseline would exaggerate the ratio; METR's all-lab figure is only 1.4x.
• Access and conflicts. METR lacked query access to unreleased models and says Anthropic's feedback on the Risk Report review was sometimes "overly forceful," though it found no significant distortion. One team member had discussed a role at Anthropic, declined an offer, and held equity in a competitor as a former employee; their work was checked by an unconflicted colleague. METR also relied heavily on Claude for analysis.
• Uplift polls are weak. The 12x is self-reported by a self-selected group.
• Not a doom forecast. METR agrees that catastrophic risk from the July models was low; its concern is that continued acceleration, with no convincing bottleneck in sight, is a significant risk.

What to watch next

1. Anthropic's own response. Whether Anthropic releases any of the redacted evidence or publishes its own account of the findings.
2. The S-1 and RSP. Whether Anthropic formally designates a model as meeting the "dramatically accelerated progress" threshold, and what safeguards follow. See our coverage of the S-1.
3. Newer models' AECI scores. Whether upcoming Anthropic models keep tracking the steeper post-October-2025 trend; a flattening would weaken the story, continued steepness would strengthen it.
4. Replication elsewhere. Whether OpenAI, Google DeepMind or another lab permits a similar high-access audit; METR's all-lab figure is 1.4x versus 1.9x for Anthropic.
5. The cancelled compute audit. Whether anyone restores verification of how compute is split between capabilities and safety.
6. Related agent-safety work. Our pieces on Anthropic's AI red team and AI agent teams that stopped telling humans cover how oversight breaks when AI does more of the work.

How this was made: researched and written by an AI model (Claude) from the primary sources listed above, then checked claim-by-claim against those sources in a separate AI fact-check pass. Spotted an error? Email [email protected] and we correct it publicly. Our process.