--- title: "What Is Anthropic's AECI Benchmark and Why Log-Odds?" url: "https://goremy.ai/blog/articles/aeci-benchmark-log-odds-saturation-explained" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-11-10T14:20:09.330Z" updated: "2026-11-10T16:57:23.101Z" topic: AI Models & Industry description: Anthropic reportedly tracks progress on a composite benchmark in log-odds. Here is how saturation hides gains and why Model 4 is hard to measure. --- # What Is Anthropic's AECI Benchmark and Why Log-Odds? > Anthropic reportedly tracks progress on a composite benchmark in log-odds. Here is how saturation hides gains and why Model 4 is hard to measure. Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-10 ## What is the AECI benchmark? AECI is an internal composite benchmark that Anthropic reportedly uses to track model capability over time. It combines results from several tasks into one trend line, and that line is plotted in log-odds rather than raw percent correct. A 50% difficulty line marks where a model solves the typical task half the time. The public material does not spell out the benchmark's full design, so treat the details below as the general logic, not a spec. ## TL;DR - AECI is described as a **composite benchmark** shown as a trend across successive models, not a single test with a single score. - Plotting in **log-odds** keeps a chart readable when scores crowd toward 100%, where raw percentages flatten out. - The **50% difficulty line** gives a fixed reference: it marks where a model solves the typical task about half the time. - **Benchmark saturation** makes progress look like it is slowing when the test has simply run out of headroom. - **Confidence intervals widen** at the frontier because few unsolved tasks remain to measure with. - The newest internal model, called Model 4 in the coverage, reportedly beat the benchmark in a way earlier models did not, so it is **hard to place on the scale**. ## Why does benchmark saturation hide progress? A benchmark has a ceiling. Once a model scores 90% or 95%, there is little room to show improvement. A model that is twice as capable as its predecessor might only move from 95% to 98%. On a percent chart, that looks like a rounding error. The tasks are also not equal. Easy ones fall first. What remains are the hardest items, and some of those may be flawed, ambiguous, or mislabeled. At the top of the range, the benchmark increasingly measures noise and quirks of the test instead of capability. This is why a model that "broke the benchmark" is a measurement problem as much as a win. If a model solves nearly everything, you know it is better than its predecessors. You do not know by how much. ## What does plotting in log-odds actually do? Log-odds (the logit) converts a probability p into ln(p / (1 − p)). It stretches the ends of the 0 to 1 range so that equal steps on the chart mean equal multiplicative changes in the odds of success. A few reference points show the effect: | Success rate | Odds | Log-odds | |---|---|---| | 50% | 1 to 1 | 0 | | 90% | 9 to 1 | about 2.2 | | 99% | 99 to 1 | about 4.6 | Going from 50% to 90% moves the line about 2.2 units. Going from 90% to 99% moves it another 2.4. On a percent axis, the first jump is 40 points and the second is 9. In log-odds, they look comparable, which matches the intuition that cutting the failure rate from 10% to 1% is a real achievement. This also explains a remark in the video coverage that steady exponential improvement can be displayed as a straight line. Log scales do that: constant proportional growth becomes constant slope. A straight line on this chart does not mean slow progress. It can mean steady multiplicative progress. ## Why is there a 50% difficulty line? The 50% line is an anchor. Log-odds is zero there, so it works as the chart's midpoint. Tasks can be sorted by how hard they are, and a model's position can be read as "it handles tasks up to roughly this difficulty about half the time." That framing has two advantages. First, 50% is where a measurement is most sensitive. Small capability changes move the success rate the most near the middle of the curve, so the estimate there is the most precise. Second, it gives a consistent yardstick across model generations. Each new model can be compared against the same line instead of against whatever percentage happened to be reported that month. The sources here do not describe exactly how Anthropic builds its difficulty scale, so the details of that calibration remain unclear. ## Why do confidence intervals widen at the frontier? The coverage describes a chart where the shaded band is narrow for older models and fans out for the newest ones. Several general reasons produce that shape. - **Few informative tasks remain.** If a model solves almost everything, only a handful of items carry information about its upper limit. - **Log-odds amplifies small differences near the extremes.** One more or one fewer solved task near 100% moves the logit a lot. A perfect score maps to infinity, so estimators need adjustments, and the uncertainty grows. - **There are no peers to calibrate against.** Earlier models fail on the hard tasks, which tells you those tasks are hard. A model that clears them leaves nothing to compare with. A wide band is an honest signal. It says the benchmark can confirm the model is at the top, but it cannot say how far above the previous model it sits. ## What happened with Model 4? According to the video coverage, which draws on Anthropic's IPO page and a report from the evaluation group METR, Anthropic has several internal models that are not publicly available. The coverage lists Model 2 as finishing training in July 2026, Model 3 in August to September 2026, and Model 4 in September to October 2026. Model 4 is described as the first to clear a line on this benchmark that no earlier model crossed. The key consequence is the one above. The newest point sits where the chart is least reliable. The coverage summarizes it this way: the model is clearly getting better, but how fast is unknown. The same coverage ties this to a separate measure. Anthropic reportedly estimates AI-driven acceleration of its own research at about 1.9x, against a safety tripwire at 2x, with an interval reaching as high as 3x. It also says the share of projects where AI leads the R&D work went from 1% in March to 26% in August. Those are the video's reported figures, not numbers verified against the primary reports here. ## Does a saturated benchmark prove runaway growth? No. The video's own framing is careful on this point: a model blowing past a benchmark is striking, but it is not evidence of hyperbolic growth by itself. A ceiling effect can make a modest improvement look dramatic, and it can make a large one look small. The chart cannot tell the difference once the model is off the scale. Much of the surrounding commentary is interpretation. The METR report reportedly contains heavy redactions, and the video speculates about what sits behind them, including an unnamed agent scaffold and compute allocation. That is conjecture. What the evidence supports is narrower: the measurement tools are running out of range at the same time that the stakes of measuring accurately are going up. ## What would a better measurement look like? Standard fixes for saturation apply here: 1. Add harder tasks so the top of the scale has resolution again. 2. Use long-horizon tasks, where success depends on sustained work instead of one-shot answers. 3. Report intervals prominently, so a wide band is not hidden by a clean line. 4. Cross-check with real-world indicators, such as how much actual research work AI does, instead of relying on one synthetic score. The shift toward measuring AI contribution to R&D itself, rather than exam-style scores, reflects the same problem. When tests are solved, the useful question becomes what the model does in practice. ## Frequently Asked Questions ### What does AECI stand for? The available material does not define the acronym, so it is not safe to guess an expansion. What the coverage does establish is that it is a composite benchmark, plotted as a trend across models, in log-odds, with a 50% difficulty reference line. ### Why not just report percent correct? Percent correct compresses everything near the top. Going from 95% to 99% is a large reduction in errors but only four points. Log-odds spreads those gains out so they stay visible, and it makes a steady rate of improvement show up as a steady slope. ### Does a wide confidence interval mean the result is unreliable? It means the result is imprecise, not wrong. The interval shows the range of capability levels consistent with the data. For a model near the ceiling, that range is wide because the benchmark has little left to separate it from its peers. ### Can anyone run this benchmark? Based on the sources, it is an internal Anthropic measure, and the models it tracks include ones not released to the public. Nothing in the material indicates it is available for outside use. ### Is Model 4 available to the public? The coverage describes Models 2, 3, and 4 as internal and unreleased. The publicly available models mentioned are the Fable-class and Mythos-class systems, with Mythos reportedly limited to some trusted organizations.