How Well Does AREX-2 Score? MLE-Bench Lite and Skills Ablation

How well does AREX-2 score on benchmarks?

AREX-2 scores 81.8 on MLE-Bench Lite, ahead of GPT-5.6 Soul at 72.7. On FrontierCS, a set of open-ended algorithmic problems, it scores 70.7, the best result among open-weights models but roughly six points behind GPT-5.6 Soul. It is a 27B model released under Apache 2, fine-tuned from Qwen3.8 27B. The catch is that the MLE-Bench Lite number was produced with skills in context, and those skills account for most of the gain over the base model.

What is AREX-2?

AREX-2 comes from the Beijing Academy of Artificial Intelligence (BAAI) with two other Chinese universities. There is a paper on arXiv. Outside it, the main write-up appears to be a Chinese-language blog post, so English coverage is thin.

The pitch is about where the agent loop lives. In most agent systems, the harness does the try, check, retry, keep-the-best, and when-to-stop logic. The model produces one attempt at a time. AREX-2 is trained to run that loop itself. You give it a budget, for example five hours or a set number of rounds, and it paces its own work: early rounds measure, later rounds act on what was measured.

How does AREX-2 handle long-running work?

BAAI breaks the capability into two parts.

1. Reflection. Given the current state and the feedback just received, can the model produce a better attempt? This determines how much each round is worth.
2. Long-horizon execution. Can it keep going for hours, remember what it has tried, and recover when a change makes things worse? This determines how many rounds stay useful before the run stalls.

The two multiply. With weak reflection, extra rounds just burn tokens. With a short horizon, even sharp reflection plateaus. That split is a useful diagnostic for your own agents. If yours stalls, ask whether it is making bad changes (a feedback-signal problem) or losing the thread and repeating old attempts (a state and memory problem). Different failures, different fixes.

How was AREX-2 trained?

Typical agent training data keeps only verified passes. That shows what a correct answer looks like, not how an answer gets better. The failed attempts, feedback, and fixes are discarded, even though messy runs that start badly and end well are the most informative.

BAAI built its own data in two domains, machine learning engineering and algorithmic programming. Both give unambiguous feedback (a validation score moves or it does not, tests pass or fail) and leave room to improve for a long time. The pipeline has three steps:

1. A teacher model turns GitHub repos into environments, each with a task and a scoring function.
2. A teacher agent works on an environment for hours, making hundreds of tool calls: reading code, searching papers, running small experiments.
3. Whole trajectories are kept based on how they finish. Failures stay in context, but training applies only to steps that moved things forward.

Despite the launch wording about reinforcing these tasks, this is supervised fine-tuning, mixed with deep research data from the first AREX model. It is not RL. The reusable idea for anyone fine-tuning on their own agent traces is to select trajectories by outcome and mask the loss so the model learns only from recovery and progress. The sources do not say which teacher model generated the data, and neither the datasets nor the training code appear to be released.

What does the 5-hour curve show?

The time curve may be the most interesting chart. AREX-2 does not start in front. It scores 54 after one hour, 66 after two, and 70.7 after five, the same figure as its FrontierCS result. DeepSeek 4 Pro, by comparison, flattens at about 45 after two hours and barely moves afterward.

That sustained improvement is the point of the design. The long-horizon training also lets AREX-2 beat earlier AREX models on all four of BAAI's research benchmarks, including the 122 billion parameter version.

How much comes from skills versus training?

MLE-Bench Lite was not run on the bare model. AREX-2 had skills in context: general ML-practice skills plus task-specific ones built by researching each problem. BAAI published an ablation that separates the contributions.

Skills and extra rounds take the untrained base model from 28.8 to 68.2, about 39 points. Swapping in AREX-2 under identical conditions adds about 14 more. So the training is real, but it is the smaller lever. Note that the third row bundles task-specific skills with additional rounds, so the table does not isolate skills alone. Anyone who has built good skills for an agent will find the size of the effect unsurprising.

The comparison against GPT-5.6 Soul also deserves care. The sources do not detail what scaffolding that model received, so the 81.8 versus 72.7 gap is a headline number, not a clean like-for-like measurement of model quality.

Is this recursive self-improvement?

BAAI frames it that way, but what is shown is self-improvement at test time. Inside one task the model improves over many rounds, yet the weights never change. The next task starts from scratch. It is like an intern who tries 20 approaches in a week and then has their memory wiped before the next assignment.

True recursive self-improvement would feed successful trajectories back into training so the next generation does not need the 20 attempts. The paper itself names that as the path forward.

What should harness builders take from this?

If models are trained like this, harnesses do not disappear, but their job shifts away from controlling retries, keeping the best version, and deciding when to stop. The reviewer's reading is that three things remain:

• A real feedback signal: scores, logs, and errors.
• Operational knowledge, which is the skills layer the ablation shows matters so much.
• A budget that sets how long the model should run.

There is likely friction. If your framework has its own reflect-and-retry nodes and the model paces itself across one long session, you get two loops fighting. A harness that resets context every few steps could make a model like this worse.

What did the hands-on loop test look like?

The reviewer built a small bench called round lab to test whether the loop really lives in the model. AREX-2 and base Qwen3.8 27B ran through the same loop with the same prompt, budget, and settings. Each round the model submits a full solution, it runs in a sandbox, and the model gets back its score, logs, and rounds remaining. The harness is the reviewer's own simplified one, not BAAI's.

The main task is a traveling salesman problem with 300 random points, scored against a strong reference solver. It is a good pick because improvement is visible. Variants included a stripped-down prompt with 12 attempts and no planning advice, a three-round versus 12-round budget to see if round-one planning changes, and a loose version of the skills ablation. The transcript I have cuts off before the results, so I am not reporting outcomes here. The design itself is worth copying if you want to test any model's loop behavior.