--- title: "Did GPT-6 Astra Really Make Qwen 6x Better at Math?" url: "https://goremy.ai/blog/articles/gpt-6-astra-weight-space-programming-qwen-math" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-11-06T13:20:06.764Z" updated: "2026-11-09T18:54:08.583Z" topic: Local AI description: "An MIT-linked paper says GPT-6 Astra edited 0.016% of Qwen3.5-0.8B's weights for a sixfold math gain. Here is what changed and what to doubt." --- # Did GPT-6 Astra Really Make Qwen 6x Better at Math? > An MIT-linked paper says GPT-6 Astra edited 0.016% of Qwen3.5-0.8B's weights for a sixfold math gain. Here is what changed and what to doubt. Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-06 · Updated 2026-11-09 ## Did GPT-6 Astra really make Qwen 6x better at math? Partly. A frontier model called Astra edited two tensors in Qwen3.5-0.8B, and MATH-500 accuracy rose from 7.9% to 48.1%, roughly sixfold. But the gain came mostly from stopping the model from looping until it hit the token limit. The edit removed a blocker. It did not teach new math. The paper calls the broader idea "weight space programming." ## TL;DR - Astra changed **131,072 weights, or 0.016% of the model**, by subtracting one direction from the `down_proj` matrices in layers 15 and 19. - The base model hit the token limit on **93.0% of problems**, and the edit cut that to 33.1%. That drop explains most of the gain. - Reported scores went from 7.9% to 48.1% on MATH-500, 13.8% to 67.1% on GSM8K, and 21.0% to 62.3% on IFEval. - An **independent re-run** on MATH-500 measured 51.0% for the edited checkpoint, so the result reproduces. - The paper's setup (thinking mode, lower temperature than recommended) is **the condition Qwen warns about**, so the baseline is unusually weak. - Astra also read weights to identify games and wrote Atari-style networks from all-zero weights, which is a separate and arguably more novel result. ## What is weight space programming? Every neural network is a long list of numbers, its weights. Training nudges them in small steps from a random start. Weight space programming asks whether a frontier LLM can read and write those numbers directly. You give the model a goal in plain English plus context, and it edits the weights with no gradient descent and no training. Astra did not work blind in the Qwen case. It had a Python session over the weights and a budget of 50 forward passes on practice problems from the GSM8K and MATH training sets. Those runs gave it feedback. The process looks like a search: a strong model proposes an edit, scores it, and keeps what works. ## Can a model read weights and tell what a network does? On small networks, yes. The researchers trained a small network on each of four games, all with identical shape. They pasted every weight into the prompt as text and asked Astra which game each one was trained on. No tools, no list of candidates. Astra named the right game every time. For Breakout, it noticed the player weights were strongest along the bottom row, where a paddle sits, and a scoring weight clustered near the top, where the bricks are. One caveat: the prompts told Astra what each input meant, so it knew where to look. ## Can it write a network from zeroed weights? For three small Atari-style games, Astra got the rules, the game's source code, and a network with every weight set to zero. It had to write all the numbers in one reply, with no training and no tools. In Breakout, the result scored many times higher than the best network trained with reinforcement learning. The method matters more than the score. Astra wrote a small program: the first layer predicts where the ball will land, and the second layer moves the paddle there. Then it compiled that program into weights. Access to the source code likely helped a lot, since the reinforcement learning network had to infer the rules from score alone. And in SeaQuest the trained network still won. Knowing the rules helps, but it does not guarantee a better player. ## What did Astra actually change inside Qwen? The target was Qwen3.5-0.8B. Astra noticed the model rarely finished its answers. It started reasoning, then repeated the same lines until the token limit. It compared internal activations on answers that made progress against answers that looped, and found a direction (a specific mix of internal values that shows up together) that appeared just before repetition. It then traced that direction to the MLPs of layers 15 and 19 and subtracted it from 64 output columns of `down_proj` in each layer. Removing a direction is a known interpretability technique. What was new is that Astra chose where to apply it by itself. | Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit | |---|---|---|---|---| | Base | 7.9% | 13.8% | 21.0% | 93.0% | | Edited | 48.1% | 67.1% | 62.3% | 33.1% | These figures use thinking mode, up to 32,768 tokens, and several samples per problem. ## Is the sixfold gain real? The measurement is real. A re-run of the published checkpoint covered all 500 MATH-500 problems with 4 samples each (2,000 generations) using vLLM on an H200. It scored 51.0%, with 31.6% of answers hitting the limit. The edit is also published as a bit-exact diff, and only two tensors differ from the base model. The interpretation is where caution belongs. The "six times better" framing suggests new ability. The logs say otherwise. ## Where did the gain come from? From finishing answers. Across more than 1,000 practice problems, almost every answer the edit rescued had previously hit the token limit. When the original model did finish, it was already right almost every time. On the real benchmark, the original hit the limit on nearly every problem, and after the edit that fell to about a third. So the model could already solve many of these problems. It just never got to say the answer. Qwen's own model card matters here. Qwen runs this model in non-thinking mode by default and warns that in thinking mode it is more prone to loops than other Qwen3.5 models, because it is so small. The paper ran every test in thinking mode, and with a lower temperature than Qwen recommends. Separate research has found reasoning models loop more at lower temperatures. The baseline was set up to fail in exactly the way the edit fixes. ## What should skeptics ask before trusting a result like this? 1. **What is the baseline?** Check whether the starting model was run the way its makers recommend. Here it was not. 2. **Where did the improvement come from?** Count how many answers never reach a final answer. If most are cut off, the problem is looping, not reasoning. 3. **Is it repeatable?** The experiments were run once, and the improvement came from problems where the model could finish. A different small model that loops, a Gemma model, saw its math score barely move after similar edits. That suggests editing can remove an obstacle hiding an ability but cannot add a missing one. Also note the feedback loop. Astra did not use gradients, but it did iterate against practice problems, which is a form of optimization. ## What does this mean for people building with small models? Two practical rules follow. First, when a small reasoning model scores badly, measure the share of truncated outputs before changing anything. The fix may be the mode, the temperature, or a larger output limit, not a weight edit. Second, do not expect weight editing to make a weak model smarter. In this paper it removed a blocker and added no skill. The more durable finding may be interpretability work: a frontier model can explore a small network, find which parts drive a behavior, and propose a targeted fix. ## Frequently Asked Questions ### How much of Qwen did Astra change? Astra edited 131,072 weights, which is 0.016% of Qwen3.5-0.8B. The changes sit in the `down_proj` matrices of layers 15 and 19. Only those two tensors differ from the original model. ### Did the edit involve any training? No gradient descent or training took place. Astra had a budget of 50 forward passes on practice problems and used the scores as feedback. That makes the process closer to a guided search than to training. ### Does the edited model reason better? Mostly it finishes more often. The share of answers hitting the token limit fell from 93.0% to 33.1%. Problems the base model completed were already answered correctly almost every time. ### Can I try the edited model? A checkpoint of the edited model is published on Hugging Face, along with the diff and rebuild code. The listed evaluation used vLLM, thinking mode, temperature 0.6, and a 32,768-token output limit. ### Does this work on other small models? Not clearly. A small Gemma model that also loops saw its math score barely change after similar edits. The authors conclude that editing can remove an obstacle but cannot add a missing ability.