How to Run the Edited Qwen3.5 0.8B Locally With vLLM

How do you run the edited Qwen3.5 0.8B locally?

Install vLLM in a virtual environment, download both the Hugging Face rebuilt edited model and the original Qwen3.5 0.8B, and serve each on its own port (for example 8000 and 8001). Then send both the same prompts through the OpenAI-compatible API. In one demonstrated setup on a single 48 GB RTX card, both models together used just under 14 GB of VRAM. The edited model finished the test prompts. The original looped.

What is weight-space programming?

A model's weights are just numbers. The usual way to change behavior is to train, which costs time and compute. Weight-space programming takes a different route. A system reads the raw weights, works out which ones drive an unwanted behavior, and edits a small number of them directly. No gradient training is involved.

The behavior targeted here is looping. Small reasoning models sometimes keep reasoning, double-check, and re-derive until they exhaust their token budget without ever committing to an answer. That is a practical failure for anyone building agents or pipelines, because a request that never terminates blocks a slot and burns compute.

How was the edited Qwen3.5 0.8B made?

According to the source, an AI system called Astra edited around 131,000 weights in two layers of the Qwen3.5 0.8B model to reduce looping. The original authors released the edit as an exact set of changes, but not the finished model.

Hugging Face then rebuilt it. An AI assistant applied the published edit to the base weights, checked that every changed tensor matched by hash, ran a benchmark, and published the result. Keep that distinction in mind. What you download is Hugging Face's reproduction, not the authors' own files.

How do you serve both models side by side with vLLM?

The setup in the demonstration was an Ubuntu machine with one NVIDIA RTX card with 48 GB of VRAM. The steps:

1. Create and activate a virtual environment.
2. Install vLLM.
3. Download the edited model and the original Qwen3.5 0.8B from Hugging Face.
4. Start one vLLM server per model on different ports.

python -m venv .venv
source .venv/bin/activate
pip install vllm

# Replace the placeholders with the actual Hugging Face repo IDs
vllm serve <edited-model-repo> --port 8000 --gpu-memory-utilization 0.4
vllm serve <original-qwen3.5-0.8b-repo> --port 8001 --gpu-memory-utilization 0.4

The --gpu-memory-utilization flag matters when two servers share a card. Each vLLM instance reserves a fraction of VRAM by default, so the second server can fail to start if the first claims most of it. Splitting the budget explicitly avoids that. Also, a 0.8B model is small, so you do not need a 48 GB card. The demo's combined usage was under 14 GB.

The demo showed warnings about CUDA and related libraries during install, which the creator ignored because the environment was already working. Expect that kind of noise, and check that the servers actually come up before testing.

Qwen3.5 support depends on your vLLM version. If the model fails to load, upgrading vLLM is the first thing to try.

Does the edited model actually stop looping?

Three tests were run. Here is what each showed.

The 2^100 mod 7 prompt

Both models were asked the same question: what is the remainder when 2 to the power of 100 is divided by 7? Both reached the correct answer along the way. The difference was what happened next. The original wrote out one power of two after another, past 90, tripped over a cycle it had already found, and kept second-guessing itself with "let me double check" until the context window ran out. The edited model finished and answered.

This is a useful test because it isolates the failure. Correctness was not the issue. Termination was.

The 12-problem set

The second script sent 12 math questions, ranging from simple arithmetic to prime finding and greatest common divisor. All 12 came back correct in around 48 seconds, and none exhausted its token budget. That is a small sample, so treat it as a smoke test, not a benchmark.

The MATH-500 batch

The third test pulled 50 problems from MATH-500, a well-known set of competition-style math questions. All were sent at once, each with room for up to 32,000 tokens of thinking. When shown, the run was progressing without loops or breakdowns. No final accuracy number was given, so this test shows stability under load, not a verified score.

How can you reproduce the comparison yourself?

Use the same prompt against both endpoints and cap output length so a loop ends in a measurable way.

• Send the 2^100 mod 7 question to ports 8000 and 8001 with streaming on, and watch which one terminates.
• Set the same max_tokens for both servers. Record whether each response ends with a natural stop or a length cutoff. In the OpenAI-compatible API, that is the finish_reason field.
• Run your own batch of math problems concurrently. vLLM handles many simultaneous requests well, which is why the 50-problem run could be sent all at once.
• Count length cutoffs, not just wrong answers. A loop is a different failure from a wrong answer.

Is the edited model worth using?

For experimentation, yes. It is a cheap, small model that demonstrates a technique, and the side-by-side test is easy to run. The results shown are promising, but the evidence is thin: one prompt, 12 problems, and an unfinished-in-frame batch of 50. The field is also new, and the creator expected no major changes quickly.

Where it could matter is long-horizon work, such as agent loops, where a model that terminates reliably is worth more than one that reasons forever. The creator suggested combining it with a harness to fix looping in such setups. That is a reasonable hypothesis, not a demonstrated result.

Two caveats. This is a reproduction by Hugging Face, not the authors' original files. And a 0.8B model has limits on hard problems regardless of whether it loops, so measure accuracy on your own tasks before relying on it.