--- title: "Distillation vs Reinforcement Learning: Can Qwen 4B Match 27B?" url: "https://goremy.ai/blog/articles/distillation-vs-reinforcement-learning-qwen-4b-27b" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-10-31T06:00:11.989Z" updated: "2026-10-31T06:00:11.989Z" topic: AI Model Reviews & Comparisons description: "Qwen 27B taught Qwen 4B three ways: teacher answers, probabilities, on-policy. All landed near 58% versus the 64% teacher, but one cut tokens 30%." --- # Distillation vs Reinforcement Learning: Can Qwen 4B Match 27B? > Qwen 27B taught Qwen 4B three ways: teacher answers, probabilities, on-policy. All landed near 58% versus the 64% teacher, but one cut tokens 30%. Edited by Luis Chavez-Mattos, Director, Product ยท Published 2026-10-31 ## Can distillation push a 4B model past the reinforcement learning ceiling? Not in this experiment. A Qwen 4B model that reinforcement learning (RL) had lifted to about 58% on a 394-task data-analysis exam stayed in the 56% to 59% band after three different distillation methods from a Qwen 27B teacher, which scores about 64%. Distillation did change how the small model behaves. Training on the teacher's answers cut token usage by roughly 30% with no accuracy loss. It did not raise the ceiling. ## TL;DR - **Every route plateaued** between 56% and 59%, against a 27B teacher at about 64% and a 4B starting point near 52%. - **Training on teacher answers** kept accuracy at about 58% while dropping average transcripts from roughly 2,000 tokens to roughly 1,400. - **Probability matching** got the student to agree with the teacher's top token 96% of the time, yet the exam score was about 56%, a statistical tie with RL. - **On-policy distillation** peaked at 58.9% at step 20 and was back near 58% by step 60, while changing behavior on individual tasks. - **A confident teacher makes probability distillation pointless**, because its written answers already carry nearly everything its probabilities hold. Check that before you save gigabytes of logits. ## What was the experiment setup? The tasks come from an open Hugging Face set called small data environments. Each task is a real Kaggle dataset with a question and a verifiable answer. The model works in a locked-down sandbox with a terminal, inspects CSV files, writes Python, and saves a final answer to a file. A strict checker compares that answer to the key. The held-out exam has 394 tasks, 87 of them on datasets the model never saw in training. The hardware was two Nvidia DGX Sparks: one hosted the 27B teacher, the other trained the 4B student. Every distillation run started from the best RL checkpoint, which scored 58.4%. Results were compared task by task on the same exam. ## Why did reinforcement learning stall in the first place? The RL method was GRPO. The model attempts each task several times, and attempts that beat the group average get reinforced while worse ones get pushed down. That lifted the 4B model from about 52% to about 58%, closing more than half the gap to the 27B model. Then it stopped. The cause is structural. If a model fails a hard task on every attempt, every attempt gets zero reward, none beats the average, and the task teaches nothing. The same is true for tasks it solves every time. Once RL had sharpened everything the model sometimes got right, the never-solved tasks gave no signal. Distillation is supposed to fix that by bringing in a teacher that already solves more of those hard tasks. ## What are the three ways to distill a model? | Method | What the student learns from | Result on the exam | |---|---|---| | Teacher answers | Fine-tuning on the teacher's correct transcripts | About 58% | | Teacher probabilities | Matching the teacher's token distribution on those transcripts | About 56% | | On-policy distillation | Teacher grades every token of the student's own attempts | Peak 58.9%, about 58% at the end | The probability method only works when both models share a tokenizer. Qwen's 4B and 27B models do, so it was possible here. The idea traces back to a 2015 Geoffrey Hinton paper that trained small models on a big model's probabilities rather than just its final answers. To see what probabilities add, take one real moment from the 27B model. After typing a print call on a dataframe, it gave 59% to `.head`, 36% to `.shape`, and 4% to `.info`. Only `.head` appears in the transcript. The other choices are lost in plain text. ## Does training on teacher answers help? It helps efficiency, not the score. The teacher solved 2,000 training tasks, and about 1,400 transcripts passed the strict checker. The student was fine-tuned on those. The exam score stayed around 58%, a tie with the RL model. A further round of RL on top did not move the overall number either, though it improved results on the unseen-dataset subset. The behavior shift was large. The RL model had learned by trial and error and wrote messy transcripts averaging about 2,000 tokens per task. After training on teacher answers, the student averaged about 1,400 tokens, almost identical to the teacher. That is roughly 30% shorter, and so faster and cheaper to run, without losing accuracy. ## Does matching the teacher's probabilities add anything? Not here. The teacher went back over all 1,400 winning transcripts, nearly 5 million tokens, and saved its top 20 token choices and their probabilities at every step. The student trained to match them and ended up agreeing with the teacher's top token 96% of the time. The exam score was about 56%, still a statistical tie with RL. The reason showed up when the creator inspected the teacher's probabilities across more than 4,000 tokens: - On about three quarters of tokens, the teacher put at least 99% of its probability on one choice. - Its top choice matched the transcript word 96% of the time. - Only about 2.6% of tokens were genuinely uncertain, and those were mostly harmless wording splits, such as "data" versus "dataset" at 50/50. When a code-writing teacher is this sure of itself, the transcript already contains almost everything the probabilities know. A cheap check on a small sample tells you whether saving full distributions is worth it. ## How does on-policy distillation work, and did it break the ceiling? The first two methods share a weakness. The student only sees clean teacher transcripts. When it takes the exam alone and makes an early mistake, it lands in situations the teacher never demonstrated. On-policy distillation addresses that. The student attempts the task itself in the sandbox, then the teacher scores every token the student wrote. In this setup, one Spark ran the student's attempts and sent them to the other Spark, where the 27B teacher graded them. This also targets the RL wall directly. A failed attempt that earns RL a flat zero still gets per-token feedback. In one example, the student was working on Chipotle order data and computed an average order value of 18.81 instead of 21.39. The teacher's disagreement spiked on the `.sum` call in the line where the student added up orders incorrectly. Overall the teacher flagged only two or three tokens in every hundred. Thinking Machines has argued that this newer form of distillation can beat RL for a fraction of the compute, and the Qwen technical report reports on-policy distillation beating RL on math for about a tenth of the GPU hours. On this task, the gap between student and teacher dropped quickly over the first 20 steps and then flattened. The student scored 58.9% at step 20, the best of any small model in the project, but that is statistically a tie with RL. By step 60 it was back near 58%. ## What changed in the model's behavior? Scores matched, but individual tasks did not. - **Iris dataset question:** The RL model and the answer-distilled model both wrote "50 50 50," which fails a checker expecting a single number. The on-policy model wrote "50." - **El Nino temperature question:** The on-policy model kept double-checking its work and used all 12 turns without writing an answer. The 27B teacher finished in seven steps. So longer on-policy training fixed some formatting failures while introducing over-cautious behavior that cost easy tasks. The source material cuts off before the creator's final recommendations on running it, so treat those details as unconfirmed. ## Why do all the methods hit the same ceiling? The creator's explanation points to what the small model actually knows, rather than which training signal you use. When four different routes converge on one band, the likely limit is capacity or knowledge in the 4B model, not the algorithm. The 6x gap to the teacher (about 58% versus 64%) may be partly a ceiling on what a 4B model can represent for this task family. The sources do not prove that, so read it as a working hypothesis. ## Is distillation worth it for a small model? It depends on what you want. If you want a higher exam score than RL gave you, this experiment says no. If you want a cheaper model, training on teacher answers delivered about 30% fewer tokens at equal accuracy, which is a real deployment win. On-policy distillation gave a brief peak and behavioral fixes but needs two machines' worth of orchestration and careful stopping. ## Frequently Asked Questions ### Can a 27B teacher make a 4B student as good as itself? In this test, no. The teacher scored about 64% and the best student result was 58.9% at one checkpoint, with most runs between 56% and 59%. The student closed part of the gap from its roughly 52% baseline, but most of that gain came from RL before distillation began. ### Why does RL get stuck on hard tasks? GRPO rewards attempts relative to the group average. If every attempt at a task fails, all get zero and none stands out, so there is no gradient signal. Tasks solved every time give no signal either. Only tasks the model sometimes solves teach it anything. ### When is probability distillation worth the effort? When the teacher is often uncertain between meaningfully different choices. Here the 27B teacher put at least 99% on one token for about three quarters of its output and was unsure on only about 2.6% of tokens, mostly over trivial wording. Measure that on a small sample before generating full distributions. ### Do the teacher and student need the same tokenizer? For probability-based distillation, yes, because the student must match the teacher's distribution over identical token pieces. Qwen's 4B and 27B models share a tokenizer, which is why this approach was possible. Training on teacher answers has no such requirement.