How to Distill a Small LLM: Tokenizers, Pass@4, and Pitfalls
A practical guide to distilling a large LLM into a small one: the four-tries headroom test, tokenizer matching, vLLM memory limits, and when to stop training.
How should you set up and sequence distillation for a small local model?
Run cheap checks first, then escalate. Test whether the student has headroom by giving it four tries per task. Confirm teacher and student share a tokenizer. Fine-tune on the teacher's correct transcripts. Only save teacher probabilities if the teacher is uncertain. Run on-policy distillation last, and stop when the teacher-student gap flattens. These steps come from a 4B student and 27B teacher experiment on two DGX Sparks.
TL;DR
- Run a four-tries headroom test before you spend a day of GPU time, because tasks the student never solves are where sampling-based methods stall.
- Identical tokenizers are a hard requirement for probability-based distillation. Models from the same family (here, a 4B and a 27B from one Qwen family) usually qualify.
- Plain fine-tuning on the teacher's winning transcripts held accuracy at about 58% and cut the student's transcripts from roughly 2,000 to 1,400 tokens per task, about 30% shorter.
- Check teacher confidence on a small sample before saving probabilities. A teacher that is nearly certain on most tokens adds little beyond its text.
- On-policy distillation gave per-token feedback even on failed attempts, but its best score came early (step 20), and longer training made behavior worse on easy tasks.
- Cap vLLM memory and tune chunk size so teacher and student coexist without out-of-memory crashes. One run used 0.45 and 248-token chunks.
What is distillation, and why use it after reinforcement learning stalls?
Distillation trains a small student model to imitate a larger teacher. The setup here was a 4B student and a 27B teacher from the same family, tested on a held-out exam of 394 data-analysis tasks. The model works in a sandbox with a terminal, inspects CSV files, writes Python, and submits an answer a strict checker verifies.
Before training, the student scored about half. The teacher scored about 64%. GRPO reinforcement learning lifted the student to roughly 58%, then stopped helping. The cause is structural. If a model tries a task several times and fails every time, every attempt gets zero reward, none beats the group average, and nothing is reinforced. Tasks it always solves teach nothing either. A teacher can show the student what to do on tasks it never gets right.
What are the three ways to distill a model?
| Method | What the student learns from | Tokenizer match needed? | Result in this test |
|---|---|---|---|
| Teacher answers (SFT) | Teacher transcripts that passed the checker | No | About 58%, much shorter transcripts |
| Teacher probabilities | Teacher's top-20 token distribution on those transcripts | Yes | About 56%, a statistical tie |
| On-policy distillation | Teacher grades every token of the student's own attempt | Yes | Peak 58.9% at step 20, back near 58% by step 60 |
Every route landed within noise of the RL model's 58.4%. The methods still changed how the student behaves, which matters if you care about cost per task and not only the score.
How do you check headroom with the four-tries test?
Before training, run each task in your evaluation set four times with the current student and record whether any attempt succeeds. This is a pass@4 style check, and it takes minutes.
Read the result as a ceiling estimate for methods that learn from the student's own samples. Tasks solved at least once are ones that RL or similar sampling-based training can sharpen. Tasks that fail all four attempts give those methods no signal. If most of your remaining errors sit in that second group, more RL will not help, and you need a source of new information, such as a teacher.
Treat four as a convention, not a law. Use a number of attempts that matches how many samples your training loop would draw per task. The test is cheap enough to repeat after each stage.
Do the tokenizers have to match?
For probability-based and on-policy distillation, yes. The student learns to match the teacher's distribution over next tokens at each position. That comparison only makes sense if both models split text into the same token pieces with the same vocabulary. If they differ, position-by-position probabilities do not line up.
Check this before you generate anything:
- Load both tokenizers and compare vocabulary size and special tokens.
- Encode a few hundred real transcripts with each and confirm the token IDs are identical.
- If they differ, fall back to training on the teacher's text, which does not require a match.
Models from the same family often share a tokenizer, as the 4B and 27B did here. Do not assume it across families or even across every size in one family. Verify it.
Is saving teacher probabilities worth it?
Often not, and you can find out cheaply. In the experiment, the teacher revisited about 1,400 winning transcripts (nearly 5 million tokens) and saved its top 20 token choices at each step. The student matched the teacher's top choice 96% of the time by the end. On the exam it scored about 56%, no better than plain fine-tuning.
The reason showed up when the teacher's probabilities were inspected over more than 4,000 tokens. On about three-quarters of tokens, the teacher put over 99% on a single choice. Its top choice matched the transcript's actual token 96% of the time. In the 2.6% of tokens where it was unsure, it was mostly weighing harmless wording, such as "data" versus "dataset."
A code-writing teacher that confident has already put nearly everything it knows into its text. So sample a few thousand tokens first and measure how often the top probability is above 0.99 and how often the top choice equals the written token. If both numbers are high, skip the gigabytes of stored distributions.
What did plain fine-tuning on teacher answers change?
The student's accuracy stayed near 58%, but its behavior shifted. The RL-trained model had learned by trial and error and wrote long, messy transcripts averaging about 2,000 tokens per task. After fine-tuning on the teacher's 1,400 passing transcripts (filtered from 2,000 teacher attempts), it dropped to about 1,400 tokens, close to the teacher's own length. That made it roughly 30% shorter and faster to run with no accuracy loss. If your goal is cheaper inference, this simple step may be the main win.
How does on-policy distillation work, and when should you stop?
The student attempts the task itself. The teacher then scores every token of that attempt. Unlike RL, a failed run is not a flat zero. In one example, the teacher's strongest disagreement landed on a .sum call in the line where the student added up order values incorrectly. The teacher agreed with most of the script and flagged only two or three tokens in every hundred.
Training behavior was clear. The student-teacher gap fell quickly over the first 20 steps, then flattened. Step 20 gave 58.9%, the best of any small model in the project, though still a statistical tie with RL. By step 60 the score was back near 58%. Longer training also hurt some easy tasks. On a simple temperature-data question, the over-trained model kept double-checking its work and used up its 12-turn limit without answering, while the teacher finished in seven steps.
So watch the gap curve and evaluate checkpoints early. When the curve flattens, stop. More steps bought nothing measurable and introduced new failure modes.
How do you keep two models from crashing your GPUs?
The setup used two DGX Sparks. One ran the 27B teacher. The other ran the 4B student, which attempted tasks in the sandbox and sent its attempts across a high-speed link for scoring. If you serve models with vLLM, memory is the usual failure point, because vLLM reserves a fraction of GPU memory up front and leaves less for training or scoring.
The runs used a memory fraction of 0.45 and 248-token chunks. In vLLM that fraction is the gpu_memory_utilization setting:
vllm serve <model> --gpu-memory-utilization 0.45
Treat 0.45 and 248 as starting points from one hardware setup, not universal values. Lower the fraction if training steps run out of memory. Raise it if generation is starved. Change chunk size the same way, and rerun a short smoke test before committing to a long job.
Frequently Asked Questions
Do I need a teacher from the same model family?
Not for training on the teacher's text, which works across families. For probability-based or on-policy distillation you need identical tokenizers, which same-family models usually have. Verify by encoding sample transcripts with both.
Will distillation beat reinforcement learning on my task?
Not necessarily. In this test every method tied the RL model's score within noise. Distillation changed behavior, such as shorter transcripts and per-token feedback on failures, so judge it on cost and reliability as well as accuracy.
How long should I run on-policy distillation?
Only until the gap between student and teacher flattens. Here that took about 20 steps. The best checkpoint came at step 20, and by step 60 the score had slipped while easy-task behavior worsened.
What does the four-tries test tell me?
It shows how many tasks the student solves at least once in four attempts. Those tasks can be improved by sampling-based training. Tasks that fail every attempt need outside information, such as a teacher's demonstrations or token-level grading.
How we source, write, and correct our articles. Editorial standards
