---
title: "How Does MoE-to-Dense Distillation Work? RAON-1 Nano"
url: "https://goremy.ai/blog/articles/moe-to-dense-distillation-raon-1-nano-explained"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-19T03:20:24.218Z"
updated: "2026-11-19T03:20:24.218Z"
topic: Local AI
description: "Krafton's RAON-1 Nano turns a 29B mixture-of-experts model into a 1.5B dense one. Here is the recipe, the 6.3-point gain over pruning, and what it means."
---

# How Does MoE-to-Dense Distillation Work? RAON-1 Nano

> Krafton's RAON-1 Nano turns a 29B mixture-of-experts model into a 1.5B dense one. Here is the recipe, the 6.3-point gain over pruning, and what it means.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-19

## How do you convert a mixture-of-experts model into a small dense model?

You score every expert in the big model to see which ones matter, select and group the useful ones, merge each group into a single ordinary dense feed-forward layer, and then train the new small model to imitate its parent. Krafton, the South Korean company behind PUBG, did this to turn its roughly 29 billion parameter RAON-1 Mini into the 1.5 billion parameter RAON-1 Nano. The method was written up in a paper that earned an oral slot at NeurIPS.

## TL;DR

- Krafton built **RAON-1 Nano (1.5B, dense)** from its own mixture-of-experts model, RAON-1 Mini (about 29B total, 4.3B active per token), rather than training a small model from scratch.
- The recipe has four steps: **score the experts**, group them, merge each group into a dense layer, then distill from the parent.
- The paper reports its method beating pruning a dense model to the same size by **6.3 points on average**, while training 1.6 times faster.
- The biggest factor out of 350 configurations tried was **how experts are scored**. Krafton's diversity-aware scoring keeps experts that do different jobs.
- The idea that this conversion sheds facts more easily than reasoning is **a plausible reading, not a tested result**. Neither the paper nor Krafton claims it.
- It matches Andrej Karpathy's "cognitive core" idea: a small model that reasons well and looks facts up instead of memorizing them.

## What is a mixture-of-experts model, and why convert it?

A mixture-of-experts (MoE) model splits much of its internal machinery into many specialist blocks called experts. For each word (token) it processes, a router picks a handful of experts and ignores the rest. So the model can hold a lot of total parameters while using only a fraction of them at any moment.

RAON-1 Mini is built this way. It has about 29 billion parameters in total, 128 experts, and sends each token through six of them, plus one shared expert. Only about 4.3 billion parameters are active for any one token. Krafton trained it from scratch, not as a fine-tune of someone else's model, and released it under Apache 2.0.

Why go smaller? Krafton ships AI characters in its games, including NPCs (non-player characters) driven by an on-device model. A model that runs on the player's machine, next to the game and sharing the same GPU, has to be small. Running a large model in the cloud for every conversation gets expensive fast.

## How does the conversion recipe work, step by step?

The paper's approach, as described in coverage of it, has four stages:

| Step | What happens | Why it matters |
|---|---|---|
| 1. Score | Each expert is rated on how much it matters | Decides what survives |
| 2. Select and group | Useful experts are chosen and clustered | Keeps variety instead of near-duplicates |
| 3. Merge | Each group is glued into one normal dense feed-forward layer | Removes the router and the sparse structure |
| 4. Distill | The new dense model is trained to match its MoE parent | Recovers quality lost in the merge |

Scoring is where the work is. The team tried 350 different configurations, and the choice that mattered most was how experts get scored. They developed a diversity-aware scoring method that tries to keep experts that do different things, so you don't end up with a pile of experts that all do the same job.

The result is compared against a more familiar approach: taking a dense model and pruning it down to the same size. The paper reports the MoE-to-dense route wins by 6.3 points on average across its benchmarks, and makes training 1.6 times faster.

## Why is a 20x size cut less extreme than it sounds?

Going from about 29 billion parameters to about 1.5 billion is roughly a 20x reduction. Distillation (training a small model to copy a big one) usually doesn't jump that far in one go. One large company, for example, reportedly stepped a model down from 7B to 1.7B and then to 600M, in stages.

But the 29B figure overstates the gap. Because only about 4.3 billion parameters fire for any token, the compute the parent actually uses per token is much closer to a 4B model. You can think of the parent as a roughly 4B model that rearranges itself for each token. Distilling that into 1.5B is a much more normal step.

## Is the paper's model the final RAON-1 Nano?

No. The paper's best student models are still below Qwen3 1.7B. To get to Nano, Krafton did more work after the conversion:

1. About 3 trillion more tokens of training.
2. Supervised fine-tuning.
3. On-policy distillation from Mini (the student learns from its parent's outputs on the student's own attempts).
4. Reinforcement learning.

The conversion is a strong starting point, not the finish line. Still, there is no outside base model anywhere in the family tree, which means Krafton knows what went into the model.

## Why would the model keep reasoning but drop knowledge?

This is the part to treat carefully. A common view is that much of a model's factual knowledge lives in its feed-forward layers, and in an MoE model those layers are the experts. When you convert, most experts are discarded, but the attention layers (which handle how the model relates tokens to each other) carry over.

So it is reasonable to guess that this recipe sheds stored facts more easily than it sheds the ability to work through a problem. But that is a guess. Krafton doesn't claim it, and the paper doesn't test it. Treat it as a hypothesis that fits the evidence below.

The evidence comes from the parent, Mini. On MMLU Pro, a broad general-knowledge test, Mini ranked last of the five models in its comparison table, at 76.5 against 85.9 for Qwen 3.6 36B. Yet it ranked first on an instruction-following benchmark. It also ranked first on the Artificial Analysis Omniscience benchmark, which penalizes wrong answers and rewards a model for saying it doesn't know. It is also weak at Terminal Bench, so it isn't built as a coding agent. That is a pattern of a model tuned to follow instructions and admit gaps, not to be a trivia machine. The reading that Krafton designed it this way is the video creator's interpretation, not a Krafton statement.

## What is Karpathy's "cognitive core," and how does this relate?

Andrej Karpathy has argued that big models are big partly because they memorize enormous amounts of knowledge. Strip out the memorized facts, keep the part that reasons and solves problems, and the model could be very small. He has suggested something around a billion parameters, which looks facts up when needed, essentially using retrieval (RAG, short for retrieval-augmented generation: fetching relevant documents and putting them in front of the model).

RAON-1 Nano isn't presented by Krafton as an attempt at that. But the overlap is clear. A game NPC needs to follow instructions, remember what you said 20 minutes ago, hold a tone, and use the game state it is handed (your inventory, your zone, where your squad is). It doesn't need to know the capital of the US. For that job, general world knowledge is dead weight.

## What does this mean for you?

Most readers won't convert a model. But the direction affects what you buy and build.

**If you're evaluating AI tools at work:** don't judge a small model by general-knowledge scores alone. A model that ranks low on MMLU Pro can rank high on following instructions and on knowing when it doesn't know. Match the benchmark to the job.

**If you're paying for AI at scale:** the economics here are why companies pursue small on-device models. Cloud inference per conversation adds up. Small models running locally shift the cost to hardware you already own.

**If you're building an internal assistant:** the cognitive core pattern is practical today. Keep your facts in documents you control, retrieve them at question time, and let the model reason over what it is shown. That makes answers easier to check and update than facts baked into model weights.

**If you're technical:** the model is open under Apache 2.0, and Mini has a free API for now and can run locally (a compressed version fits on a single 5090 card). Try it on your own task before trusting any summary, including this one.

## Can you build the "look it up, don't memorize it" pattern yourself?

If the cognitive core idea appeals to you, the practical version is a retrieval tool over your own documents: policies, product specs, past proposals. With [Remy](https://goremy.ai), you describe that app in plain English, such as an internal question-and-answer tool where staff sign in, ask a question, and get answers with cited sources from your uploaded files. Remy apps include managed document search with cited results and built-in sign-in, so you aren't stitching together a database and an AI service. It runs on models from Remy's catalog, so it is not a way to run RAON-1 Nano itself.

## Frequently Asked Questions

### What is RAON-1 Nano?
It is an open 1.5 billion parameter dense language model from Krafton's AI group. It was derived from the company's own mixture-of-experts model, RAON-1 Mini, through expert scoring, grouping, merging and distillation, followed by further training.

### Why does converting an MoE model beat pruning a dense model?
According to the paper, the MoE-to-dense method scored 6.3 points higher on average than pruning a dense model to the same size, and trained 1.6 times faster. The paper points to how experts are scored as the most important choice, and diversity-aware scoring keeps experts that do different things.

### Does the small model really lose knowledge but keep reasoning?
That is a hypothesis, not a proven result. Discarding most experts removes parts of the model where facts are thought to live, while attention layers carry over. Krafton doesn't claim this and the paper doesn't test it. Mini's benchmark pattern (weak on general knowledge, strong on instruction following) is consistent with it.

### Is RAON-1 Nano used in PUBG?
Krafton hasn't said so. Its chief AI officer describes Nano as a better base model for edge AI applications. Krafton already ships on-device AI characters and an AI teammate in its games, but a direct link to Nano has not been confirmed.

### Where was this research presented?
The paper earned an oral presentation at NeurIPS, a major AI research conference. An oral means the work was accepted and chosen for its own talk. This year's main event is in Sydney.
