--- title: "How Did Nvidia Build a 2-Bit 74B Model From a 120B One?" url: "https://goremy.ai/blog/articles/how-nvidia-built-2-bit-model-pruning-distillation-qat" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-11-10T18:20:07.856Z" updated: "2026-11-10T18:20:15.494Z" topic: AI Models & Industry description: Nvidia shrank a 120B model to a 74B model with 2-bit experts that fits in 24 GB. Here is the pruning, distillation and QAD recipe, and why it matters. --- # How Did Nvidia Build a 2-Bit 74B Model From a 120B One? > Nvidia shrank a 120B model to a 74B model with 2-bit experts that fits in 24 GB. Here is the pruning, distillation and QAD recipe, and why it matters. Edited by Luis Chavez-Mattos, Director, Product ยท Published 2026-11-10 ## How did Nvidia compress a 120B model into a 2-bit 74B model? Nvidia did it in three stages, and quantization (rounding weights to fewer bits) came last. First it pruned the 120-billion-parameter Nemotron 3.5 Super down to 74 billion parameters using a method called Star Elastic, sized from the start to fit a 24 GB memory budget. Then it distilled and post-trained the pruned model, including reinforcement learning for agent tasks. Finally it used quantization-aware distillation (QAD) to teach the model to work with 2-bit experts. The result is Nemotron 3.5 Bolt 74B A9B. ## TL;DR - Nvidia's **Nemotron 3.5 Bolt** has 74 billion total parameters and about 9 billion active per token, with the expert layers stored at 2 bits so the model fits in 24 GB. - The recipe is **prune, distill, post-train, then quantize**. The 2-bit step is the last of several, not the trick on its own. - Rounding a finished model to 2 bits after training usually hurts badly. **Training the model to live at low bits** is what keeps the quality. - It is a **mixed precision** model, not 2-bit everywhere. Some layers are 4-bit and the embeddings and output layer are 8-bit. - Nvidia's numbers show a small quality drop (roughly 90 to 88 on one agent benchmark, about 89 to 86 on GPQA Diamond) for a file that shrinks from 148 GB to 22 GB. - It is **smarter per gigabyte, not faster**. Nvidia's own table has it near 84 tokens per second on RTX Spark laptops, behind some similar-sized rivals. ## Why does normal 4-bit quantization break down at 2 bits? Most people who run models locally pick a quantized file, such as a Q4 or Q8 GGUF. These are post-training quantizations. The model was trained at 16-bit precision, and someone rounded every weight to fewer bits afterward, usually using a small calibration dataset to keep the rounding errors from damaging the outputs. At 8 bits, this costs almost nothing. At 4 bits, most models stay close to the original, which is why 4-bit became the default. Below that, things fall apart. The sharp drop in quality tends to appear between 3 bits and 2 bits. At 2 bits the model's internal representations need to change substantially, and plain rounding gives it no chance to adapt. That is why community 2-bit and 1-bit files are often rough. A Meta paper called ParetoQ made the key point Nvidia is betting on. If you train properly for low bits, 2-bit and 3-bit models can beat post-training quantized 4-bit models at the same memory budget. A bigger model with fewer bits per weight can win over a smaller model with more bits. ## What is Nemotron 3.5 Bolt, exactly? Nemotron 3.5 Bolt 74B A9B is a mixture-of-experts model. "74B" is the total parameter count. "A9B" means about 9 billion parameters are active for any given token. It uses Nvidia's hybrid Mamba-transformer design, accepts images and video, and supports up to a 256K context window. Microsoft showed it at the launch event for its RTX Spark laptops, which have 24 GB of memory. The "2-bit" label needs a caveat. Only the experts are at 2 bits. The Mamba layers use NVFP4 (a 4-bit format), and the embeddings and output layer are 8-bit. Nvidia is open about this, and it makes sense: each layer gets the precision it can tolerate. Treat it as a mixed precision model. ## Stage one: how does Star Elastic pruning work? Nvidia started from Nemotron 3.5 Super, which has 120 billion parameters with 12 billion active. Star Elastic works like an architecture search with a memory budget. A smaller router learns, layer by layer, how many experts to keep and how wide each expert block should be. Nvidia also cut the number of experts active per token from 22 to 9. The budget is the important part. Nvidia did not pick 74 billion as a round target. The constraint was fitting under 24 GB at a batch size of one with a 128K context window. That budget already assumed the experts would end up at 2 bits, with the Mamba state staying at 16 bits. So the pruning step knew about the quantization coming later. After pruning, the model recovered through distillation from its parent, the larger model acting as teacher. This used about 120 billion tokens in total. Most of it ran at roughly 8K context because that is faster, with the final chunks at 128K so long-context ability came back. ## Stage two: what happens in post-training? This stage is conventional. Nvidia reattached the vision encoder, which had been separated earlier, ran more supervised fine-tuning, merged checkpoints, and ran several rounds of reinforcement learning aimed at agent tasks. The model was still in BF16 (16-bit) at this point, so all of this work happened at full precision. ## Stage three: what is quantization-aware distillation? Quantization-aware training (QAT) simulates low-bit rounding on every forward pass during training, while keeping a full-precision copy of the weights for updates. The model learns to perform well on the coarse grid of values it will actually use, rather than being rounded after the fact. QAT is becoming common. Google released QAT versions of Gemma 3 at Q4, and in June released QAT checkpoints across the Gemma 4 family. For the two small mobile models, E2B and E4B, token generation layers were pushed to 2 bits while core reasoning layers kept higher precision, getting E2B under a gigabyte. OpenAI did similar work for its GPT-OSS models, which is why the 120B version fits on a single 80 GB GPU. Moonshot has done something comparable for Kimi. Big labs now plan for quantization during training instead of leaving it to the community. Nvidia adds a twist: quantization-aware distillation. The quantized model is the student, and the unquantized model is the teacher. The advantage is practical. All the earlier work (fine-tuning, merging, reinforcement learning) happens at full precision, and distillation at the end transfers the result to the low-bit version without needing to rerun the whole pipeline or reuse all the original training data. ## What did the compression cost in quality? Going from the BF16 version to the 2-bit one, Nvidia reports PinchBench (an agent benchmark) dropping from about 90 to 88, and GPQA Diamond, a hard science question set, from about 89 to 86. Those are real losses. But the full-precision model is 148 GB and the compressed one is 22 GB. A few points of quality for a file roughly one-seventh the size is a trade many people will take. | | Full precision | 2-bit mixed precision | |---|---|---| | File size | 148 GB | 22 GB | | PinchBench | about 90 | about 88 | | GPQA Diamond | about 89 | about 86 | These are Nvidia's figures. Third parties have a hard time reproducing this approach, because it works best when the model's creator does the low-bit training. Independent testing will matter. ## Is a lower-bit model actually faster? Partly. Local use usually means one request at a time, so you are limited by how fast weights stream through memory, not by raw compute. Nvidia's model card says kernel latency at low batch sizes improves roughly in line with how much you compress the weights. Fewer bytes to move means more tokens per second. In practice, Nvidia's own table puts Bolt near 84 tokens per second on RTX Spark, behind Qwen 3.6 and Gemma 4 models. One reviewer measured about 220 tokens per second on a ZGX Fury workstation, faster in absolute terms but slower than the larger full-precision sibling would suggest was possible. The point of Bolt is capability per gigabyte, plus leftover memory for the KV cache, an embedding model, a browser or an IDE running alongside. ## What are REAP and BitNet, and how do they fit in? Two related techniques show where this field is heading. **REAP (expert pruning).** Cerebras's REAP drops whole experts from a mixture-of-experts model instead of shrinking everything. It found that pruning experts works better than merging them, because merging breaks the router's ability to choose. The catch is that which experts survive depends on your calibration data. Calibrate on code and you get a model that stays good at code and likely gets worse at creative writing. **BitNet.** Microsoft's BitNet trains a model from the first step with weights limited to -1, 0 or 1, the true 1.58-bit model. It works, but GPUs lack proper hardware support for it today. Nvidia's route (train big at 16-bit, prune, distill, then distill to low bits) is the more realistic path for now. ## What does this mean for you? If you only use hosted chatbots, nothing changes today. But the direction matters for anyone who cares about privacy, cost or offline work. - **If your company restricts sending data to outside services:** models of this capability on a 24 GB machine make local agents more realistic. Ask your IT team what hardware refresh plans look like. - **If you pick local models:** the old rule (biggest model that fits at 4-bit) is no longer the whole story. Check whether the lab released its own low-bit version, trained for it, rather than grabbing a community quant. - **If you evaluate AI tools:** ask whether a quality claim comes from the model's maker or from independent tests, and whether the model is mixed precision. - **If you care about speed:** do not assume smaller files mean faster answers. Check tokens per second on your hardware. Test on your own work. Benchmarks like GPQA Diamond say little about whether a model handles your contracts, spreadsheets or support tickets. ## Can you build a tool to compare models on your own tasks? Yes, and it is a good use of a small internal app. A team weighing a local model against a hosted one could build a simple tracker where people paste in real tasks, record each model's answer, and score quality, speed and cost side by side. With [Remy](https://goremy.ai), you describe that tool in plain English and it builds and deploys a full-stack app with a database and sign-in, so the results live in one shared place instead of scattered spreadsheets. Remy offers close to 400 AI models inside the apps it builds, which can serve as the hosted side of the comparison. ## Frequently Asked Questions ### Is Nemotron 3.5 Bolt really a 2-bit model? Only partly. The expert layers are stored at 2 bits, but the Mamba layers use NVFP4 (4-bit) and the embeddings and output layer are 8-bit. It is a mixed precision model, and Nvidia states this openly. ### What is the difference between QAT and post-training quantization? Post-training quantization rounds a finished model's weights to fewer bits. QAT simulates the low-bit rounding during training, so the model adapts to it. QAT usually holds quality better at low bit widths, especially at 2 bits. ### What does quantization-aware distillation add to QAT? In QAD, the quantized model learns from the unquantized version as a teacher. This lets you finish fine-tuning and reinforcement learning at full precision, then transfer the result to the low-bit model without rerunning the whole pipeline. ### Can I make my own 2-bit model this way? Not easily. The process needs access to training pipelines, large amounts of data and compute, and ideally the original model's creators. That is why the approach is mostly seen from big labs, and why independent verification of their claims matters.