Can You Run Nvidia Nemotron 3.5 Bolt 74B on 24GB?

Can you run Nemotron 3.5 Bolt 74B on a 24GB machine?

Yes, that is the design goal. Nemotron 3.5 Bolt 74B A9B has 74 billion parameters in total, but its expert layers are stored at 2 bits, which shrinks the file to roughly 20 to 22 GB. That fits a 24GB device, though with limited room left over for other apps. Nvidia's speed table puts it at about 84 tokens per second on an RTX Spark laptop. That is slower than some rivals, but you get a larger model for the same memory.

What is Nemotron 3.5 Bolt 74B A9B?

It is an open model from Nvidia built for local agents: AI that can plan, call tools and work through multi-step tasks on your own machine instead of in a cloud service. The name tells you the size. The "74B" is total parameters, and "A9B" means about 9 billion are active for any given token.

It is a mixture-of-experts model. Inside are many small specialist sub-networks called experts, and a router picks a few of them for each token. You pay the memory cost for all 74 billion parameters, but each token only uses about 9 billion of them. Nvidia built it on its hybrid Mamba-transformer design, which mixes standard attention layers with Mamba layers, a different kind of layer that handles long sequences efficiently.

The model accepts images and video and supports up to a 256K context window. It was announced alongside Microsoft's RTX Spark laptops, which have 24GB of memory. The pitch is an agent model that runs on one 24GB device next to your other apps.

How does a 74B model fit in 24GB?

Through compression. A model's file size is roughly the number of parameters times the bits stored per parameter. Here is the rough math:

Bolt gets there by combining three precision levels:

Because the experts hold most of the parameters, putting them at 2 bits does most of the work. The more sensitive parts keep more precision. Nvidia is open about this mix, and it makes sense to give each layer the precision it needs.

Why was 2-bit considered a bad idea before?

The old rule for running models locally was to pick the biggest model that fits at 4-bit and not go lower. The usual way to get a smaller file is post-training quantization. The model is trained at 16 bits, and someone rounds the weights down afterward. At 8 bits that costs almost nothing. At 4 bits it usually costs little. Below 3 bits, quality tends to fall apart, because the network has to behave very differently without ever getting to adapt. That is why many community 2-bit and 1-bit files feel rough.

Research from Meta (a paper called ParetoQ) found that if you train properly at low bit widths, 2-bit and 3-bit models can beat 4-bit post-training quantized models at the same memory budget. A bigger model with fewer bits per weight can win. Bolt is Nvidia's large-scale test of that idea.

How did Nvidia make it?

The 2-bit step comes last. The earlier steps matter just as much.

1. Pruning. Nvidia started from Nemotron 3.5 Super and used a method called Star Elastic, essentially an architecture search under a memory budget. A small router learned, layer by layer, how many experts to keep and how wide each block should be. It also cut the experts used per token from 22 to 9. The budget was explicit: fit under 24GB with a batch size of 1 and a 128K context, with the later 2-bit experts already accounted for.
2. Distillation from the parent. The smaller model learned from the larger one over about 120 billion tokens. Most of this used shorter contexts (around 8K) because it is faster, with the final stretch at 128K so long-context ability came back.
3. Normal post-training. Nvidia reattached the vision encoder, ran more supervised fine-tuning, merged checkpoints and ran several rounds of reinforcement learning for agent tasks, still at 16-bit precision.
4. Quantization-aware training and distillation. During training, the model simulates low-bit rounding on every forward pass while a full-precision copy handles the updates. The model learns to work on the low-bit grid. Nvidia uses a variant where the quantized model is the student and the unquantized model is the teacher. That lets it rerun the whole pipeline once, then distill into the quantized form without needing all the original training data.

Other labs are moving the same way. Google released quantization-aware versions of Gemma, and the Gemma 4 mobile models push token-generation layers to 2 bits while keeping the core reasoning layers at higher precision. OpenAI did similar work so its 120B GPT-OSS model fits on a single 80 GB GPU. The pattern is that labs now plan for quantization during training instead of leaving it to the community.

How fast is Bolt, and how does it compare to Super?

Local AI is usually limited by memory bandwidth, not raw compute. With one request at a time, the machine spends its time streaming weights through memory. Fewer bits per weight means fewer bytes to move, which helps speed. Nvidia's model card makes this point about low batch sizes.

Even so, Bolt is not the fastest option in its class. Nvidia's speed table shows about 84 tokens per second on the RTX Spark, behind Qwen 3.6 and Gemma 4 models. That is still comfortable for reading along as text streams in, but you are paying for capability, not speed.

A hands-on test on a much bigger machine, an HP ZGX Fury, showed about 220 tokens per second. Treat that as a ceiling for stronger hardware, not what a 24GB laptop will give you. The same test compared Bolt against the larger Nemotron 3.5 Super, but the figures for Super were cut off in the material I had, so I am not quoting a number. The 148 GB Super needs far more memory than a 24GB device offers, which is the reason Bolt exists.

What do you give up?

Nvidia's reported numbers, going from the 16-bit version to the 2-bit file:

Those are small but real drops, in exchange for a file about one-seventh the size. Other trade-offs:

• Tight headroom. A 22 GB file on a 24GB device leaves little for the KV cache (the memory that holds your conversation), a browser, an IDE or a second model. Nvidia sized the model for a 128K context at a batch of one, so the full 256K window will likely demand more memory than that budget. I have not seen a measured figure for it.
• Slower than rivals. About 84 tokens per second is behind some similar-size competitors.
• Not a pure 2-bit model. Parts of it run at 4 and 8 bits.
• Hard to copy. This approach works best when the model's maker does the quantization. Third parties cannot easily reproduce the training-time steps.

A related caution: if you prune experts yourself using a method like Cerebras's REAP, which experts survive depends on your calibration data. A model calibrated on code will stay good at code and may get worse elsewhere.

Is Bolt worth running locally?

It makes sense if you have a 24GB device and want a capable agent model that stays on your hardware, for example for private documents, screenshots or internal workflows you do not want to send to a cloud API. It is a weaker fit if you need the fastest responses or you regularly run many apps alongside the model.

Before committing, do three things:

1. Check the Hugging Face model card for the supported runtimes and file format.
2. Test it on five or ten real tasks from your own work, not benchmark prompts.
3. Watch memory use with your usual apps open, since the margin is thin.

Do not confuse it with the small Nemotron 3.5 Bolt E4B 4GB model, a separate Gemma-4-based release (8B total parameters, 4B effective, Q4_K_M, 64K context) aimed at voice and device assistants. It is a different model with different goals.

What can you build around a local model?

A local model on its own is a chat window. The value comes from wiring it into something you use: a tracker that triages incoming requests, a dashboard that summarizes documents, a form that routes work. If you want a small internal tool like that without stitching together spreadsheets and subscriptions, Remy builds and deploys full-stack apps from a plain-English description, and it can reach local models over a secure tunnel as well as its catalog of nearly 400 hosted models. You could prototype with a hosted model first, then point the app at a local one if privacy matters.