Blog
Edge0-145BEdge0 145B NVFP4run 145B model locally

How to Run Edge0-145B NVFP4 on a Single 24GB GPU

Edge0-145B NVFP4 is a 145B open-weight MoE built for one 24GB GPU. See the VRAM and RAM it needs, how NVFP4 and Q3_K differ, and how to check your PC.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Engineering and IT

A single large graphics card with a blank fan shroud and no lettering, shown at an oversized scale on a plain background.

What hardware do I need to run Edge0-145B NVFP4 locally?

You need a GPU with 24 GB of video memory, roughly 38 GiB of free system RAM, and enough disk space for the weights. The Edge0-145B NVFP4 release lists 23.25 GiB of VRAM plus 38.27 GiB of host RAM when it runs with "residency," meaning the parts of the model that get used most stay on the GPU while the rest waits in regular memory. In practice, that means a 24 GB consumer card and a machine with 48 GB or, more comfortably, 64 GB of RAM.

TL;DR

  • Edge0-145B is a 145-billion-parameter open-weight model that is designed to run on one consumer GPU, not a server rack.
  • The published footprint is 23.25 GiB of VRAM plus 38.27 GiB of host RAM, about 61.5 GiB combined, so a 32 GB RAM machine is too small.
  • It works because it is a sparse mixture of experts: 48 layers, 512 experts, and only 10 experts active for each token.
  • The NVFP4 build is trained natively at 4 bits, according to the release's own description, rather than squeezed down after the fact.
  • A Q3_K build and a 120B variant also exist, but the material I reviewed gives no memory figures for them, so check the model page before you download.
  • With under 1 GiB of headroom on a 24 GB card, close other GPU-hungry apps before you load it.

How can a 145B model fit on a 24GB card?

It cannot hold all 145 billion parameters in 24 GB of VRAM, and it does not try to. Two things make it workable.

First, most of the model sits idle on any given word. Edge0-145B is a sparse mixture of experts (MoE). Instead of one giant block of numbers that all fire every time, the model has many smaller specialist blocks called experts, plus a router that picks a few of them for each token (a token is a word or word fragment). Here, the model has 512 experts per layer set and activates 10 per token. That is about 2 percent of the experts doing work at any moment.

Second, the model splits its memory between the GPU and system RAM. This is the residency part. The GPU holds what it needs constantly, and the rest lives in host RAM and gets pulled in as needed. That is why the published requirement has two numbers: 23.25 GiB of VRAM and 38.27 GiB of host RAM.

Add them up and you get about 61.5 GiB of working memory. For scale, 145 billion parameters at exactly 4 bits each would be around 67 GiB of raw weights, so the published numbers sit in the same neighborhood. That is the point of the design: the model is large on disk and in total memory, but small in what the GPU touches per step.

What does "latent sparse MoE" mean?

Two terms are packed into that phrase.

Sparse means only a fraction of the model runs for each token, as described above. The benefit is speed and memory pressure: you pay compute for 10 experts, not 512.

Latent refers to a design choice in how the experts work internally. The release describes the architecture as a "latent MoE" but the material I had does not spell out the mechanics, so I will not guess at them. What matters for you is the outcome the release claims: a design aimed at making a very large expert pool practical on modest hardware.

The numbers from the release, in one place:

SpecValue
Total parameters145B
Layers48
Experts512
Active experts per token10
Context length256K
VRAM (with residency)23.25 GiB
Host RAM (with residency)38.27 GiB

A 256K context window means the model can, in principle, take in roughly a few hundred thousand tokens of text at once. Long context uses extra memory for what is called the KV cache (the model's working notes on everything it has read so far). I could not confirm what context length the 23.25 GiB figure assumes. If you plan to feed it very long documents, expect memory use to rise and test with shorter inputs first.

What is NVFP4, and why does it matter here?

NVFP4 is a 4-bit floating-point number format from NVIDIA. Storing each weight in 4 bits instead of 16 cuts memory to roughly a quarter, which is the main reason a model this big can be considered for a home machine at all.

The usual route to a 4-bit model is to train at higher precision and then compress it afterward, which can cost some quality. The Edge0 release says this model was "trained natively at 4 bits." That means the model learned to work with 4-bit weights from the start. Whether that closes the quality gap is something you should test on your own tasks, because I have no benchmark data from the source to cite.

One hardware point from general knowledge: NVIDIA's newer GPU generation (Blackwell, which includes the RTX 50-series) supports NVFP4 in hardware. On older cards, software may need to convert the weights to something the card can handle, which can cost speed. Check the model page and the project's GitHub repository (github.com/Edge0-AI/Edge0) for which GPUs and software are supported.

NVFP4, Q3_K, or 120B: which build should I pick?

The release comes in more than one form. Here is what I can say with confidence and what I cannot.

BuildWhat it isWhat to know
NVFP4 (145B)4-bit, trained natively at that precisionThe main release. Published figures: 23.25 GiB VRAM, 38.27 GiB host RAM
Q3_KA roughly 3-bit quantization format from the llama.cpp familyGenerally smaller than 4-bit with some added quality loss. Check the page for sizes and supported software
120B variantA smaller version of the modelFewer parameters should mean less memory, but I have no confirmed figures

A rough rule for choosing:

  1. Start with NVFP4 if your hardware fits it. It is the build the release is centered on.
  2. Try Q3_K if you are short on RAM or your software stack is built around llama.cpp-style files.
  3. Consider the 120B variant if the full model will not fit and you would rather have a smaller model that runs well than a big one that struggles.

How do I check whether my PC can run it?

Do this before downloading anything that large.

  1. Check your GPU memory. On an NVIDIA card, open a terminal and run nvidia-smi. Look at total memory. You need a 24 GB card, and the model's 23.25 GiB leaves less than 1 GiB of slack.
  2. Free up the GPU. If the card also drives your monitor, your desktop and browser use some VRAM. Close games, video editors, and extra browser windows, or plug your display into the motherboard's video output if it has one.
  3. Check your system RAM. You need at least 38.27 GiB free for the model, on top of what your operating system and other apps use. A 32 GB machine will not do. 64 GB is the comfortable target.
  4. Check disk space. The weights are large. Confirm the file size on the model page and leave extra room.
  5. Read the model page and the GitHub repo. Use the loading instructions and runtime they recommend. I am not listing commands here because the material I reviewed did not include them, and a wrong command on a download this size wastes hours.
  6. Test short before long. Run a few short prompts, watch memory with nvidia-smi, and only then try long documents.

Is running a 145B model locally worth it?

It depends on why you want it.

It makes sense when your data cannot leave your machine, you want to experiment without per-token fees, or you want to see how far open-weight models have come. A model of this size running on a single consumer card is a notable shift from needing multi-GPU servers.

It makes less sense when you just need quick answers for everyday work. A hosted AI service is simpler, needs no 60 GiB of memory, and does not require you to manage drivers and software. Also keep in mind the limits of what I can tell you: the release material I reviewed does not give speed figures, so I cannot promise how many words per second you will get. Offloading parts of a model to system RAM is generally slower than keeping everything on the GPU, so test it on your own hardware.

Who does this change things for?

  • Hobbyists with a high-end gaming PC. A 24 GB card plus a RAM upgrade may be all it takes to try a very large model.
  • Teams with privacy rules. If sensitive documents cannot go to an outside service, a capable model that runs on one workstation is a real option to evaluate.
  • IT and analytics staff. The hardware bar for local AI just moved, so it is worth revisiting what your team assumed needed a server.

What you can do this week: check your GPU memory and RAM with the steps above, read the model page, and try the smaller builds first if your machine is borderline.

What can I build around a local model like this?

A model running on your own machine is only useful if something connects to it. If you want an internal tool, such as a request form, a document intake page, or a small dashboard that sends text to your model, Remy can build the full app from a plain-English description. Remy's model setup lets you register custom or self-hosted models, or reach local models over a secure tunnel, so the app can use a model you run yourself. Apps get a database and sign-in built in, which saves you from stitching together spreadsheets and extra subscriptions.

Frequently Asked Questions

4 questions

Can I run Edge0-145B on a 16 GB or 12 GB GPU?

The published figure is 23.25 GiB of VRAM, so a 16 GB or 12 GB card does not meet it. The smaller 120B variant or the Q3_K build may need less, but I have no confirmed numbers for them. Check the model page.

Do I need a lot of system RAM too?

Yes. The release lists 38.27 GiB of host RAM in addition to the VRAM. That rules out 32 GB machines once your operating system is counted. Plan on 48 GB at the minimum and 64 GB for comfort.

Why only 10 experts out of 512?

Activating a small number of experts per token keeps the work per token low while the model as a whole stays large. That is how a 145B model can be practical on one consumer card. The tradeoff is that all the experts still have to live somewhere, which is why you need the extra system RAM.

Is the NVFP4 version lower quality than the full-precision model?

The release says it is trained natively at 4 bits, which is different from compressing a finished model. I have no benchmark numbers from the source, so test it on your own prompts before relying on it for real work.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.