Blog
will model fit in VRAMestimate tokens per secondKV cache memory

Will an LLM Fit in GPU Memory? Estimate Tokens per Second

Estimate whether a local AI model fits in GPU memory and how fast it will run, using simple formulas for weights, KV cache, memory bandwidth, and MoE models.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For IT and Engineering

Overhead view of a dense even grid of identical blank matte memory modules, with one red module slightly raised and casting a tiny hard shadow.

How can you tell if a model will fit in GPU memory before you download it?

Multiply the model's parameter count (in billions) by the bits stored per weight, then divide by 8. The result is the memory in gigabytes the weights need. A model quantized to roughly 4.5 bits per weight needs about 0.56 GB per billion parameters, so a 100 billion parameter model takes about 56 GB. Then leave free space for the KV cache, which holds each conversation. Speed has its own formula: memory bandwidth divided by the size of the weights read per token.

TL;DR

  • Weights memory is parameters times bits, divided by 8. At about 4.5 effective bits, that is roughly 0.56 GB per billion parameters.
  • "4-bit" usually means closer to 4.5 bits because some layers stay unquantized or use variable quantization.
  • Fit means more than the weights. The KV cache grows with context length and number of users, so you need free memory beyond the model file.
  • Generation speed is capped by bandwidth divided by active weights. Real single-user speed in one test came to about a third of that ceiling.
  • Mixture of experts (MoE) models read only some weights per token, so a huge MoE can out-run a smaller dense model, as long as all the weights fit in fast memory.
  • Agents need roughly 80 tokens per second per user. Chat can get by on about 30.

How do you calculate how much memory a model's weights need?

The formula: GB = billions of parameters × bits per weight ÷ 8.

A model is a very large pile of numbers (weights). Each number takes some bits to store. Fewer bits per weight means a smaller file. That is what quantization does: it stores the same weights at lower precision.

Most "4-bit" quantizations are not exactly 4 bits. Some layers are kept at higher precision, and many formats store a scale value for each small block of weights. In the ZGX Fury walkthrough this article draws on, the presenter used NVFP4, where each weight takes four bits and blocks of 16 share an 8-bit scale. He suggests planning at 4.5 bits to be safe.

Effective bitsGB per billion parameters100B model30B model
162.0200 GB60 GB
81.0100 GB30 GB
4.5about 0.56about 56 GBabout 17 GB

Check the rule against a real example from the video: a 320 billion parameter model at 4.5 bits comes to about 180 GB, which matches the 320 × 0.56 estimate.

Why isn't "fits in memory" the same as "fits in fast memory"?

Some hardware has more than one kind of memory. The machine in the video has 252 GB of high-bandwidth memory (HBM) on the GPU, moving about 7.1 TB per second. It also has 496 GB of unified memory attached to the CPU, at around 396 GB per second. The GPU can use both, but the first is about 18 times faster.

So "it fits" should mean "it fits in the fast memory." A model that spills into the slower pool will run, but at a fraction of the speed. Check which memory you are really counting. On a typical desktop card, the number that matters is the VRAM on the card, not your system RAM.

DeepSeek V4 Flash shows the problem. At 4-bit NVFP4 it is about 420 GB, far more than 252 GB of HBM. A 1-bit version does fit. The presenter measured about 80 tokens per second on it and found the output better than he expected, because the quantization keeps some parts of the model at higher precision. He would still pick a larger quantization for coding or other precision-sensitive work.

What is the KV cache, and how much free memory do you need?

The KV cache is the working memory a model uses to keep track of what has been said in each conversation. It grows with the context length and with the number of users. Weights are a fixed cost, but the KV cache is not.

You cannot fill every gigabyte with weights. Leave room for the cache, and if you run several models on one GPU, remember that the memory fractions you assign to them must add up to slightly less than one.

A practical way to budget:

  1. Add up the weights of every model you want loaded.
  2. Reserve a chunk of memory for the KV cache. Reserve more if you expect long prompts or many simultaneous users.
  3. If the total goes over, drop to a smaller quantization, a smaller model, or fewer concurrent users.

The video's "jigsaw" example makes the same point. After loading a 180 GB model, a 19 GB model, a small 4B model, an embedding model and a half-gigabyte fine-tuned decision model, the total passed 212 GB. The presenter did not fill the remaining space with more models because that space is where the cache has to live.

How do you estimate tokens per second?

Generation has two phases, and they hit different limits.

  • Prefill is the model reading your prompt. Thousands of tokens go through at once, so each weight loaded is used for a lot of math. This is limited by compute.
  • Decode is the model writing the answer, generally one token at a time. For every token it reads all its active weights from memory again. This is limited by memory bandwidth.

That is why two machines with similar compute can feel very different. In the video, GB10-class boxes are described as having good compute, so prefill is fine, but only about 270 GB per second of bandwidth, so decode is slow. The ZGX Fury's 7.1 TB per second is why it decodes so much faster.

Ceiling for decode speed = memory bandwidth ÷ GB of active weights read per token.

Worked example from the video: a model with 18 billion active parameters reads about 10 GB per token. 7.1 TB per second divided by 10 GB gives roughly 700 tokens per second. A single user measured about 225, around a third of the ceiling. Attention, KV cache reads and kernel overhead eat the rest. Treat the formula as an upper bound and expect a real result well below it.

A hypothetical example you can adapt: a card with 24 GB of memory and 900 GB per second of bandwidth. After reserving some space for the cache, perhaps 19 GB goes to weights, or roughly a 34 billion parameter model at 4.5 bits. A 14 billion parameter dense model at 4.5 bits is about 8 GB, so the ceiling is around 110 tokens per second and a realistic figure is perhaps a third of that. These are illustrative numbers, not benchmarks. Always check the actual specs of your own hardware.

Why can a bigger MoE model be faster than a smaller dense model?

A dense model uses every parameter for every token. A mixture of experts (MoE) model has many parameters in total but activates only a subset for each token.

The video compares two:

  • A 65 billion parameter dense model reads all 65 billion parameters per token, capping it around 200 tokens per second.
  • A 320 billion parameter MoE model has only 18 billion active per step, so it can go much faster despite being about five times larger.

The catch is memory. Every expert's weights still have to sit in memory, even the ones not used for a given token. A big MoE gives you speed only if the whole thing fits in fast memory. Dense models are not worse: the presenter notes that people often like them for coding, and a 27B dense model is a popular example. Pick on quality for your task first, then use the math to check speed.

How does speculative decoding speed things up?

Because decode is limited by memory reads, spare compute goes unused. Speculative decoding puts it to work. A small draft model guesses the next few tokens, and the big model checks them all in one pass. You trade some spare compute for fewer trips to memory.

In the video, this lifted the 18B-active model from about 225 to about 283 tokens per second for one user. The gain only lasts while compute is spare. With many concurrent users, the compute gets used up and you drop back to the slower speed.

How many users can one machine serve, and how fast is fast enough?

Batching means one read of the weights serves many users at once. With 16 users on the same model, each got a bit under 80 tokens per second, which adds up to well over 1,000 tokens per second in total. Decode becomes a compute problem instead of a bandwidth one. Prompt length matters a great deal: those tests used 8,000 token prompts, and results differ a lot between models.

What counts is the per-user floor, and it depends on the job:

UseRough per-user speedWhy
Chat, document Q&A (RAG)about 30 tokens per secondA person reads as the text appears
Agentsabout 80 tokens per second or moreNobody is reading, and the work stacks up

The agent math is stark. At 30 tokens per second, a 3,000 token step takes almost 2 minutes. Forty steps for one task is over an hour. At 80 tokens per second, the same 40 steps take about 25 minutes. Sub-agents running in parallel multiply the demand.

Agents are also prefill-heavy, because every turn resends the whole conversation, files and tool results. Prefix caching, which reuses work on the part of the prompt that has not changed, saves a lot. Servers such as vLLM and SGLang support it, and the video shows the cache hit rate as a number to watch.

What this means for you

If you are choosing hardware or a model for work: write down three numbers before buying or downloading anything: weights size (parameters × bits ÷ 8), cache budget, and bandwidth ÷ active weights for the speed ceiling. That takes five minutes and prevents most bad purchases.

If you handle sensitive documents (law, healthcare, government), local inference lets data stay on your own hardware. The math tells you what size of model your hardware can really run.

If you plan to use agents, test per-user speed, not just peak speed. A model that feels fine in chat can be painfully slow as an agent.

If you are not technical, ask your IT team three questions: how much fast memory, what bandwidth, and how many people at once.

Could you build your own model-fit calculator?

Spreadsheets for this tend to rot: someone changes a bit width, forgets the cache, and the numbers stop matching. A small internal tool can hold the formulas once. With Remy, you can describe a model-fit calculator in plain English (inputs for parameters, bits, active parameters, bandwidth, users and context length, outputs for memory needed and a speed ceiling) and get a deployed app with a login. You could add a table where teammates log measured tokens per second for each machine, so estimates sit next to real results.

Frequently Asked Questions

4 questions

What does 0.56 GB per billion parameters mean?

It is the memory cost of a model quantized to about 4.5 bits per weight: 4.5 ÷ 8 is roughly 0.56 GB per billion parameters. Multiply by the parameter count for a quick weights estimate. It excludes the KV cache and other overhead.

Why is my real tokens per second so much lower than the formula says?

The formula (bandwidth ÷ active weights) is a ceiling. Attention, KV cache reads and software overhead reduce it. In the example above, a measured single-user speed was about a third of the theoretical figure, which the presenter calls normal.

Does a model with more parameters always run slower?

No. In a mixture of experts model, only some parameters are active for each token. A 320 billion parameter MoE with 18 billion active can decode faster than a 65 billion parameter dense model, provided all its weights fit in fast memory.

Is a 1-bit model worth running?

It can be, depending on the task. The presenter found a 1-bit DeepSeek V4 Flash build better than expected, partly because some parts of the model stay at higher precision. For work that needs precision, such as coding, he would pick a larger quantization like 4-bit.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.