Local LLM Hardware: What Tokens Per Second Can You Expect?

What GPU or Mac do you need to run local models at usable speeds?

You need enough fast memory to hold the model, and for a single agent that means a high-end GPU or a high-memory Mac. An RTX 4090 (24 GB) can run a compact model like Qwen 3.8 27B at roughly 25 to 40 tokens per second with a tight fit. That is the low end of usable. Faster cards raise speed. Larger models need more memory than any consumer GPU offers, and even when you quantize them down you are still limited, so they push you toward big unified-memory Macs or data centers.

What does each hardware tier look like?

The table mixes figures quoted in the source discussion with general specs. Where a price or speed wasn't established, it says so.

Prices for GPUs have swung hard. The 4090 and 5090 figures above are well over their original launch prices, and these cards are not easy to find anymore. Check current listings before planning a purchase.

How fast is fast enough?

The practical thresholds are simple.

• 15 to 30 tokens per second, one agent: fine for short question and answer. Poor for agentic coding or long outputs.
• 70 to 100 tokens per second, one agent: quite usable for real work.
• Anything under that, split across several agents: slow in practice.

This is the number local AI discussions tend to skip. People post which model they got running. They rarely post how fast it generates. A 27B model that loads but crawls at 20 tokens per second is a demo, not a workflow.

Why does a single GPU struggle with multiple agents?

Inference on one GPU has a fixed budget of throughput. If you start five or ten agents in parallel, they share it. A card that gives one agent 80 tokens per second does not give five agents 80 each. You get a fraction per agent, and the whole setup feels slow again.

Hosted services avoid this by spreading requests across racks of GPUs. At home, the only way to match that is to build a small data center: multiple 5090s linked together, or several compact AI boxes. That can run a fairly large model, and the result can come close to hosted speed. It also costs a lot of money and a lot of setup time.

One real advantage remains. A local machine can run around the clock without hitting a usage quota. Since you already paid for the hardware, keeping it busy spreads that cost over three to five years. A hosted subscription would throttle or push you to a pricier plan under the same load.

Where do the RTX Pro 6000 and Mac Studio fit?

These two exist for the same reason: capacity. A consumer card tops out at 24 or 32 GB. The RTX Pro 6000 Blackwell offers 96 GB of VRAM on a single professional card, which opens the door to larger models, or to a mid-size model with more room for context. Mac Studio takes a different route. Apple's unified memory lets the GPU address a very large pool, so big models load that would never fit on a gaming card.

The tradeoff is general rather than specific to any one model. Memory capacity decides whether a model loads. Memory bandwidth and compute decide how fast it generates. A Mac can hold a huge model and still generate more slowly than a dedicated Nvidia card running a smaller one. The sources here did not give measured tokens per second for these two tiers, so treat any specific number you see elsewhere as something to verify against your own model, quantization, and context length.

Why do trillion-parameter open models still need quantization or data centers?

The best open-weights models, many of them Chinese, sit about one generation behind the top closed frontier models. They are good. They are also huge. Models near that quality run to trillions of parameters, and that does not fit on consumer hardware.

Quantization shrinks a model by storing weights at lower precision. You give up some quality and gain a much smaller footprint. It helps, but it does not close the gap for the largest models. Even after quantizing, you are often limited to low tokens per second, and parallel agents are out of reach.

That is why hosted providers, from the big clouds to smaller GPU clouds, load these models onto their own GPU racks. Open weights and local are different things. Every local model is open weights. Plenty of open-weights models are only practical to run in someone's data center.

Does the math favor local AI?

For most people, no. A ChatGPT or Claude plan runs from about $20 to $100 a month. A GPU that can run a decent compact model costs thousands. Even a strong setup trails the frontier on quality, and near-frontier open models need far more hardware than one card.

The reasons to go local are different:

• Data sovereignty: your prompts and files stay on your machine.
• Access control: a provider can change its rules or cut off a tool you rely on. A local model can't be revoked.
• Offline use: it works without internet, which matters for niche cases.
• Owning the capability: some people value not depending on a handful of labs.

Setup is also a real cost. You have to deal with differences between GPU architectures, operating systems, model formats, and settings, and model architectures are not standardized. Tools like LM Studio smooth this considerably, though it is still not plug-and-play.

Who should buy hardware now?

If you want privacy, tinkering, or an always-on worker for simple jobs, a 24 GB or 32 GB card running a model in the 27B class is a reasonable start. If you want multiple agents or something near frontier quality, expect to spend far more, or keep using hosted models and open-weights models served from the cloud. Whichever route you take, measure tokens per second on your actual workload before you commit.