---
title: How to Run Private Offline RAG and Coding Agents Locally
url: "https://goremy.ai/blog/articles/private-offline-rag-coding-agents-local-hardware"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-18T17:40:11.135Z"
updated: "2026-11-18T17:40:11.135Z"
topic: Local AI
description: "Run RAG and coding agents with no cloud: how to size memory, pick models, use prefix caching, and keep legal, health or incident data on your own hardware."
---

# How to Run Private Offline RAG and Coding Agents Locally

> Run RAG and coding agents with no cloud: how to size memory, pick models, use prefix caching, and keep legal, health or incident data on your own hardware.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-18

## How do you run RAG and coding agents entirely offline?

You run the models on a machine you control, and you keep every part of the pipeline (document index, embedding model, language model, agent tools) on that machine. No request leaves the box. The hard part is not the software. It is fitting the models in memory and getting enough speed per user, especially for agents. Once you can estimate both before you download anything, an air-gapped setup becomes a sizing exercise instead of guesswork.

## TL;DR

- **A model has to fit in fast memory** to run well. A quick estimate is parameters (in billions) times about 0.56 for a 4-bit model, which gives gigabytes.
- **Memory bandwidth, not raw compute, limits how fast a model writes its answer.** That is why two machines with similar compute can feel very different.
- **Agents need much higher per-user speed than chat.** A demo creator's rule of thumb is at least 80 tokens per second per user for agentic work.
- **Prefix caching** stops the system from re-reading the same long prompt on every agent turn, which matters because agents resend everything each time.
- **Offline matters for real reasons:** incident response, legal, healthcare and government work often cannot send documents to a cloud API.
- **Leave headroom.** Models are not the only thing in memory. The conversation cache (KV cache) grows with context length and user count.

## Why would anyone run AI with no internet connection?

Two situations come up again and again.

The first is a security incident. When a team is dealing with an intruder who may be pulling data out of the network, they often have to disconnect from the internet entirely. In that moment they cannot call a hosted model, whether it is a proprietary one or an open one served by someone else. If they want AI help analyzing logs or writing scripts, it has to run locally.

The second is plain privacy. Lawyers, clinicians and government staff often work with documents that cannot go to a cloud API at all. A viewer in the source described a law firm of 20 to 30 people. That firm may have internet access on every desk, but the document pipeline still has to keep client files inside the building.

The demonstration behind this article, a walkthrough of the HP ZGX Fury (sponsored by HP and Nvidia), showed this by pulling the network cable and running a team of coding agents on the box alone. The hardware is specific, but the sizing logic applies to any local setup.

## How much memory does a local model need?

Use this formula: parameters times bits per weight, divided by 8, gives gigabytes.

A "4-bit" model is closer to 4.5 bits in practice, because some layers are not quantized (compressed to lower precision). At 4.5 bits, the shortcut is parameters in billions times 0.56.

| Model size | Approx. memory at ~4.5 bits |
|---|---|
| 10B | about 5.6 GB |
| 50B | about 28 GB |
| 100B | about 56 GB |

Then account for the second memory cost, the **KV cache**. This is the memory each conversation uses to remember what has been said. It grows with context length and with the number of simultaneous users. If you fill every gigabyte with model weights, you have nowhere to put it. When you run several models on one GPU, their memory fractions also need to add up to slightly less than the whole.

The ZGX Fury has two pools: 252 GB of high-bandwidth memory (HBM) on the GPU, and 496 GB of unified memory attached to the CPU. The GPU can use both, but the HBM is about 18 times faster. So "fits" in the demo means "fits in HBM." In the demo, a roughly 180 GB model plus several small ones (including an embedding model and a 270 million parameter fine-tuned decision model at about half a gigabyte) already totaled over 212 GB. That is a useful picture: one big model, a few small helpers, and room left over for the cache.

## Why are some local models fast and others slow?

Inference has two phases, and they hit different limits.

- **Prefill** is the model reading your prompt. Thousands of tokens are processed at once, so it is limited by compute.
- **Decode** is the model writing its answer, mostly one token at a time. Every token requires reading the active weights from memory again, so it is limited by memory bandwidth.

That explains why smaller "GB10-class" boxes feel slow to answer. They have good compute, so prefill is fine, but their bandwidth is around 270 GB per second. The ZGX Fury's HBM runs at about 7.1 TB per second.

You can estimate a speed ceiling yourself: bandwidth divided by the size of the active weights. For a model with 18 billion active parameters (about 10 GB per token at this precision), 7.1 TB/s divided by 10 GB suggests roughly 700 tokens per second. A single user actually got around 225, about a third of the theory. Expect that kind of gap, because attention, cache reads and software overhead all take a share.

Two things change the picture:

- **Speculative decoding.** A small draft model guesses the next few tokens and the big model checks them in one pass. In the demo this lifted one model to around 283 tokens per second for a single user. It only helps while spare compute exists, so the benefit shrinks as more users pile on.
- **Mixture of experts (MoE) versus dense models.** A dense model reads all its parameters for every token. An MoE model has many parameters in total but activates only some per step. The demo compared a 65 billion parameter dense model, capped near 200 tokens per second, with a 320 billion parameter MoE model that has only 18 billion active. The bigger MoE was faster. The catch is that all of an MoE model's weights still have to sit in memory.

## How fast do coding agents need to be?

Faster than chat. A person reading a chat answer is fine at about 30 tokens per second. An agent is not reading. It writes thousands of tokens of reasoning and code before it even calls a tool, and it does that dozens of times per task.

The arithmetic from the demo: at 30 tokens per second, a 3,000-token step takes almost two minutes. Forty such steps is over an hour for one task. Sub-agents running in parallel multiply the load.

So the guideline is to pick models you can serve at roughly 80 tokens per second per user or better for agent work. When you size a team setup, look at the per-user floor, not just total throughput. In the demo, with 8,000-token prompts, one large model served 54 concurrent users at a minimum of about 40 tokens per second each, and a small model served 188. Prompt length changes these numbers a lot, so test with prompts like your own.

## What is prefix caching, and why do offline agents need it?

Agents are prefill heavy. Every turn resends the whole conversation: the instructions, the files already read, the earlier tool results. Without help, the model recomputes all of that from scratch each turn.

Prefix caching stores the already-processed beginning of the prompt so only the new part needs work. Serving frameworks such as vLLM and SGLang offer this. The agent harness in the demo displayed a cache hit rate as a percentage. A high number means the machine is skipping most of the repeated reading. If your offline agents feel sluggish after a few turns, check this setting first.

## How do you set up a private RAG pipeline?

Retrieval-augmented generation (RAG) means the model answers from your documents instead of from memory. The standard pipeline has four parts, and for a private setup every one must run locally:

1. **Embedding model.** Converts document chunks into vectors so you can search by meaning.
2. **Vector index.** Stores those vectors on your own disk.
3. **Reranker.** Re-scores the top search hits so the best passages reach the model.
4. **Language model.** Writes the answer from the retrieved passages and should cite them.

For scanned contracts, charts or forms, a vision-language model (a model that reads images) can process page images directly. That is a common approach, not something to assume your files need.

Two practical points. First, an embedding model is small. The demo loaded one alongside a 180 GB model with little cost, and the embedding model plus a small helper model came to about half a gigabyte, so the extra memory is small. Second, RAG users are people reading output, so the 30 tokens per second chat floor applies. You can serve many more RAG users than agent users on the same hardware.

For the law-firm case, also check that nothing in the pipeline phones home: no hosted embedding API, no cloud reranker, no telemetry. Internet access being "on" is fine only if the pipeline never uses it.

## What does this mean for you?

**If you work in legal, healthcare or government,** the question to ask your IT team is not "can we use AI?" but "can we run it on hardware we control?" Bring the memory formula and the 80 tokens per second agent target to that conversation.

**If you are on a security or IT team,** an offline model is part of an incident plan. Set it up and test it before you need it.

**If you manage a small team,** you do not need the largest machine. Smaller models run the same pipeline. Expect slower agents, and use chat and RAG first.

**Before buying anything,** run the memory math, check bandwidth (not only compute), and decide how many people will use it at once.

## Can you build the tools around a private AI setup with Remy?

Partly, depending on how strict your isolation is. For teams that are private but not air-gapped, such as that 20-person law firm, the missing piece is often the front end: a document request form, a matter tracker, or a search page for staff. [Remy](https://goremy.ai) builds and deploys full-stack apps from a plain-English description, and its apps can reach local or self-hosted models over a secure tunnel. A fully disconnected network is a different case, and Remy is not designed for that. Cancelling the subscription takes hosted apps offline, though the code stays yours.

## Frequently Asked Questions

### Do I need a machine like the ZGX Fury to run local RAG?

No. A RAG pipeline with a small embedding model and a mid-sized language model needs far less memory. The Fury demo shows what a large box can serve, such as many users or very large models, but the sizing method works on any hardware.

### Why is my local model slow even though the GPU is powerful?

Decode speed depends mostly on memory bandwidth, not compute. A machine with strong compute but about 270 GB per second of bandwidth reads weights slowly for every token. Check bandwidth and consider a smaller or MoE model with fewer active parameters.

### Is a 1-bit or 2-bit model good enough?

It depends on the task. Lower-bit models need less memory, but a 4-bit model (about 4.5 bits in practice) is the usual choice for coding or work needing precision.

### What speed should I aim for?

Around 30 tokens per second per user is fine for chat and RAG, where a person reads the output. For coding agents, aim for roughly 80 or more, since they generate long outputs many times per task.

### Does turning off the internet make AI private?

It keeps data from leaving the machine, which is the point for incident response. For everyday privacy, the key is that no part of the pipeline sends documents to an outside service, whether or not the network is connected.
