How to Run AI Agents Locally on Your Own Hardware

Running an AI agent locally means installing an open-weight model like Llama, Qwen, or DeepSeek on your own hardware, serving it with a runtime like Ollama, and connecting it to your files and tools with something like the Model Context Protocol. No prompt, no document, no customer record leaves your network. What follows is the actual how: which models are good enough, what hardware you need, which tool to install first, and how to wire it all into something that acts instead of just chats.

Why "just use the API" is a data problem, not just a cost one

Most teams don't adopt local models to save money. They adopt them because employees are already leaking sensitive data through unapproved tools, and nobody signed off on it. Fifty-nine percent of U.S. employees use AI tools their employer hasn't approved, a number that jumps to 93% among executives and senior managers.¹ Three-quarters of people using those unapproved tools admit they've shared potentially sensitive information through them, most often employee records, customer data, and internal documents.¹

This isn't hypothetical. IBM's 2025 breach research found that incidents involving shadow AI cost organizations $670,000 more on average than breaches without it, and 97% of organizations that suffered an AI-related breach had no proper AI access controls in place.²³ Banning tools doesn't fix the cause. Only about a third of employees say the AI tools their company officially provides actually meet their needs, which is exactly why people route around IT policy in the first place.³ The people building shadow AI tools out of terminals and tunneled localhost servers aren't being reckless for fun. They're solving a real problem with the tools available to them. Local models give you a sanctioned way to offer them the same thing.

How good are local models, really?

A few years ago, local meant worse. That assumption doesn't hold anymore. Llama 3.1 405B approaches GPT-4o on MMLU, scoring around 86%.⁴ DeepSeek R1 matches or beats OpenAI's o1 on hard math and code benchmarks: 96.3% on AIME 2024 versus o1's 79.2%, and 97.3% pass@1 on MATH-500.⁵ Qwen 2.5-72B-Instruct outperforms the much larger Llama-3-405B on several benchmarks despite being a fifth of the size.⁵ Open models have gotten competent enough at reasoning and coding that a well-chosen 70B-class model, not the largest one available, is usually the right fit for an internal tool. And if you're picking a model for a specific job rather than chasing leaderboard position, benchmark scores tell you less than you'd think about how it will actually behave on your data.

How much hardware do you actually need to run an agent locally?

The hard constraint is memory, not compute. A 70B-parameter model needs roughly 140GB of VRAM at FP16, and around 280GB at full FP32.⁶ No consumer GPU gets close: even a top-tier RTX 4090 ships with 24GB.⁶ Quantization is what makes this workable. Drop to 4-bit precision and that same 70B model shrinks to roughly 35GB, small enough for two consumer GPUs or one high-VRAM card.⁶ Q4_K_M quantization adds only about 1.5% perplexity relative to full FP16 on Llama 3 70B, a rounding error for most internal use cases. Quality actually starts to degrade below Q3_K_M on complex reasoning tasks, so 4-bit is the practical floor, not a shortcut.

For smaller, task-specific models, a single 24GB GPU is often enough. For lighter workloads like storytelling or summarization, some builders get by with sub-3-bit quantization on one 4090.⁷ For anything doing programming or function calling, aim for 4 bits or more on 32GB or more of VRAM.⁷

Apple Silicon changes the math entirely, because unified memory isn't split between CPU and GPU. A Mac Studio with an M3 Ultra and 512GB of unified memory can run DeepSeek R1's full 671-billion-parameter model entirely in memory at 4-bit quantization, using under 200 watts.⁸⁹ No conventional multi-GPU PC touches that workload without a $50,000-plus GPU cluster. If you don't want a Mac and don't want a GPU at all, there's a CPU-only path too: large models like GLM-5.3-Flash now run on a 128GB RAM machine you already own. For a fuller breakdown of what budget maps to what model size, see the hardware tiers for local inference, from a $2,000 GPU up to a data-center rack.

Choosing your local stack: Ollama, LM Studio, or llama.cpp

Three tools dominate local inference. The right one depends on how much control you want versus how fast you want to get moving.

• Ollama. CLI-first and built to get a model running in minutes. It handles quantization formats and model management for you, which makes it the default starting point for most teams.¹⁰
• LM Studio. Adds a GUI on top of similar underlying engines. On Apple Silicon it includes an MLX backend that can outperform llama.cpp-based tools for certain models.¹⁰ Good for teams that want a visual chat interface before wiring up agent code.
• llama.cpp. The lowest-level option, and the one power users graduate to. It gives granular control over things like GPU layer offloading, at the cost of a steeper setup.¹⁰¹¹ One developer who switched from Ollama and LM Studio described getting a model running with "just two commands" once the build was in place.¹¹

Start with Ollama if you're not sure. Move to llama.cpp once you know exactly what you're optimizing for.

How do you go from zero to a working local agent?

A chatbot that answers questions isn't an agent. An agent takes actions. Here's the path from nothing to something that can actually do work.

1. Install a runtime. Install Ollama or LM Studio. This handles model downloads and serves an inference endpoint locally.
2. Pull a model sized to your hardware. Start with a quantized model that fits your VRAM or unified memory, using the tiers above as a guide. A 7B or 8B model is fine for testing the pipeline before you commit to something larger.
3. Wire up an agent framework. A lightweight framework like LangGraph, or a plain-Python pipeline, gives the model a loop: read a request, decide on an action, call a tool, observe the result, repeat. This is what turns a model into an agent instead of a chat window. Plenty of teams now build this loop themselves in plain Python instead of adopting a heavier framework, especially for simple internal tools.¹²
4. Give it a narrow first job. Point it at one internal task, like summarizing tickets or querying a read-only database, before expanding scope. This is also where you find out fast whether your model and quantization level are actually good enough for the job.
5. Test failure modes before you trust it. Feed it edge cases, missing data, and ambiguous instructions on purpose. An agent that's confidently wrong on your internal data is worse than no agent at all.

If you're building this for engineering use rather than a general internal tool, look at how teams are self-hosting sandboxed coding agent stacks instead of paying per seat for a hosted one.

Giving your local agent tools and data access with MCP

A model with no access to your files or systems is just a smart autocomplete. The Model Context Protocol, which Anthropic open-sourced in November 2024, is the standard way to connect an AI model, including a locally hosted one, to the systems where your data actually lives: file systems, Slack, GitHub, Postgres, internal APIs.¹³ Adoption moved fast. The community has built thousands of MCP servers since launch, and it's become the de-facto standard for connecting agents to tools instead of writing one-off integrations for every system.¹⁴

This matters more for local agents than hosted ones, because the whole point of running locally is keeping your data inside your perimeter. An MCP server that talks to your internal Postgres instance or document store keeps that boundary intact. That's a different posture than handing a cloud connector read access to your Drive or ticketing system so a vendor's model can index it on their servers. If you'd rather not build that indexing layer from scratch, a platform like Remy lets you point a local agent at your own document index without renting that access from someone else.

When does local hosting actually beat the API bill?

At low usage, the API almost always wins. One 2026 cost model found that at 500,000 tokens a day, a hosted API runs about $1,260 a year against $6,457 in first-year local hardware costs.¹⁵ The math flips as usage climbs. At 5 million tokens a day, local infrastructure breaks even with the API somewhere around 18 to 24 months.¹⁵ At 50 million tokens a day, the API alone would cost roughly $126,000 a year, and local infrastructure pays for itself well before the three-year mark.¹⁵ If your team runs agents constantly against internal data rather than chatting occasionally, the crossover point arrives faster than most people expect.

Local AI as a software ownership decision, not just an IT choice

The real reason to run agents locally isn't the electricity bill. It's that the model, the weights, and the data pipeline become something you own outright, not a subscription you're renting a seat on. That's the same logic behind choosing to build internal tools instead of renting them from a SaaS vendor: once the capability gap closes enough, ownership becomes the more defensible option, not just the cheaper one.

It's also the more honest way to deal with shadow AI. Banning unapproved tools doesn't stop employees who feel unsupported from finding a workaround. Giving them a sanctioned, local, governed agent does. That turns shadow AI from a liability IT hunts down into infrastructure IT actually owns.