How to Run AI Workflows Locally (And Stop Paying the Cloud Tax)

The short answer

You run AI models locally by downloading a quantized version of an open-weight model (GGUF format), loading it with a runner like Ollama or llama.cpp, and pointing your existing tools at that local server instead of an API. On a laptop with 16GB of RAM and no dedicated GPU, you can have a 7B or 8B model answering prompts in about 15 minutes.¹

This is not a niche hobby anymore. It is the fastest way to stop metering every prompt against a monthly invoice, and it is the same instinct behind Software Ownership as a category: treat the tools you run every day as infrastructure you control, not rent.

Why bother when the API is one line of code

Cloud APIs are genuinely cheap at low volume. A million tokens through Llama 3.3 70B costs about $0.12 on DeepInfra's API.² That is nothing. The same workload run on rented Lambda Labs GPUs costs about $43, a 358x gap.² If you're sending a few hundred prompts a day, renting the API is the correct call. Nobody should self-host to save money at that scale.

But cost per token is not the only lever. Three things push people toward local inference:

• Data never leaves the machine. For legal, healthcare, or anything with a client NDA, that alone settles the argument.
• No rate limits, no outages, no pricing changes. DeepInfra raised prices on Llama 3.3 70B by roughly 250% with one day's notice in 2025.² A model on your own disk cannot do that to you.
• Fixed cost regardless of volume. If you're running thousands of automated workflow steps a day, not per-request billing, hardware you already own starts to look a lot better than a growing monthly line item.

Step 1: Pick your quantization, not just your model

The number that actually gates you is not model size, it's quantization. A model's full-precision (FP16) weights need about 2 bytes per parameter. Quantized to 4-bit (the GGUF standard, usually labeled Q4_K_M), that drops to roughly 0.5-0.6GB per billion parameters. That's the difference between needing 14GB of VRAM and needing 4-5GB for the same 7B model.

Rough VRAM math, at Q4_K_M:

• 7B model: ~4-5GB
• 13B-14B model: ~8-9GB
• 32B model: ~19-22GB
• 70B model: ~40GB

Match this against your GPU before you pick a model, not after. An 8GB card (RTX 3060, 4060) comfortably runs 7B-class models. 12-16GB opens up 13B-14B. You need 24GB or more before 32B is realistic, and 70B models are a genuinely different tier of hardware commitment.

If you don't have a GPU at all, CPU inference works, just slower. A 7B model at Q4 needs about 8GB of system RAM and will run somewhere from 5 to 15 tokens per second on a modern 8-core CPU. That's usable for background automation, not for a snappy chat interface.

Step 2: Install a runner

Two tools cover almost everyone:

Ollama is the fastest path. Install it, then pull a model:

ollama pull llama3
ollama run llama3

Ollama runs on llama.cpp under the hood and exposes an API on localhost:11434 that mimics OpenAI's chat completions format, which matters a lot for step 3.¹ As of Ollama 0.30, it can also run any GGUF file directly from Hugging Face's 45,000+ public quantized models, and Vulkan support means AMD and Intel GPUs get acceleration without vendor-specific driver installs.³

llama.cpp is the lower-level option if you want more control over context length, GPU layer offloading, or you're building your own tooling around it. It compiles with a few CMake flags and gives you a llama-server binary that speaks the OpenAI API format out of the box.

LM Studio is the GUI alternative. It wraps the same llama.cpp engine but gives you a model browser, a chat window, and a one-click local server toggle. Good if you don't want to live in a terminal.

Step 3: Wire it into your actual workflow

Running a model in isolation is a demo. The point is replacing an API call inside something you already use. n8n is the most common pairing here because its OpenAI-compatible nodes will happily point at a local Ollama or LM Studio endpoint instead of OpenAI's servers.¹⁴

Practically: in n8n, set the OpenAI credential's base URL to your local server (http://localhost:11434/v1 for Ollama, or the LM Studio server URL plus /v1), pick your model by ID, and the rest of the workflow doesn't change.⁴ n8n's own self-hosted AI starter kit bundles n8n, Ollama, and a vector database in one Docker Compose file specifically to make this pairing turnkey.⁵

One real gotcha worth flagging: some setups fail on the first request with a generic "bad request" error because n8n defaults to the newer Responses API format and LM Studio expects standard chat completions. Disabling "Use Responses API" in the node settings fixes it immediately.⁴ It's a five-minute fix, but it will make you think something is broken.

When local wins on cost, concretely

Below is the honest math, using published API and hardware rates.

At low-to-moderate volume, the API is cheaper, full stop. At 1M tokens a day of Llama 3.3 70B traffic, the DeepInfra API costs about $0.21, while renting an A100 to self-host the same load costs $30-40 a day.² You'd need an extraordinary, sustained volume before self-hosted rented GPUs beat a pay-per-token API on raw cost.

Where local ownership actually changes the math is when you already own the hardware. A 7B or 14B model running on a GPU you bought for something else, or a Mac you already have, costs you electricity, not a line item. For an individual builder or a small team running frequent internal automations, that's often the more relevant comparison than API-versus-rented-cloud-GPU.

FAQ

Can I run a local LLM without a GPU? Yes. A 7B model at Q4_K_M quantization needs about 8GB of system RAM and will run on CPU at roughly 5-15 tokens per second on a modern 8-core processor. Slower than GPU inference, but workable for background jobs that don't need instant responses.

Are local models as good as GPT-4 or Claude? For general chat and many task-specific jobs (coding, summarization, classification), well-tuned 7B-32B open models get close enough that the gap doesn't matter for most workflows. For frontier reasoning tasks, cloud flagship models still lead. Many teams run local models for routine, high-volume steps and reserve API calls for the hard cases.

What's the actual difference between GGUF and the original model files? GGUF is a single-file format that bundles quantized weights, tokenizer, and chat template together, replacing the older GGML format in 2023.¹ It's the format both llama.cpp and Ollama expect, and it's what makes running a model a one-command operation instead of a multi-file setup.

Do I need to know Python to do any of this? No. Ollama and LM Studio are both installable apps with no coding required to download and run a model. You only touch code if you're converting your own fine-tuned model to GGUF or scripting something custom around the API.

What's the minimum realistic hardware to start? An 8GB VRAM GPU (a used RTX 3060 or 4060 class card) or a 16GB unified-memory Apple Silicon Mac runs 7B-class models comfortably. If you're on a laptop with no dedicated GPU, 16GB of system RAM is enough to get a smaller model running today.¹