---
title: How to Run Distilled AI Models Locally (DeepSeek-R1, Ollama)
url: "https://goremy.ai/blog/articles/how-to-run-distilled-ai-models-locally-deepseek-r1-ollama"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-10-21T10:40:06.098Z"
updated: "2026-10-21T10:40:06.098Z"
topic: Local & Open-Weight Models
description: Learn how to pick, quantize, and run distilled models like DeepSeek-R1 locally with Ollama or llama.cpp, and when self-hosting beats API costs.
---

# How to Run Distilled AI Models Locally (DeepSeek-R1, Ollama)

> Learn how to pick, quantize, and run distilled models like DeepSeek-R1 locally with Ollama or llama.cpp, and when self-hosting beats API costs.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-10-21

## How do you run distilled AI models like DeepSeek-R1 locally?

You run distilled AI models locally by matching a compressed checkpoint's parameter size to your available VRAM, quantizing it into GGUF format, and serving it through a runtime like Ollama, llama.cpp, or vLLM. It's the same approach DeepSeek used to turn its 671B-parameter R1 model into six dense checkpoints small enough to run on a gaming laptop.

**DeepSeek-R1-Distill-Qwen-32B vs OpenAI o1-mini**

| Benchmark | DeepSeek-R1-Distill-Qwen-32B | OpenAI o1-mini |
| --- | --- | --- |
| AIME 2024 pass@1 | 72.6% | 63.6% |
| MATH-500 | 94.3% | 90% |

Most people who ask "can I run this at home" are really asking a narrower question: is the thing I'd be downloading actually the frontier model, or a compressed stand-in for it? The honest answer is the second one. That distinction changes what you should expect, what you should buy, and when it's actually worth doing.

## TL;DR

- Distilled models are **smaller architectures trained to mimic frontier models**, not compressed versions of them.
- DeepSeek's 32B distilled checkpoint **beats OpenAI's o1-mini** on major math and reasoning benchmarks such as AIME 2024 and MATH-500.
- Quantizing to GGUF, with **Q4_K_M as a sensible default**, lets a distilled model fit on consumer hardware at some cost to output quality.
- Ollama is the fastest way to start, llama.cpp gives you more control, and **vLLM suits production throughput**.
- Self-hosting is only cheaper than renting APIs if your daily volume exceeds **roughly 500 million tokens**.
- **Compliance and data privacy** remain the most compelling reasons to run AI workloads locally.
- Smaller distills still underperform their teacher on the hardest tasks, so **pick the model for the job** it's suited to.

## What does "distilled" actually mean?

Knowledge distillation is a teacher-student setup. A large "teacher" model generates outputs, in DeepSeek's case, 800,000 reasoning traces produced by the full 671B R1 model, and a smaller "student" model, built on an existing open architecture like Qwen2.5 or Llama 3, is fine-tuned on those traces to imitate the teacher's reasoning behavior. [1] [2] It's not compression in the zip-file sense. DeepSeek didn't shrink R1; they trained separate, smaller models to mimic it. [3]

That's why the community consensus, stated bluntly on Hacker News, is that these checkpoints are "better thought of as fine-tunes of Qwen or Llama using DeepSeek output," not the actual DeepSeek model running at a smaller size. [4] The real R1 is a 671B mixture-of-experts model that needs something like 700GB of VRAM or RAM to run at usable precision, and hardware to serve it properly runs into six figures. [4] Meta took a similar approach with Llama 3.2's 1B and 3B edge models, distilling behavior down from larger teacher checkpoints so the small models could run directly on phones. The broader research literature calls this exact pattern, transferring capability from proprietary frontier models into open-source checkpoints, one of the pivotal techniques making open models viable at all. [5]

**What It Actually Takes to Run Real R1**

- 700GB: VRAM/RAM needed to run full 671B R1 at usable precision
- €100,000+: Approximate cost of hardware to serve it properly

**AIME 2024 pass@1 by Distilled Model Size**

| Model size | AIME 2024 pass@1 |
| --- | --- |
| 1.5B | 28.9% |
| 32B | 72.6% |

_Same architecture family (DeepSeek-R1-Distill-Qwen), illustrating how parameter count drives reasoning accuracy._

Understanding this upfront saves you from a bad first impression. An 8B download that argues like a much bigger model, then gets a logic puzzle wrong, isn't broken. It's doing exactly what a distilled model does.

## Which distilled model fits your hardware?

DeepSeek shipped six R1-distilled sizes, and the accuracy ladder across them is steep:

- **1.5B** (Qwen2.5-based). Runs on almost anything, including laptops with no GPU, but scores far behind the larger sizes: 28.9 on AIME 2024 pass@1 and 16.9 on LiveCodeBench. [1]
- **7B / 8B** (Qwen2.5 or Llama3-based). A good sweet spot for a single consumer GPU or unified-memory Mac. An 8B checkpoint quantized to Q8_0 downloads at around 8.5GB. [3]
- **14B**. Noticeably stronger reasoning, still fits comfortably on a 16-24GB card.
- **32B**. The standout: DeepSeek-R1-Distill-Qwen-32B beats OpenAI's o1-mini on AIME 2024 (72.6 vs 63.6 pass@1) and MATH-500 (94.3 vs 90.0), genuinely new territory for an open dense model at that size. [1]
- **70B**. The largest distill. Needs serious multi-GPU or high-RAM hardware, and delivers the closest approximation of R1-style reasoning you can self-host.

**70B Model Memory Footprint: FP16 vs INT4 Quantization**

| Precision | VRAM required |
| --- | --- |
| FP16 | 140 GB |
| INT4 (Q4) | 35 GB |

If you're unsure which tier your GPU actually supports, our breakdown of what fits on a 12GB card walks through the VRAM math size by size. As a rule, start smaller than you think you need. One widely-shared local setup guide puts it plainly: test the waters with a smaller model first before committing disk space and download time to the 70B. [6]

## Why does quantization matter before you download?

Every practical local deployment runs on GGUF, the file format llama.cpp introduced to make quantized weights portable across runtimes, including Ollama. [7] Quantization reduces the bit-depth of the model's weights: instead of storing every number at 16-bit precision, you store most of them at 4 or 5 bits, which shrinks file size and memory footprint at some cost to output quality.

The naming (Q4_K_M, Q5_K_M, Q8_0, and so on) tells you the tradeoff. The "K" variants use a smarter allocation scheme, giving more bits to layers that need precision and fewer to layers that don't, which is why k-quants generally beat older legacy quantization at the same file size with lower error. [8] Q4_K_M is the default most people should reach for first: it cuts memory need dramatically while keeping the model's actual reasoning intact for most tasks. If you're on tight VRAM, Q4 gets you running. If you have headroom, Q8 or higher keeps you closer to the model's benchmarked scores.

## Which runtime should you install: Ollama, llama.cpp, or vLLM?

You have three realistic options, and they trade ease of use for control:

**Quantized 8B Download Size**

- 8.5GB: 8B distilled model at Q8_0 quantization

1. **Ollama** is the fastest path from zero to running model. It handles quantized weight downloads, model switching, and a local API server in one command.
2. **llama.cpp** gives you the underlying engine directly. More control over quantization, context length, and hardware-specific build flags, at the cost of a build-and-configure step instead of a one-liner.
3. **vLLM** is built for production throughput: batched requests, multiple concurrent users, serving infrastructure rather than a single chat session. It's the right choice once you're putting a distilled model behind an internal tool instead of just talking to it yourself.

For a first run, Ollama wins on speed to result. For teams serving a model to more than one person or app, moving to vLLM later is a natural next step, not a step backward.

## How do you run and validate the model locally?

With Ollama installed, pulling and running a distilled reasoning model is one command:

**The Self-Hosting Break-Even Math**

- 11B tokens/mo: Break-even point for self-hosting
- 10×: Cost inflation from a GPU idling at 10% utilization
- 733×: More expensive than API pricing at low daily volume

```
ollama run deepseek-r1:8b
```

Ollama maps size directly to tag: swap `8b` for `1.5b`, `14b`, `32b`, or `70b` depending on what your hardware supports. [6] Add a chat front end like Open WebUI or a CLI wrapper if you want something friendlier than a terminal prompt.

Configuration matters more than most people expect. DeepSeek's own usage notes recommend a temperature between 0.5 and 0.7, with 0.6 as the sweet spot, skipping the system prompt entirely, and letting the model's response begin with a `<think>` tag so it reasons before answering. [1] Independent benchmarking that reproduced DeepSeek's evals used that same 0.6 setting to stay faithful to the published numbers. [9] If your local model feels dumber than the benchmarks suggest it should be, check your defaults before you blame the weights, a topic we cover in more depth in our [guide to tuning local models](https://goremy.ai/blog/articles/how-to-make-a-local-llm-smarter-a-practical-tuning-guide).

## How does it benchmark against the frontier model you were renting?

Don't take the published numbers on faith for your own use case. Ollama pairs cleanly with OpenAI's simple-evals framework, and one detailed walkthrough runs GPQA-Diamond against a locally-served distilled model to see how it holds up. [9] The honest conclusion from that exercise: distilled models "may not yet rival the 671B parameter original model on challenging reasoning benchmarks," but remain genuinely useful for less demanding tasks. [9] Run the same eval against whichever API model you're currently paying for, on the same prompts, before you decide the local version is good enough to replace it.

**The Cost of Skipping Local Compliance**

- $4.44M: Average cost of a data breach (IBM)

## When does running locally actually beat the API bill?

Owning the model is not automatically cheaper. One detailed cost analysis puts the self-hosting break-even point around 11 billion tokens per month, roughly 500 million tokens a day, below which per-token API pricing wins once you account for idle GPU time, electricity, and the DevOps labor to keep the box running. [10] The same analysis notes a blunt failure mode: a GPU sitting at 10% utilization inflates your effective per-token cost tenfold, and at low daily volume, self-hosting on cloud GPU rental can run over 700 times more expensive than just using an API. [10] A simpler way to frame it: local cost is hardware amortization plus electricity, API cost is linear per-token spend, and which one wins depends entirely on your volume curve. [11] If you want to run your own numbers rather than someone else's, our [break-even calculator piece](https://goremy.ai/blog/articles/local-ai-models-vs-api-cost-the-real-break-even-math) is built for exactly that.

## Why else run locally: privacy, control, and no vendor lock-in?

Cost isn't the only reason to run local. Self-hosted models keep every token on your own infrastructure, which is the difference between compliance by policy and compliance by architecture for GDPR, HIPAA, and SOC 2 workloads. [12] IBM's widely-cited average cost of a data breach, $4.44 million, is the number compliance teams point to when they explain why "we'll be careful with the API" isn't a strategy. [12]

**Choosing a Local Runtime: Ollama vs llama.cpp vs vLLM**

| Option | Setup Ease | Customization Control | Production Throughput | Best For |
| --- | --- | --- | --- | --- |
| Ollama (recommended) | high | low | low | One-command downloads, model switching, and a local API for a single user |
| llama.cpp | medium | high | medium | Custom quantization, context length, and hardware-specific build flags |
| vLLM | low | medium | high | Batched requests and production throughput behind an internal tool |

_Ratings are relative across these three options, not absolute scores. 'Recommended' reflects the article's guidance for a first run, not the best choice for every workload._

There's a second, quieter benefit. A distilled model you've downloaded and quantized is weights and code sitting on a drive you control, not a subscription that can [reprice or disappear](https://goremy.ai/blog/articles/13b-reason-to-run-local-ai-models-over-cloud-apis) the moment its owner changes strategy. It's worth treating the model as an asset to inventory, like any other internal tool.

## What are the real limits of distillation?

Be realistic about what you're running. The pattern holds across the community: the smaller distills genuinely underperform their teacher on the hardest benchmarks, and running the "not a distilled Qwen or Llama" full model requires hardware most people will never own. [13] The 1.5B model's 28.9 on AIME versus the 32B's 72.6, within the same architecture family, is the clearest evidence that parameter count alone still drives distilled reasoning ability. [1]

That said, distillation quality isn't always a straight line down. DeepSeek's later R1-0528 distill, built on Qwen3-8B, reportedly achieves performance similar to the much larger Qwen3-235B, showing that a good teacher and a well-chosen base model can occasionally punch well above the checkpoint's size. [14]

The practical takeaway: pick a distilled model for the job it's actually suited to, drafting, summarizing, coding assistance, moderate-complexity reasoning, and don't expect it to match the API you were renting on the hardest 10% of tasks. Own it for what it is, not for what the name on the box implies.

## Frequently Asked Questions

### What does it mean for an AI model to be "distilled"?

Distillation means a smaller model is fine-tuned on outputs generated by a much larger "teacher" model, so it learns to imitate the teacher's reasoning style. DeepSeek's distilled checkpoints are Qwen2.5 and Llama3 base models fine-tuned on 800,000 reasoning traces from the full 671B R1 model, not a shrunken version of R1 itself. [1] [2]

### Can a distilled model like DeepSeek-R1-Distill-Qwen-32B actually match the real R1?

On some benchmarks it comes close or even beats comparable proprietary models — it outperforms OpenAI's o1-mini on AIME 2024 and MATH-500 — but independent testing and community consensus agree it falls short of the actual 671B R1 on the hardest reasoning tasks. [1] [9] [13]

### What's the easiest way to run a distilled model locally?

Install Ollama and run a single command like `ollama run deepseek-r1:8b`, swapping the size tag for whatever fits your hardware. It handles the quantized download and gives you a local chat session with no extra setup. [6]

### How much hardware do I actually need?

It depends on the checkpoint size, not on running the "real" model. A distilled 7B model can run as a roughly 4-5GB quantized file on a laptop with no dedicated GPU, while the true 671B DeepSeek-R1 model needs around 700GB of VRAM or RAM and six-figure hardware to serve well. [4]

### When does running a distilled model locally actually save money over an API?

Cost analyses put the self-hosting break-even point around 11 billion tokens a month. Below that volume, per-token API pricing usually wins once you factor in idle GPU time, electricity, and the labor to maintain the setup. [10]

## Sources

1. [DeepSeek-R1 GitHub Repository (README)](https://github.com/deepseek-ai/deepseek-r1) (DeepSeek AI / GitHub)
2. [DeepSeek-R1-Distill-Qwen-1.5B model card](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) (Hugging Face / DeepSeek AI)
3. [DeepSeek-R1 and exploring DeepSeek-R1-Distill-Llama-8B](https://simonwillison.net/2025/Jan/20/deepseek-r1/) (Simon Willison's Weblog)
4. [Hacker News comment thread on DeepSeek distilled models vs. actual R1](https://news.ycombinator.com/item?id=42896812) (Hacker News)
5. [A Survey on Knowledge Distillation of Large Language Models](https://arxiv.org/abs/2402.13116) (arXiv (Xu et al.))
6. [Got DeepSeek R1 running locally - Full setup guide and my personal review](https://www.reddit.com/r/LocalLLaMA/comments/1i6gahy/got_deepseek_r1_running_locally_full_setup_guide/) (Reddit r/LocalLLaMA)
7. [llama.cpp quantize tool README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) (ggml-org / GitHub)
8. [Overview of GGUF quantization methods](https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/) (Reddit r/LocalLLaMA)
9. [How to Benchmark DeepSeek-R1 Distilled Models on GPQA Using Ollama and OpenAI's simple-evals](https://towardsdatascience.com/how-to-benchmark-deepseek-r1-distilled-models-on-gpqa-using-ollama-and-openais-simple-evals/) (Towards Data Science)
10. [Self-Hosted LLM vs API: The $4,200/mo Break-Even Point](https://www.braincuber.com/blog/self-hosted-llms-vs-api-based-llms-cost-performance-analysis) (Braincuber Technologies)
11. [LLM Local vs API Cost Calculator: Monthly Inference Cost](https://agentcalc.com/llm-local-vs-api-cost-calculator) (AgentCalc)
12. [Local LLM Deployment: Privacy-First AI Complete Guide](https://www.digitalapplied.com/blog/local-llm-deployment-privacy-guide-2025) (Digital Applied)
13. [Running Deepseek R1 Locally Not a Distilled Qwen or Llama – Setup and Running Notes](https://digitalspaceport.com/running-deepseek-r1-locally-not-a-distilled-qwen-or-llama/) (Digital Spaceport)
14. [DeepSeek-R1-0528: How to Run Locally](https://unsloth.ai/docs/models/tutorials/deepseek-r1-0528-how-to-run-locally) (Unsloth AI)