---
title: "How to Run Arete-27B Locally: Hardware and Quantization"
url: "https://goremy.ai/blog/articles/how-to-run-arete-27b-locally"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-10T12:00:12.431Z"
updated: "2026-11-10T17:00:52.413Z"
topic: Local AI
description: "Hardware needed to run Nous Research's Arete-27B locally: VRAM by quantization level, the 64k training context, text-only limits, and runtime choices."
---

# How to Run Arete-27B Locally: Hardware and Quantization

> Hardware needed to run Nous Research's Arete-27B locally: VRAM by quantization level, the 64k training context, text-only limits, and runtime choices.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-10

## What hardware do you need to run Arete-27B locally?

Arete-27B is a dense 27B-parameter model, so weights alone take roughly 54 GB at 16-bit, about 27 GB at 8-bit, and around 14 to 17 GB at 4-bit. Add memory for the context cache and runtime overhead. A 24 GB GPU is a realistic floor for a 4-bit quant with moderate context. A 48 GB card handles 8-bit or long contexts more comfortably. The model card gives no official hardware figures, so these are estimates from parameter count.

## TL;DR

- Arete-27B is a **27B-parameter SFT fine-tune of Qwen3.8-27B**, with no architectural changes, so it should size like its base model.
- Weight memory is about **2 bytes per parameter at 16-bit, 1 at 8-bit, and roughly 0.5 to 0.6 at 4-bit**, before cache and overhead.
- The model was trained at **64k context**, so treat 64k as the sensible ceiling and expect cache memory to grow with it.
- Post-training data was **text-only** and the vision encoder was frozen, so this is a text model for practical purposes.
- Nous's own evaluations ran with **thinking off**, which also saves tokens and latency on local hardware.
- The card does not list quantized builds, so check the Hugging Face repo for GGUF or other formats before assuming they exist.

## What exactly are you running?

Nous Research post-trained Alibaba's Qwen3.8-27B to change how it answers politically sensitive topics. The goal was to reduce refusals, deflections and state-framed answers. The training was supervised fine-tuning only. The card says there was no RL, no preference optimization and no architectural modification.

That last point matters for local use. Because the architecture is unchanged, anything that runs Qwen3.8-27B should load Arete-27B the same way. Tokenizer, chat template and runtime support should carry over from the base model, though you should confirm against the repo files.

Training details that affect how you run it:

| Detail | Value | What it means locally |
| --- | --- | --- |
| Parameters | 27B | Sets the weight memory floor |
| Training context | 64k | Practical context ceiling |
| Training data | Text-only | No new multimodal skills |
| Frozen weights | Vision encoder | Vision parts were not tuned |
| Disabled in training | MTP head | Don't count on speculative decoding from it |
| Training precision | bf16 compute | bf16 is the natural full-precision format |

## How much VRAM does each quantization level need?

Weight memory is parameters times bytes per weight. These are weights-only estimates for 27B parameters:

| Precision | Approx. weights | Typical fit |
| --- | --- | --- |
| bf16 / fp16 | ~54 GB | 80 GB card, or two 48 GB or 24 GB cards split |
| 8-bit | ~27 GB | 32 GB or 48 GB card |
| 6-bit | ~21 to 23 GB | 32 GB card, tight on 24 GB |
| 4-bit | ~14 to 17 GB | 24 GB card with room for context |

Real files run a bit larger than the naive math because some tensors stay at higher precision and quant formats store scales. Budget a few extra gigabytes for the runtime's compute buffers.

The context cache comes on top. Its size depends on layer count, attention head layout and cache precision. The model card doesn't give those numbers, and I won't guess them. Measure instead. Load the model at a small context, note memory use, then raise the context in steps and watch the growth.

One real-world data point from a separate demo: a creator ran the base Qwen 3.8 27B in Ollama on a single 48 GB NVIDIA RTX card alongside a small embedding model (Nomic Embed). That setup worked for a tool-using application. It shows that 48 GB is comfortable for this model class, though the demo doesn't say which quantization was used.

## Which quantization should you pick?

Pick by the memory you have, then spend what is left on context.

1. **24 GB GPU:** Use a 4-bit quant. Keep context modest at first (8k to 16k) and grow it while watching memory.
2. **32 GB GPU:** A 6-bit quant or a good 4-bit with long context both work. Choose based on whether you value fidelity or context length.
3. **48 GB GPU:** 8-bit gives near-lossless quality with real room for a long cache. This is the comfortable tier.
4. **80 GB or multi-GPU:** Run bf16 if you want to match the evaluated checkpoint most closely.
5. **Apple silicon or CPU:** Unified memory or system RAM works for the same sizes, but token speed will be lower than on a discrete GPU.

Quantization can shift behavior slightly. Nous measured the full-precision checkpoint, so if you care about the values and factual behavior the card reports, test your quant on your own prompts. The evaluation set (`Propaganda_Eval`, 803 prompts) is public, so you can run it against your quantized build and compare with the published numbers.

## How long a context can you use?

Training ran at 64k context, with about 52.8 million tokens over two epochs and no sequence packing. That makes 64k the length the fine-tune actually saw. The card does not make claims about longer contexts, so stay at or under 64k if you want behavior close to what Nous tested.

Longer context costs memory in two ways: the cache grows, and prompt processing slows. If you are VRAM-limited, you have a few options:

- Lower the context window to what the task really needs.
- Use a quantized cache if your runtime supports it, and check output quality afterward.
- Drop one quantization tier to free memory for cache.

## Should you run it with thinking on or off?

Nous reported its headline numbers with thinking off, at temperature 0 for the main runs and 0.6 in a secondary run. The evaluation capped output at 32k tokens. If you want results comparable to the card, mirror those settings.

Locally, thinking mode has a cost. Reasoning tokens eat time and cache. In the Ollama demo mentioned earlier, the creator called the base Qwen 3.8 27B "quite an overthinker," with delayed replies in chat. For quick assistant use, thinking off is faster. For hard multi-step problems you can turn it on, but expect longer waits and more memory use. How you toggle it depends on your runtime and chat template.

## Which runtime should you use?

The card doesn't prescribe one. These are the usual options for a model of this shape:

- **llama.cpp or Ollama:** Easiest for GGUF quants and single-GPU setups. Ollama also exposes an OpenAI-style local API, which many tools can point at.
- **LM Studio:** A desktop front end for local GGUF models with a built-in server.
- **vLLM or SGLang:** Better for serving multiple users or high throughput, typically with full-precision or other supported quantized formats.

If you plan to use Arete-27B in an agent or app, check that the model and its template support tool calling. One tutoring app shown running on local Ollama required a tool-use-capable model, and it verified that in the model's listed capabilities. Nous reports Terminal-Bench-2.1 at 70% for Arete-27B versus 67% for the base model, but it says it does not claim credit for that difference. Hermes Bench was essentially flat (65.60% versus 65.77%), which Nous reads as noise.

## What about the MTP head and vision?

The multi-token-prediction head was disabled during training. Don't plan on relying on it for speculative decoding. Whether it is present in the released weights isn't stated, and most local runtimes ignore it anyway.

The vision encoder was frozen and the post-training data was text-only. The card says nothing about image behavior, so treat Arete-27B as a text model. If a runtime exposes image input for the Qwen base, test it yourself before depending on it.

## Is Arete-27B right for your use case?

Nous describes it as intended for general assistance and agentic work where users want the shifted viewpoint, and for research on how post-training changes values. On its own evaluation, the fact score rose from 2.32 to 2.98 out of 5, and answers containing propaganda dropped from 22% to 0.9%. Those results come from Nous's own benchmark and an LLM judge (Grok 4.7), and Nous says it did not remedy every behavior. Test it on your own topics before relying on it.

## Frequently Asked Questions

### Can I run Arete-27B on a single 24 GB GPU?
Probably, with a 4-bit quantization. Weights at 4-bit land around 14 to 17 GB, leaving a few gigabytes for the context cache. Start with a short context and increase it while watching memory.

### Is there an official GGUF of Arete-27B?
The model card doesn't mention one. Check the Hugging Face repository and its model tree for quantized versions. If you convert one yourself, verify output quality against the full-precision model on your own prompts.

### What context length should I set?
Up to 64k, which matches training. Shorter is fine and saves memory. The card makes no claims about performance beyond 64k.

### Do I need the vision components?
No for text work. The fine-tune used text-only data and kept the vision encoder frozen. Treat the model as text-only unless you test image input yourself.

### Will quantization change its behavior?
It can, slightly. Nous's reported results come from the unquantized checkpoint. You can run the public 803-prompt Propaganda_Eval set on your quant to check.