How to Run Arete-27B Locally: VRAM, Quantization, Context
Run Nous Research's Arete-27B locally: VRAM estimates by quantization, context limits from its 64k training, and runtime settings to start with.
For Engineering and IT
How do you run Arete-27B locally and what hardware does it need?
Arete-27B is a 27B-parameter supervised fine-tune of Alibaba's Qwen3.8-27B from Nous Research. At 16-bit precision the weights alone take roughly 54 GB, so you need a 48 GB to 80 GB GPU setup. An 8-bit quant needs about 27 GB, and a 4-bit quant fits in roughly 14 to 18 GB before context overhead. The model card lists no official hardware requirements, so these are arithmetic estimates, not vendor numbers.
TL;DR
- Arete-27B has 27B parameters, so weight memory is about 2 bytes per parameter at bf16, 1 byte at 8-bit and around half a byte at 4-bit.
- The card gives no VRAM figures and no quantized releases, so treat every memory number here as a rule-of-thumb estimate and check your runtime's actual usage.
- The model was trained at 64k context, so that is the safest ceiling for prompt plus output length.
- Training used text-only data and froze the vision encoder, so expect text work to be the supported use case.
- The MTP head was disabled during training, so do not assume multi-token-prediction speedups work well without testing.
- Because quantization can shift fine-tuned behavior, rerun the public evaluation if the viewpoint change matters to your use case.
What do the model card specs tell you about running it?
The card is a research and training document, not a deployment guide. It still gives several facts that affect how you run the model.
| Spec | Value | Why it matters locally |
|---|---|---|
| Base model | Qwen3.8-27B | Use a runtime that supports that architecture |
| Parameter scale | 27B | Sets weight memory |
| Post-training | SFT only, no RL, no preference optimization | Same architecture as the base, no architectural modification |
| Training data | Text-only | Vision encoder was frozen, not trained |
| Training context | 64k | Practical context ceiling |
| Training precision | bf16 compute, fp32 master weights | bf16 is the natural full-precision format |
| MTP head | Disabled during training | Treat MTP-based speculative decoding as untested |
The card says no architectural modification was made. In practice, any stack that loads Qwen3.8-27B should load Arete-27B, though the card does not promise this. The weights live at NousResearch/Arete-27B on Hugging Face.
How much VRAM does Arete-27B need?
Weight memory is parameters times bytes per parameter. The KV cache and runtime buffers come on top, and the card does not publish the attention layout, so I can't give an exact cache figure.
| Precision | Approx. weight size | Realistic hardware |
|---|---|---|
| bf16 / fp16 | ~54 GB | One 80 GB GPU, or two 24 to 32 GB GPUs split, or a large unified-memory Mac |
| 8-bit | ~27 GB | One 32 GB or 48 GB GPU |
| 4-bit | ~14 to 18 GB | One 24 GB GPU, with room for moderate context |
Add several GB for the KV cache and buffers. Long contexts cost more, and at tens of thousands of tokens the cache can become a major part of memory use. If you run out, shorten the context first, then drop to a smaller quant.
The training setup used tensor parallel 4. That describes how Nous fine-tuned the model with long 64k sequences and fp32 optimizer states. It says nothing about what inference needs.
Is there a GGUF or quantized version of Arete-27B?
The model card does not mention GGUF, AWQ, GPTQ or any other quantized release. If you see community quantizations, check who made them, which base checkpoint they used and which quantization method they applied. The card's evaluated checkpoint is step 1,202, so make sure any conversion came from that checkpoint.
If you quantize it yourself, the usual route for local use is converting to GGUF with llama.cpp tooling, provided it supports the Qwen3.8 architecture. For GPU serving, AWQ or GPTQ through vLLM-style stacks is common. Check your runtime's supported-architecture list before you download 54 GB of weights.
The vision encoder is a wrinkle. The base model includes one, and Nous froze it during training. Some converters handle the vision tower separately or drop it. For a text-only workflow, that is fine, but verify your chosen file works for plain chat.
How do you set up a runtime?
Pick the runtime by hardware.
- llama.cpp or a wrapper around it: best for consumer GPUs, CPU offload and Apple silicon. Needs a GGUF.
- vLLM or SGLang: best for a dedicated GPU box and multiple concurrent requests. Loads the Hugging Face weights directly.
- Transformers: simplest for testing, slowest for serving.
A generic vLLM launch looks like this. Adjust the length to your memory:
vllm serve NousResearch/Arete-27B \
--max-model-len 32768 \
--tensor-parallel-size 1
Raise --tensor-parallel-size if you split across GPUs. Start with 32k context and move toward 64k only once you know how much memory remains. If your installed version does not recognize the architecture, update the runtime before troubleshooting anything else.
What context length should you use?
The card says training ran at 64k context with no sequence packing. Treat 64k as the upper bound for prompt plus generated tokens. The card does not say how the model behaves beyond that, and the base model's own limit is not stated in these materials. Pushing past the trained length is an experiment, not a supported mode.
Most local setups will not want 64k anyway. The evaluation in the card capped outputs at 32k tokens, and typical chat needs far less. A 16k to 32k window is a sensible default on a 24 GB card with a 4-bit quant.
What sampling and thinking settings should you start with?
The card reports results with thinking off. Arete-27B and the starting model ran at temperature 0, while the comparison rows used 0.6. At temperature 0.6 with thinking off, Arete-27B scored 2.88 on the fact metric, against 2.98 at temperature 0. That is a small gap, so either setting is reasonable for general use.
The card does not report results with thinking on, so you are on your own there. If you use a reasoning mode, test it against your own prompts first.
Can you use the MTP head for faster decoding?
Multi-token prediction heads can drive speculative decoding for speed. Nous disabled the MTP head during training. The card doesn't say whether the head ships in the checkpoint, or whether it still behaves well after the other weights changed. If your runtime offers MTP-based speculation, benchmark it against plain decoding, and compare outputs. A draft head that was not trained alongside the fine-tuned weights may accept fewer tokens, which erases the speedup. Plain decoding is the safe default.
Will quantization change what Arete-27B was trained to do?
Possibly. The fine-tune was a modest one: roughly 52.8 million tokens over two epochs at a peak learning rate of 1e-5. Its behavioral shift is measured, not guaranteed to survive aggressive compression. The card reports that on its propaganda evaluation, the fact score rose from 2.32 to 2.98 out of 5, and answers containing propaganda fell from 22% to 0.9%.
Nous published the 803-prompt Propaganda_Eval dataset on Hugging Face. You can run your quantized build against it and compare to the card's numbers. Keep the decoding settings matched, since the card notes that rows were not all run with identical settings. A drop in fact score or a rise in the propaganda rate at 4-bit would tell you to try 6-bit or 8-bit.
General capability looks intact at full precision per the card. Hermes Bench came in at 65.60% against 65.77% for the base, and Terminal-Bench-2.1 at 70% against 67%. Nous says it takes no credit for the Terminal-Bench gain. Those figures describe the unquantized model on Nous's hardware.
Who should run this model?
The card describes two audiences: users who want the viewpoint change for general assistance and agentic work, and researchers studying how post-training shifts a model's values. It is not a claim of having fixed every behavior tied to the original viewpoint. Test it on your own tasks before relying on it.
Frequently Asked Questions
Can Arete-27B run on a single 24 GB GPU?
Likely yes, with a 4-bit quant, since the weights should come to roughly 14 to 18 GB. That leaves some room for context. The card publishes no official numbers, so confirm in your runtime and keep the context window modest.
Does Arete-27B support images?
The card describes text-only post-training and says the vision encoder was frozen. It makes no claims about image performance. Use it for text, and test carefully if your runtime exposes multimodal input.
What context length is safe?
Stay within 64k tokens, the length it was trained at. For local hardware, 16k to 32k is usually the practical range. Longer contexts raise KV cache memory use.
Is there an official GGUF release?
The model card does not list one. Check the Hugging Face page for the current state of the repo, and verify the provenance of any third-party quantization before using it.
Do I need the same settings Nous used?
No, but they are a good baseline. The card's results used thinking off, with temperature 0 for Arete-27B and the base model, and 0.6 for several comparison models. Start there and adjust for your task.
Topics
Useful for
How we source, write, and correct our articles. Editorial standards
