---
title: "How Do You Run NVIDIA Dory Locally? Loops, KV Cache, Formats"
url: "https://goremy.ai/blog/articles/how-to-run-nvidia-dory-locally-vllm"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-03T18:20:05.720Z"
updated: "2026-11-03T18:20:05.720Z"
topic: Open Model Run Guides
description: "Run NVIDIA's 3.97B Dory looped model locally: pick a loop count in vLLM, choose per-loop or last-loop KV cache, and pick BF16, MXFP8, NVFP4 or GGUF."
---

# How Do You Run NVIDIA Dory Locally? Loops, KV Cache, Formats

> Run NVIDIA's 3.97B Dory looped model locally: pick a loop count in vLLM, choose per-loop or last-loop KV cache, and pick BF16, MXFP8, NVFP4 or GGUF.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-03

## How do you run NVIDIA Dory locally?

Dory is a 3.97B-parameter looped transformer from NVIDIA's Nemotron labs, with up to 256K context. You download the weights in BF16, MXFP8 or NVFP4 form, with other GGUF builds also released, and serve them, for example with vLLM. The key detail is that the loop count is fixed when the server starts. Pick two loops as the default, and run a separate four-loop instance if you need it for harder agentic tasks.

## TL;DR

- Dory has **3.97B parameters and up to 256K context**, and ships under NVIDIA's Open MDW license, which the creator who tested it described as allowing commercial use.
- Weights come in **BF16, MXFP8 and NVFP4**, plus GGUF builds, so you can match the format to your hardware.
- You set the **loop count when vLLM starts**, so each loop setting needs its own server instance for now.
- **Two loops is the sensible default**, with four loops held back for harder agentic work.
- Loops add compute per token but **do not add weight memory**. The KV cache is where memory grows.
- A **last-loop KV cache mode** keeps cache size flat as loops increase, which matters for long context and many concurrent users.

## What is Dory and why does the loop matter?

A standard transformer runs every layer once. Dory splits its layers into three blocks: a 16-layer input block, a 40-layer recurrent block, and a 16-layer output block. The input and output blocks run once per token. The recurrent block runs repeatedly, reusing the same weights each pass.

Each lap re-adds the input block's output to the recurrent state, a bit like a residual connection in a ResNet. The difference is that the weights are shared across laps instead of being new layers.

The output block is trained to read the state after any loop count. That means one, two or four loops are all valid configurations of the same checkpoint. NVIDIA calls this elastic inference compute. You pick the depth at deployment, with no retraining.

Effective depth grows with the loop count:

| Loops | Effective layer depth |
|---|---|
| 1 | 72 |
| 2 | 112 |
| 4 | 192 |

The weights in GPU memory stay at roughly 4B parameters throughout. You trade extra compute per token, and slower decoding, for more depth.

## Which weight format should you download?

NVIDIA released several formats. The model card is the authority on exact file sizes and hardware requirements, so check it before you provision anything.

| Format | Typical use |
|---|---|
| BF16 | Reference quality, simplest to serve, largest footprint |
| MXFP8 | Lower memory with 8-bit weights, on hardware that supports the format |
| NVFP4 | Smallest footprint, aimed at NVIDIA hardware with FP4 support |
| GGUF | Community-style local runners and llama.cpp-type workflows |

The training history is relevant here. Dory was pre-trained on 6 trillion tokens, and the first 4.5 trillion were done natively in NVFP4. NVIDIA then switched to 16-bit for 300 billion tokens of context extension, followed by SFT and reinforcement learning with GRPO. A 4-bit format is therefore not a bolt-on afterthought for this model. I haven't seen quality numbers per format in the source material, so test your own workload rather than assuming parity.

## How do you choose the loop count in vLLM?

You choose it at server start. That is the current limitation. One vLLM process runs one loop setting, so comparing one, two and four loops means running three instances of the same weights. Check the model card for the exact launch option your vLLM version expects.

A practical way to work with this:

1. Start with a two-loop instance as your default. The creator who tested Dory noted this is also what NVIDIA recommends.
2. Add a four-loop instance for hard agentic tasks or problems that need more thinking.
3. Put a simple router in front if you want hard requests to go to the deeper instance automatically.
4. Measure tokens per second and answer quality on your own prompts before committing.

Do not treat more loops as a reasoning dial. In testing, the four-loop version often produced better answers than one loop, but not always. Many prompts got equivalent quality at every setting, and output token counts did not reliably rise with loops. The clearest reliable effect is speed: one loop decodes fastest, and four loops is noticeably slower in wall time.

## How does the KV cache behave with more loops?

Weights stay constant, but the KV cache does not. There are two modes:

- **Per-loop mode.** Every loop keeps its own cache. More loops means a bigger KV cache per sequence.
- **Last-loop mode.** Each recurrent attention layer keeps one cache and overwrites the old ones. Going from one loop to four does not change the cache size.

NVIDIA says accuracy and output length stay close to per-loop mode when you use last-loop. The source gave a rough illustration at a 32K context with two loops, where per-loop needed about 2 GB per sequence, but the comparison figure for last-loop was garbled in the material I had, so I won't quote it. The direction is clear, though.

Last-loop mode is worth trying if you serve long contexts to many concurrent users. KV cache, not weights, will be your limiting factor there. For a single user at modest context, per-loop mode is the safer starting point because it is the reference behavior. Compare the two on your own tasks before switching.

## How does Dory compare with Gemma 4 26B?

NVIDIA's composite score puts Dory ahead of Gemma 4 26B. Gemma 4 26B is a mixture-of-experts model with about 4B active parameters per token, so per-token compute is roughly comparable. The breakdown by loop count, across 10 benchmarks, looks like this:

| Dory loops | Benchmarks won vs Gemma 4 26B (of 10) |
|---|---|
| 1 | 1 |
| 2 | 4 |
| 4 | 5 |

The wins cluster in agentic work, instruction following, long context and SWE-bench Verified. The losses are in knowledge-heavy tests such as GPQA and Humanity's Last Exam. That fits the design. Looping adds depth, not stored knowledge, and a 4B-weight model has less room for facts than a 26B MoE.

If your workload is tool use, coding agents or long documents, Dory is a reasonable candidate. If it is closed-book factual Q&A, expect it to trail larger-weight models.

## What does the Open MDW license allow?

NVIDIA released Dory under its Open MDW license. The creator who tested the model said it can be used commercially. I did not have the license text, so read it directly on the model card before shipping a product, especially for redistribution or fine-tuned derivatives.

## Is Dory ready for production?

Treat it as a research release. The model card is the main documentation, and there is no accompanying blog walkthrough. The fixed-at-startup loop count is awkward for operations, since you need multiple instances to offer multiple depths. It would be natural for serving stacks to eventually pick loop depth per request, but that is speculation, not something the sources confirm.

The idea is still worth testing. Dory scales compute per token rather than generating longer chains of thought, and you control how much when you deploy.

## Frequently Asked Questions

### How many loops should I use by default?
Two. NVIDIA's recommendation, as relayed by the creator who tested the model, is two loops as the default. Reserve four loops for harder agentic tasks where the slower decoding is worth it.

### Does more loops mean more VRAM for weights?
No. The weights stay at about 4B parameters regardless of loop count. What grows is compute per token and, in per-loop KV cache mode, the cache.

### Can I change the loop count without restarting vLLM?
Not according to the available material. You set the loop count when starting vLLM, so different loop settings need separate instances. A router in front of those instances can send hard requests to the deeper one.

### Is Dory better than Gemma 4 26B?
Mixed. It wins on NVIDIA's composite score and on agentic, instruction-following and long-context benchmarks, with more wins as loops increase. It loses on knowledge-heavy benchmarks such as GPQA and Humanity's Last Exam.

### Which quant format is best?
The sources give no per-format quality comparison. BF16 is the reference, NVFP4 and MXFP8 reduce memory on supporting hardware, and GGUF suits local runners. Run your own evaluation before choosing.