What Is NVIDIA Nemotron Dory 1.0 4B? Looped Model Explained

What is NVIDIA Nemotron Dory 1.0 4B?

Dory1.0-4B (NVIDIA-Nemotron-Labs-Dory1.0-4B) is an open-weight, 4-billion-parameter looped language model from NVIDIA. Instead of pushing a token through a long stack of unique layers once, it runs a shared block of layers several times. More loops mean more compute per token, with no extra parameters. It supports up to 256K tokens of context, handles English, and ships under the OpenMDW-1.1 license.

How does a looped language model work?

A standard transformer has N distinct layers, and each token passes through each one once. Depth equals parameter count. A looped model breaks that link. It reuses the same weights multiple times, so effective depth can grow while the parameter count stays put.

Dory splits its layer stack into three groups:

The recurrent block sits between input and output blocks, and the model card says the input is reinjected into every loop. That reinjection keeps the original signal available to each pass instead of letting it drift.

The arithmetic is simple. One loop gives 36 layer applications (8 + 20 + 8). Two loops give 56. Four loops give 96. Parameters stay at 4B throughout. You pay in compute, not in weights.

The "latent" part matters. The extra computation happens inside the hidden state, before a token is emitted. That differs from chain-of-thought, where the model spends compute by writing more tokens.

What is normalized-GPT and why NVFP4?

Dory builds on a normalized-GPT (nGPT) backbone. It also interleaves sliding-window attention (SWA) layers with full-attention layers, a common way to keep long-context cost down.

According to the model card, Dory is pre-trained natively in NVFP4, NVIDIA's 4-bit floating-point format, and then continues training in BF16. The released checkpoint is BF16. Other formats on offer are MXFP8, NVFP4 and Q5_K_M (a GGUF export for llama.cpp). The card notes that its benchmark numbers are for BF16, not the quantized exports.

What does elastic compute mean in practice?

One checkpoint serves several compute budgets. You change how many times the shared block runs, and the model gets slower and more accurate, or faster and cheaper. No retraining and no second model.

That is useful if your workload is mixed. A routing or classification call might run at one loop. A hard coding task might run at four. In vLLM the loop count is a config override (n_recurrent_loops), so you set it per deployment. The model card does not describe changing it per request, so check your serving stack before assuming that.

How does last-loop KV caching save memory?

Looping normally shares parameters but not the cache. Each loop produces its own keys and values, so a naive implementation stores one KV cache per loop. Memory grows with every loop you add.

Dory supports two modes:

• per_loop: a separate KV cache for each recurrent loop. Cache memory grows with loop count.
• last_loop: one cache per recurrent attention layer. Each loop overwrites the previous loop's entries, so only the final loop's keys and values remain.

With last-loop caching on, NVIDIA says Dory carries no KV-cache overhead relative to an equivalent non-looped model, at any inference loop count. That frees capacity for longer contexts or more concurrent requests. In NVIDIA's experiments, accuracy with last-loop caching stays close to per-loop mode, with nearly unchanged output lengths. "Close" is not "identical," so test on your own tasks.

How many loops should you use?

NVIDIA recommends two loops as the starting point. Most of the accuracy gain comes from the second loop, with smaller gains after that.

• One loop when minimizing compute per token is the priority.
• Two loops as the balanced default.
• Four loops when accuracy matters more, especially for agentic work like Terminal-Bench and SWE-bench.

The llama.cpp instructions allow loop counts of 1, 2, 3 or 4. Because more loops can also shorten outputs, NVIDIA suggests measuring end-to-end latency and cost on your workload rather than judging by per-token speed alone.

How does Dory1.0-4B perform on benchmarks?

NVIDIA measured accuracy under a consistent harness (NeMo Gym and the Nemo Evaluator SDK) and compared Dory against Qwen3.5-4B, Qwen3.5-9B, Gemma 4 26B A4B and Qwen3.6-35B-A3B. A selection from the model card:

† Reported score.

Three patterns stand out.

1. Looping helps a lot on agentic and coding tasks. SWE-bench Verified rises from 50.3 at one loop to 61.8 at four, which edges past Qwen3.5-9B (61) in this table. Terminal-Bench 2.1 climbs from 12.5 to 22.2.
2. Instruction following and long context look strong. At four loops Dory scores 79 on IFBench and 73 on AA-LCR, ahead of every listed comparison model.
3. Knowledge and hard science reasoning lag. GPQA Diamond, HLE and AA-Omniscience all trail the comparison models. AA-Omniscience barely moves with loops (about 10 at every setting). That fits the idea that extra compute helps reasoning and procedure, not stored facts. A 4B model has limited room for facts, and looping does not add parameters.

Also note that the gap between two and four loops is often small, and on GPQA Diamond and HLE four loops scores slightly below two. The larger models still win on Terminal-Bench 2.1 and most knowledge tests.

How do you run Dory1.0-4B?

Support is not yet in released builds of the main inference engines, according to the model card. vLLM support is under review in a pull request, and llama.cpp support lives in a separate pull request. Until releases include them, you build from those sources.

For vLLM, NVIDIA's example serves the BF16 model with two loops and per-loop caching:

vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
    --hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
    --max-model-len 262144 \
    --enable-prefix-caching \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

To change loops, edit n_recurrent_loops. To cut KV memory, set recurrent_kv_cache_mode to "last_loop". If you are memory-constrained, lower --max-model-len from the validated 256K.

For local use on Apple Silicon, the card documents a llama.cpp build and a Q5_K_M GGUF of about 3.67 GB. The loop count is set with an --override-kv flag (dory.num_loops). The GGUF is quantized, so expect results that differ somewhat from the BF16 benchmark table.

Is a looped model worth using?

It depends on what limits you. Looping suits cases where memory is tight but you can spare compute: edge devices, many concurrent users, or long contexts. A 4B model with a flat KV cache is attractive there, and the ability to turn a loop dial is a useful knob.

It is a weaker fit if your task is mostly factual recall, where the benchmark gaps are widest. It also carries a practical cost today: you depend on pull requests rather than stable releases, and the model card lists English only.

The token-efficiency claim deserves your own measurement. NVIDIA shows a chart of output tokens falling as loop count rises, but each loop costs compute, so real latency and cost depend on your hardware and prompts.