What Is a Looped Transformer? NVIDIA Dory Explained
A looped transformer reruns a shared block of layers per token to gain depth without more weights. Here is how NVIDIA's Dory does it, and what it costs.
What is a looped transformer?
A looped transformer (also called a latent recurrent transformer) reuses the same block of layers several times for each token instead of running every layer once. The weights stay the same on each pass, so the model gets more computational depth without more parameters or more weight memory. NVIDIA's Dory, a roughly 4B model from its Nemotron labs, is a downloadable example you can run yourself.
TL;DR
- Dory splits its layers into an input block, a looping recurrent block, and an output block, with the input and output blocks running once and only the middle block repeating.
- The layout is 16 input layers, 40 recurrent layers, and 16 output layers, so one loop gives 72 layers of effective depth, two loops give 112, and four loops give 192.
- Each lap re-injects the input block's output, similar to a residual connection in ResNet, but with the same weights every time.
- The output block reads the state after any loop, so the loop count is a dial you set at inference time with no retraining for a specific number.
- Weight memory stays flat, but the KV cache grows with loops in the default mode, and a "last loop" mode trades some of that back.
- Against Gemma 4 26B, more loops win more benchmarks, mostly agentic and long-context tasks, while knowledge-heavy benchmarks stay weak.
How does Dory split its layers?
A standard transformer sends a token through every layer once, from input to output. Dory divides the stack into three parts.
| Block | Layers | Runs |
|---|---|---|
| Input block | 16 | Once |
| Recurrent block | 40 | 1, 2, 4, or more times |
| Output block | 16 | Once |
The recurrent block is the only part that loops. Each pass refines the token's internal representation before the output block turns it into a prediction. The model has about 3.97 billion parameters and supports up to 256K of context. NVIDIA released it under a license that permits commercial use, with BF16, MXFP8, and NVFP4 versions plus GGUFs.
How does residual re-injection keep the loop on track?
Repeating a block with no guardrails could let the representation drift. Dory avoids that by adding the input block's output back in on every lap. This resembles a ResNet, where the original input is added to a transformed version of itself.
The difference is the weights. A ResNet uses different weights in each block. Dory applies the same recurrent weights on every lap. Each pass reminds the block what token it is working on, a bit like rereading the question before another attempt at answering.
How do you get more depth without more parameters?
Unroll the loop and you see a much deeper network. The weights in the repeated block are identical, but the compute path is longer.
| Loops | Effective depth |
|---|---|
| 1 | 72 layers |
| 2 | 112 layers |
| 4 | 192 layers |
The weights in GPU memory stay at about 4B parameters regardless. In the demo, the four-loop setting did roughly the compute of a dense model around 9B in size while keeping the weight footprint of the 4B model. NVIDIA calls this elastic inference compute: compute per token rises with the loop count, but the weights do not.
This works as a practical dial because the output block was trained to read the state after any loop. One, two, or four loops are all valid configurations of one checkpoint. You pick the loop count when you serve the model.
What does looping cost?
Weight memory is flat, but other costs are not.
- Speed. More loops mean more compute per token, so decoding slows down. The one-loop version is the fastest, and the four-loop version took noticeably longer wall time in the demo.
- KV cache. In the default per-loop mode, every loop keeps its own cache, so memory grows with the loop count. A last-loop mode keeps one cache per recurrent attention layer and overwrites the earlier ones. NVIDIA says accuracy and output length stay close to per-loop mode, which makes it a candidate for long-context serving with many concurrent users.
- Serving friction. In vLLM, you currently set the loop count when you start the server. Running several loop settings means running several instances. The creator who tested it suggested a router that sends harder questions to a higher-loop instance, and speculated that serving engines may handle this natively later.
Does more looping actually improve results?
Sometimes, but not uniformly. In the creator's side-by-side runs of one, two, and four loops, the four-loop version often produced better answers, yet many prompts gave the same quality across settings. More loops also did not reliably produce more tokens. The creator was clear that this is not the same as turning up a reasoning setting.
A custom PyTorch port that reads internal state after each lap showed the model growing more confident in specific tokens as loops increased. One example went from 15% on the first loop to 68% on the second, then 74% and 75% on the next two. That is one illustrative example, not a general result.
On NVIDIA's comparison against Gemma 4 26B, the picture looks like this:
| Dory loops | Benchmarks won (of 10) |
|---|---|
| 1 | 1 |
| 2 | 4 |
| 4 | 5 |
Gemma 4 26B is a mixture-of-experts model with about 4B active parameters per token, so per-token compute is in the same range. Dory's wins cluster in agentic work, instruction following, long context, and SWE-bench Verified. It loses on knowledge-heavy tests such as GPQA and Humanity's Last Exam. That fits the design. Looping adds compute, but it does not add the stored facts that a larger total weight count provides.
Why is this a different way to scale?
Most current reasoning gains come from longer chains of thought: more tokens generated before the answer. A looped model spends extra compute inside each token step instead. It thinks harder per token rather than writing more tokens, and the deployer chooses how hard.
For memory-constrained hardware, this is attractive. You get depth without storing a larger set of weights. The tradeoff is that knowledge capacity stays tied to the parameter count.
For deployment today, the guidance from the source is to default to two loops, which NVIDIA also appears to recommend, and keep a four-loop instance for harder agentic tasks. Dory is mostly a research release, and the model card is the main documentation.
Is GPT-6 a looped transformer?
Nobody has confirmed it. The source mentions rumors that GPT-6 uses looping, with different loop counts for different model sizes. Treat that as speculation. What is verifiable is that a looped model exists in open weights, so you can test the idea yourself.
Frequently Asked Questions
What is the difference between a looped transformer and a normal transformer?
A normal transformer passes each token through every layer exactly once. A looped transformer repeats a middle block of shared-weight layers several times. Depth increases, but the parameter count does not.
Does a looped transformer need less memory?
It needs less weight memory than a dense model with the same effective depth, since the repeated layers share weights. KV cache is a separate matter. In per-loop mode it grows with the number of loops, and last-loop mode reduces that.
Do I have to retrain Dory for a specific number of loops?
No. The output block was trained to read the state after any loop, so you choose the loop count at inference. Currently, with vLLM, you set it when starting the server.
Is Dory better than Gemma 4 26B?
It depends on the task. By NVIDIA's composite score it comes out ahead, and at four loops it wins five of ten benchmarks. It does worse on knowledge-heavy tests, where the larger Gemma model has more stored capacity.
How many loops should I use?
The source suggests two loops as a default, with four loops reserved for harder agentic work. One loop is fastest, but it won only one of the ten benchmarks in the comparison.
How we source, write, and correct our articles. Editorial standards
