How Does SenseNova-Loop-9B-Preview's Looped Transformer Work?

How does SenseNova-Loop-9B-Preview's looped architecture work?

SenseNova-Loop-9B-Preview is a 32-layer multimodal decoder in which one contiguous block of 24 layers (layers 4 to 27) runs twice with shared weights. That gives 56 decoder-layer executions per forward pass while storing only 32 layers of weights. Two residual streams, managed by manifold-constrained hyper-connections (mHC), carry information through the looped section. Attention alternates three linear layers with one full-attention layer. The card lists roughly 9B total parameters and about 14.6B inference parameters.

What does "looped" mean in a transformer?

A standard decoder passes the hidden state through each layer once, and every layer has its own weights. A looped transformer feeds the output of a block of layers back into the same block. Depth goes up, but the parameter count does not.

SenseNova-Loop-9B-Preview does this with a single block. Layers 0 to 3 run first. Then the block of layers 4 to 27, call it B, runs on the hidden state h, and runs again on its own output, so the computation is B(B(h)). Layers 28 to 31 and the output head finish the job.

The arithmetic is simple:

The second pass takes the first pass's output as input, and both passes use the same weights.

Why is it ~9B stored but ~14.6B at inference?

The card gives both numbers: roughly 9B total parameters and roughly 14.6B inference parameters. The gap comes from the repeated block. Its weights are stored once but used twice, so the compute per token looks like a larger model than the checkpoint on disk.

This matters when you plan hardware. Weight memory tracks the stored size. Per-token compute tracks the 56 executions. The card says practical speedups and costs depend on workload, hardware, and serving configuration, and it does not publish latency or throughput figures. Treat any claim about efficiency as unmeasured until the technical report or independent tests arrive.

What is two-stream mHC and why does the loop need it?

Hyper-connections generalize the residual connection. Instead of one residual path, the model keeps several parallel streams and learns how layers read from them, write to them, and mix them. Making the mixing "manifold-constrained" keeps those learned maps in a well-behaved family so the signal stays stable through deep or repeated computation.

Here is what the card documents for this model:

• mHC runs inside the looped section and carries two residual streams around the decoder-layer computations.
• Learned mappings govern how a layer reads from the streams, how stream information is mixed, and how the layer output is added back into the residual state.
• Sinkhorn normalization constrains the stream-mixing map. The documented configuration uses 20 iterations.

Sinkhorn normalization alternately rescales rows and columns of a positive matrix until it is close to doubly stochastic. A doubly stochastic mixing matrix redistributes signal between streams without amplifying or shrinking it overall. That plausibly helps when the same block is applied twice, since a badly scaled map would compound across passes. The card does not give the authors' reasoning or ablations, so read this as the general purpose of the technique, not a documented result for this model.

One practical consequence: the loop range, pass count, and mHC routing parameters are part of the checkpoint's execution configuration. An inference engine must load them and reproduce the same execution order. A stock decoder implementation will not run this model correctly.

How does the 3:1 hybrid attention fit in?

The decoder interleaves linear attention and full attention in a repeating four-layer pattern: three linear-attention layers, then one full-attention layer. Linear attention avoids the quadratic cost of full attention over long sequences. The periodic full-attention layers keep global token-to-token lookup available.

The design detail that makes this work with looping is alignment. The loop covers 24 layers, which is exactly six four-layer groups, and it starts at layer 4, a group boundary. Each pass therefore sees the same attention pattern. Across all 56 executions, that works out to 14 groups of four: 42 linear-attention executions and 14 full-attention executions (my arithmetic from the stated pattern, not a figure from the card).

Full-attention layers use 16 heads with 4 KV heads, a grouped-query setup that shrinks the KV cache compared with full multi-head attention.

What are the core specs?

The FP32 routing parameters are worth noting. The mixing maps stay in higher precision while the backbone runs in BF16.

What is it built for, and how do you run it?

The card positions the model for long-context and agentic workflows: document understanding, coding, and interaction with external tools. Its benchmark chart compares it against other models on BIRD-Dev, WorkBuddy Bench, AIDABench, TeamBench, Terminal Bench 2.1, T³-Banking, AutomationBench, and RaccoonPPTBench. The card presents those results as an image, so this article does not quote scores.

Documented ways to run it:

• Transformers: the checked environment is Transformers 5.18.0 with PyTorch 2.11.0 and CUDA 13.0, loading with trust_remote_code=True. Thinking mode toggles through enable_thinking in the chat template.
• LightLLM: a Docker-based server, shown with four-way tensor parallelism and an OpenAI-style chat endpoint.
• SGLang: requires a build containing the SenseNova Loop implementation from a specific pull request, not a stock release.
• Fine-tuning: an 8-GPU single-node recipe using a torchtitan fork, covering SFT, with DPO and MTP training described in the linked guide.

For faster decoding, MTP (multi-token prediction) is available. Dspark and Dflash2 draft-model methods are listed as training in progress.

Is it ready for production use?

Not yet, by the project's own labeling. The name says Preview, the technical report is TBD, and the Transformers implementation's publication details are still to be announced. SGLang support depends on an unmerged pull request as of the card. The card also recommends the documented deployment environment and configuration for best performance, which suggests behavior may vary elsewhere.

Open questions a builder would want answered: how much of the extra compute translates into quality versus a conventional 9B or 14B model, how the KV cache behaves across two passes, and what real throughput looks like. The card does not answer these.