How Does SenseNova-Loop Get More Depth Without More Parameters?

How does a looped transformer like SenseNova-Loop get more depth without more parameters?

It runs the same block of layers twice. SenseNova-Loop-9B-Preview stores 32 distinct decoder layers, but layers 4 through 27 are applied two times in a row with shared weights. That gives 56 layer executions per forward pass. The second pass takes the first pass's output as input, so the computation is B(B(h)). The model stores about 9B parameters but behaves like roughly 14.6B at inference, because the repeated block's weights are not duplicated.

What is the looped decoder in SenseNova-Loop?

The decoder has 32 layers, numbered 0 to 31. The forward pass goes like this:

1. Layers 0 through 3 process the input once.
2. The block of layers 4 through 27 (24 layers) runs for the first of two passes.
3. The same block runs again on its own output.
4. Layers 28 through 31 and the output head finish the job.

The arithmetic: 4 + 24 + 24 + 4 = 56 layer executions, drawn from 32 sets of weights. The card writes the repeated block as B and its input as h, so the looped section computes B(B(h)).

The idea behind looped transformers is that depth, in the sense of how many sequential transformations a token passes through, can matter more than raw parameter count for some tasks. Reuse buys depth cheaply in storage terms. You pay for it in compute.

Why are the stored and inference parameter counts different?

The card gives "~9B total parameters" and "~14.6B inference parameters." The second number counts the repeated block's weights once for every pass that uses them. It does not equal 9B times 56/32 (which would be about 15.75B). That is plausible, since components that run once, such as the embeddings, are not repeated. The card does not break the numbers down, so treat that explanation as an inference.

The practical split:

So a looped model is not free. It fits in the memory of a 9B-class model but spends compute closer to a larger one. The card also notes that practical speedups depend on workload, hardware, and serving configuration.

What does two-stream mHC do?

Standard transformers carry one residual stream: each layer adds its output to a running state. Hyper-connections widen that into several parallel streams and learn how layers read from them, mix them, and write back. Manifold-constrained hyper-connections (mHC) add a constraint so the mixing stays well behaved.

In SenseNova-Loop, mHC is active inside the looped section with two streams. Per the card, learned mappings govern three things:

• how a layer reads from the streams,
• how stream information is mixed,
• how the layer's output is added into the residual state.

The stream-mixing map is constrained with Sinkhorn normalization, and the documented configuration uses 20 iterations. Sinkhorn iteration alternately normalizes rows and columns of a matrix. In the mHC literature, this is used to keep the mixing matrix close to doubly stochastic, so it redistributes signal without amplifying or shrinking it. The card says only that Sinkhorn "constrains" the map, so that doubly stochastic reading is general background rather than a card claim.

Why would a looped model want this? When one block is applied twice, the residual state after pass one has to be a good input for the same weights in pass two. Extra routing capacity between streams gives the model a learned way to adjust what flows between passes. The card does not present ablations, so this motivation is an interpretation, not a reported result.

How does the hybrid attention pattern work?

The decoder interleaves linear attention and full attention in a repeating four-layer group: three linear-attention layers, then one full-attention layer. That is a 3:1 ratio.

The loop boundaries were chosen to match. The block [4, 28) is 24 layers, which is exactly six complete groups, and it starts on a group boundary. Each pass through the loop therefore sees the same attention pattern as the first one. The card states that the attention pattern is preserved across both passes.

The usual motivation for this mix, again general background, is long-context cost. Linear attention keeps a fixed-size state instead of a cache that grows with sequence length, while the periodic full-attention layers retain exact token-to-token lookup. The card lists 16 full-attention heads with 4 KV heads, which is a grouped-query setup. Training used sequences up to 262,144 tokens. The card does not say how the KV cache behaves across the two loop passes, so check the implementation before planning memory for long contexts.

What are the key specs?

The FP32 mHC parameters are worth noticing. The routing maps are kept at higher precision than the BF16 backbone.

How do you run SenseNova-Loop-9B-Preview?

The card's checked environment is Transformers 5.18.0 and PyTorch 2.11.0 with CUDA 13.0. The model loads through a custom implementation, so trust_remote_code=True is required.

python -m pip install "transformers==5.18.0" accelerate safetensors

model = AutoModelForCausalLM.from_pretrained(
    "sensenova/SenseNova-Loop-9B-Preview",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": "cuda:0"},
).eval()

Thinking mode is toggled with enable_thinking in the chat template. For serving, the card shows a LightLLM Docker example with tensor parallelism of 4, and an SGLang route that currently needs a build from a specific pull request (#47281) containing the SenseNova Loop implementation.

One caveat matters for any alternative runtime. The loop range, pass count, and mHC routing parameters are part of the checkpoint's execution configuration. A compatible engine has to load them and reproduce the same execution order. A stock 32-layer runner would silently get it wrong.

For speed, MTP (multi-token prediction) is available. Dspark and Dflash2 draft-model approaches are listed as in training. Fine-tuning with mHC on a single 8-GPU node goes through a torchtitan fork, with scripts for SFT, DPO, and MTP training.

Is a looped model worth trying?

It depends on what limits you. If weight memory is your constraint and you can afford extra compute per token, reuse is an appealing trade. If latency is the constraint, 56 executions cost more than 32, and a plain 9B model will be faster per token.

There is also little evidence to weigh yet. The card shows a benchmark chart (including BIRD-Dev, Terminal Bench 2.1, and AutomationBench, among others) but no tables in text, and the technical report is not out. Run it on your own workload before drawing conclusions.