---
title: "How Does SenseNova-Loop-9B-Preview's Looped Transformer Work?"
url: "https://goremy.ai/blog/articles/sensenova-loop-9b-preview-looped-transformer"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-03T01:20:07.081Z"
updated: "2026-11-03T01:20:07.081Z"
topic: AI Model Architectures
description: "SenseNova-Loop-9B-Preview reuses 24 layers twice with two-stream mHC and 3:1 hybrid attention: ~9B stored parameters, ~14.6B at inference, 256K context."
---

# How Does SenseNova-Loop-9B-Preview's Looped Transformer Work?

> SenseNova-Loop-9B-Preview reuses 24 layers twice with two-stream mHC and 3:1 hybrid attention: ~9B stored parameters, ~14.6B at inference, 256K context.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-03

## How does SenseNova-Loop-9B-Preview's looped architecture work?

SenseNova-Loop-9B-Preview is a 32-layer multimodal decoder in which one contiguous block of 24 layers (layers 4 to 27) runs twice with shared weights. That gives 56 decoder-layer executions per forward pass while storing only 32 layers of weights. Two residual streams, managed by manifold-constrained hyper-connections (mHC), carry information through the looped section. Attention alternates three linear layers with one full-attention layer. The card lists roughly 9B total parameters and about 14.6B inference parameters.

## TL;DR

- The model **reuses 24 of its 32 layers twice**, so a forward pass executes 56 layers from 32 sets of stored weights.
- The card lists **~9B total parameters but ~14.6B inference parameters**, which is the practical cost of looping: more compute per token without more stored weights.
- **Two-stream mHC** replaces a single residual path inside the loop, and Sinkhorn normalization (20 iterations in the documented config) constrains how the streams mix.
- **Hybrid attention** uses three linear-attention layers followed by one full-attention layer, and the loop boundaries line up with those four-layer groups.
- Training recipes go **up to 262,144 tokens** and cover text, images, code, and tool trajectories, aimed at long-context and agentic work.
- This is a **preview**: the technical report is listed as TBD, and the model needs `trust_remote_code=True` plus a compatible inference implementation.

## What does "looped" mean in a transformer?

A standard decoder passes the hidden state through each layer once, and every layer has its own weights. A looped transformer feeds the output of a block of layers back into the same block. Depth goes up, but the parameter count does not.

SenseNova-Loop-9B-Preview does this with a single block. Layers 0 to 3 run first. Then the block of layers 4 to 27, call it B, runs on the hidden state h, and runs again on its own output, so the computation is B(B(h)). Layers 28 to 31 and the output head finish the job.

The arithmetic is simple:

| Segment | Layers | Executions |
| --- | --- | --- |
| Prelude | 0 to 3 | 4 |
| Looped block (two passes) | 4 to 27 (24 layers) | 48 |
| Coda | 28 to 31 | 4 |
| Total | 32 distinct | 56 |

The second pass takes the first pass's output as input, and both passes use the same weights.

## Why is it ~9B stored but ~14.6B at inference?

The card gives both numbers: roughly 9B total parameters and roughly 14.6B inference parameters. The gap comes from the repeated block. Its weights are stored once but used twice, so the compute per token looks like a larger model than the checkpoint on disk.

This matters when you plan hardware. Weight memory tracks the stored size. Per-token compute tracks the 56 executions. The card says practical speedups and costs depend on workload, hardware, and serving configuration, and it does not publish latency or throughput figures. Treat any claim about efficiency as unmeasured until the technical report or independent tests arrive.

## What is two-stream mHC and why does the loop need it?

Hyper-connections generalize the residual connection. Instead of one residual path, the model keeps several parallel streams and learns how layers read from them, write to them, and mix them. Making the mixing "manifold-constrained" keeps those learned maps in a well-behaved family so the signal stays stable through deep or repeated computation.

Here is what the card documents for this model:

- mHC runs **inside the looped section** and carries **two residual streams** around the decoder-layer computations.
- Learned mappings govern how a layer **reads** from the streams, how stream information is **mixed**, and how the layer output is **added** back into the residual state.
- **Sinkhorn normalization** constrains the stream-mixing map. The documented configuration uses 20 iterations.

Sinkhorn normalization alternately rescales rows and columns of a positive matrix until it is close to doubly stochastic. A doubly stochastic mixing matrix redistributes signal between streams without amplifying or shrinking it overall. That plausibly helps when the same block is applied twice, since a badly scaled map would compound across passes. The card does not give the authors' reasoning or ablations, so read this as the general purpose of the technique, not a documented result for this model.

One practical consequence: the loop range, pass count, and mHC routing parameters are part of the checkpoint's execution configuration. An inference engine must load them and reproduce the same execution order. A stock decoder implementation will not run this model correctly.

## How does the 3:1 hybrid attention fit in?

The decoder interleaves linear attention and full attention in a repeating four-layer pattern: three linear-attention layers, then one full-attention layer. Linear attention avoids the quadratic cost of full attention over long sequences. The periodic full-attention layers keep global token-to-token lookup available.

The design detail that makes this work with looping is alignment. The loop covers 24 layers, which is exactly six four-layer groups, and it starts at layer 4, a group boundary. Each pass therefore sees the same attention pattern. Across all 56 executions, that works out to 14 groups of four: 42 linear-attention executions and 14 full-attention executions (my arithmetic from the stated pattern, not a figure from the card).

Full-attention layers use 16 heads with 4 KV heads, a grouped-query setup that shrinks the KV cache compared with full multi-head attention.

## What are the core specs?

| Property | Value |
| --- | --- |
| Input / output | Text and images in, text out |
| Distinct decoder layers | 32 |
| Looped range / passes | [4, 28) / 2 |
| Layer executions per forward pass | 56 |
| Hidden dimension | 4,096 |
| Feed-forward dimension | 12,288 |
| Vocabulary size | 248,320 |
| mHC streams / Sinkhorn iterations | 2 / 20 |
| Training context | Up to 262,144 tokens |
| Export precision | BF16 backbone, FP32 mHC routing parameters |
| License | Apache 2.0 |

The FP32 routing parameters are worth noting. The mixing maps stay in higher precision while the backbone runs in BF16.

## What is it built for, and how do you run it?

The card positions the model for long-context and agentic workflows: document understanding, coding, and interaction with external tools. Its benchmark chart compares it against other models on BIRD-Dev, WorkBuddy Bench, AIDABench, TeamBench, Terminal Bench 2.1, T³-Banking, AutomationBench, and RaccoonPPTBench. The card presents those results as an image, so this article does not quote scores.

Documented ways to run it:

- **Transformers:** the checked environment is Transformers 5.18.0 with PyTorch 2.11.0 and CUDA 13.0, loading with `trust_remote_code=True`. Thinking mode toggles through `enable_thinking` in the chat template.
- **LightLLM:** a Docker-based server, shown with four-way tensor parallelism and an OpenAI-style chat endpoint.
- **SGLang:** requires a build containing the SenseNova Loop implementation from a specific pull request, not a stock release.
- **Fine-tuning:** an 8-GPU single-node recipe using a torchtitan fork, covering SFT, with DPO and MTP training described in the linked guide.

For faster decoding, MTP (multi-token prediction) is available. Dspark and Dflash2 draft-model methods are listed as training in progress.

## Is it ready for production use?

Not yet, by the project's own labeling. The name says Preview, the technical report is TBD, and the Transformers implementation's publication details are still to be announced. SGLang support depends on an unmerged pull request as of the card. The card also recommends the documented deployment environment and configuration for best performance, which suggests behavior may vary elsewhere.

Open questions a builder would want answered: how much of the extra compute translates into quality versus a conventional 9B or 14B model, how the KV cache behaves across two passes, and what real throughput looks like. The card does not answer these.

## Frequently Asked Questions

### How many parameters does SenseNova-Loop-9B-Preview have?

The card lists about 9B total parameters and about 14.6B inference parameters. The difference exists because 24 layers are stored once but executed twice. Weight memory follows the stored count, while per-token compute follows the larger effective depth.

### Which layers are repeated?

Layers 4 through 27, written as [4, 28) in the card, run twice with shared weights. Layers 0 to 3 run before the loop and layers 28 to 31 run after it. That makes 56 layer executions from 32 distinct layers.

### What does the 3:1 attention ratio mean?

Every group of four layers has three linear-attention layers and one full-attention layer. Linear attention keeps long-sequence cost down, and the full-attention layer preserves global attention. The loop boundaries align with complete four-layer groups, so both passes share the same pattern.

### Can I use it commercially?

The model is released under Apache 2.0, a permissive license that generally allows commercial use. Check the license file in the repository for exact terms before relying on this.

### Does it support images?

Yes. It accepts text and images as input and generates text. It does not generate images.