What Is Clover-35B-A3B? Block Diffusion on a Qwen3.6 MoE

What is Clover-35B-A3B?

Clover-35B-A3B-Preview is the first model from DiffuSpace, an open-weight, text-only diffusion language model derived from Qwen3.6-35B-A3B. It keeps the Qwen mixture-of-experts backbone (about 35B total parameters, roughly 3B active per token), adds learned parameters for bidirectional attention inside a block, and generates text in blocks of masked tokens instead of one token at a time. It supports a 128K context window. DiffuSpace says it is state of the art among open-source diffusion language models at release, based on benchmarks in its launch blog.

How does Clover turn a Qwen3.6 MoE model into a block diffusion model?

Clover changes less than you might expect. The backbone is unchanged in shape: 40 layers mixing Gated DeltaNet layers with full-attention layers, each with an MoE feed-forward of 256 routed experts, top-8 token routing, and a shared expert.

Three things are new, according to the model card:

1. Bidirectional GDN parameters. Extra learned weights let tokens inside a block see each other, not only earlier tokens.
2. Block diffusion inference. Generation works on a block of masked positions that gets resolved over several steps.
3. Generation controls. Repetition penalties and gradual end-of-response biases tuned for long outputs.

The upstream Qwen assets are Apache-2.0 licensed (Copyright 2026 Alibaba Cloud), and the Clover license preserves that notice. The implementation derives from the Hugging Face Transformers Qwen3.5-MoE code used for the Qwen3.6 architecture.

How does bidirectional block attention work in a GDN model?

A standard autoregressive model is causal everywhere. Each token sees only what came before. Block diffusion relaxes that inside the block being generated. The block starts as masked tokens, and every position needs context from its neighbors on both sides to decide what to write.

Clover handles this with additional learned GDN parameters. The card names right-convolution weights and softmax controls. Together they let tokens attend within a noisy block while "preserving a causal prefix." Everything before the current block behaves like normal left-to-right context. Only the active block is bidirectional.

The card does not detail the training recipe or the exact math of these parameters, so treat anything beyond that description as unconfirmed.

How does confidence-threshold block decoding work?

The Hugging Face implementation uses LowConfidence decoding. The loop is simple:

1. Start a block of masked tokens.
2. Run the model and score each masked position.
3. Fill every position whose confidence exceeds the threshold, all in parallel.
4. If no position qualifies, fill the single most confident one so the loop always makes progress.
5. Repeat until the block is fully resolved.
6. Append the finished block to the prefix and start the next one.

Defaults are block_size=32 and threshold=0.95. A high threshold means the model commits only to tokens it is quite sure about. Easy stretches of text resolve in few steps. Hard stretches take more. The speedup over token-by-token decoding therefore depends on how predictable the output is, and the card does not give a fixed step count.

The same model also supports self-speculative decoding in the authors' SGLang and llama.cpp implementations, documented in the official repository.

What is the MoE coreset and why does it matter?

MoE models route each token to its own top-8 experts out of 256. In a block of 32 tokens being denoised together, that can touch many distinct experts. In memory-bound inference, each extra expert is more weight to read.

Clover's coreset option restricts routing to a shared subset of experts per block. The selector:

• aggregates the routing scores of each token's top-8 candidates across the block,
• ranks experts by aggregate score,
• keeps enough experts to cover the requested fraction of that mass.

A top_p of 0.9 means 90% of aggregated routing mass, not 90% of all experts. Lower values select fewer experts, with a possible quality tradeoff. It is separate from sampling top_p, and it does not shrink the stored or loaded weights. All 256 experts still sit in memory. The card says it "reduces the number of distinct experts used across the block," and does not publish a speed figure for it in the text.

Why does diffusion decoding matter for local inference?

Autoregressive decoding is typically memory-bound. Each new token requires reading the model's weights, so a long answer means many full passes over memory. Diffusion-style models try to produce several tokens per pass.

One local-hardware test of a different parallel text model (Stanford's NCM, not Clover) showed the tradeoff clearly. It was faster than a same-size autoregressive model for a single request, but slower when 64 requests were batched, because batching already uses the GPU's math capacity well. That result is about NCM and small models, so it does not transfer directly to Clover. It does show why block diffusion claims should be checked at both single-user and batched-serving settings.

How do you run Clover-35B-A3B?

The card gives a Transformers path. It uses Transformers 5.19.0 and PyTorch 2.10 or later.

pip install "transformers==5.19.0" accelerate safetensors

Load the model with trust_remote_code=True in BF16, then call generate() with block_size=32, threshold=0.95, and do_sample=False. Plan for about 69 GB of GPU memory for the weights alone.

The Hugging Face path is a standalone PyTorch implementation. Small numerical differences from SGLang's kernels can change generated text even with identical settings. For serving and evaluation, the authors recommend their SGLang implementation.

What generation controls ship by default?

The authors report that very long generations can become repetitive. Defaults are already set, including under greedy decoding.

The length bias starts at 8192 response tokens and ramps over another 8192. The thinking-end bias starts at 4096 and ramps over 4096. These are soft nudges. max_new_tokens is the hard limit. Requests that exceed 131,072 tokens, including room to resolve the final block, are rejected.