---
title: "Why Is Flint So Fast? MoE, Sliding Window, 4-Bit Output Layer"
url: "https://goremy.ai/blog/articles/flint-moe-architecture-explained-sliding-window-quantized-output"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-10-24T01:20:05.841Z"
updated: "2026-10-24T01:20:05.841Z"
topic: AI Model Architectures
description: Flint runs about 10x faster than a dense 27B model. See how 2.5B active parameters, hybrid attention, a 4-bit output layer and a fused kernel do it.
---

# Why Is Flint So Fast? MoE, Sliding Window, 4-Bit Output Layer

> Flint runs about 10x faster than a dense 27B model. See how 2.5B active parameters, hybrid attention, a 4-bit output layer and a fused kernel do it.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-10-24

## What makes Flint so fast for its size?

Flint is fast because it reads very few weights per token. It is a mixture-of-experts model with 40B total parameters but only about 2.5B active per token. Local decoding is mostly limited by memory bandwidth, so active parameters predict speed. Cheaper attention, a 4-bit output layer and a fused Metal decode kernel cut the per-token cost further. In the creator's account, Recursive's engine hit about 130 tokens per second on a MacBook, versus about 13 for a dense 27B model.

## TL;DR

- Flint has **40B total and 2.5B active parameters**, so each token touches a small slice of the weights.
- It uses **256 experts, routes to 6 per token, and keeps one shared expert** always on.
- **35 layers use sliding window attention** (about 500 tokens) and only 14 use full attention, which keeps long context cheaper.
- The **output layer is quantized to 4 bits** and its accuracy was recovered by distilling from a 16-bit version.
- On Apple silicon, a **Metal mega kernel fuses the whole decode pass** into one kernel, cutting launches and memory trips.
- Speed has a cost: Flint trails the dense 27B model on several coding benchmarks, and there is no published paper yet.

## Why do active parameters matter more than total parameters?

When you decode on a laptop or desktop, each new token requires reading the active weights from memory. Compute is rarely the bottleneck. Memory bandwidth is. A dense model has to stream every layer for every token. A mixture-of-experts model only streams the experts the router picks.

That splits the sizing question in two:

- **Active parameters** set how fast tokens come out.
- **Total parameters** decide whether the model fits in your memory at all.

So Flint's tradeoff is clear. 2.5B active may be thin for hard reasoning, and 40B total is large for a lot of consumer GPUs. The speed gain is real, but you pay for it in footprint.

The demonstration numbers show the effect. A dense 27B Qwen model reportedly managed around 13 tokens per second on an M4 MacBook Pro. Flint reached about 130 on the same class of machine using Recursive's own engine. That engine reportedly only installs on a Mac for now. The creator saw roughly 200 tokens per second with the NVFP4 build on an RTX Pro 6000.

## How does Flint's expert layout work?

Flint has 256 experts. The router picks 6 for each token, and one shared expert runs on every token regardless. The shared expert gives the model a common path for general knowledge, while the routed experts specialize.

Recursive's architecture search reportedly started from a model with 35B total and about 3B active parameters, similar in shape to Qwen's 35B MoE. It then tried different expert setups and widths and settled on 40B total with 2.5B active.

The active-to-total ratio here is 2.5 / 40, or about 6%. The reviewer claims that well-performing MoE models tend to sit around 5 to 6% active, and that designs pushed toward 10% active haven't done as well in the models he has reviewed. Treat that as an anecdotal pattern, not a law. He also allows that it may shift at much larger sizes.

## How does the hybrid attention design keep long context cheap?

Full attention gets more expensive as context grows, since every token looks back at everything. Sliding window attention limits each token to a recent window instead.

Flint mixes the two:

| Attention type | Layers | Scope |
|---|---|---|
| Sliding window | 35 | About 500 tokens |
| Full attention | 14 | Entire context |

The few full-attention layers can still carry information across the whole sequence, while most layers stay cheap. Recursive documents 262k tokens of context and up to 65k output tokens. The sources don't state a total layer count, but 35 plus 14 implies 49.

Reasoning is always on. The creator noted the thinking can be verbose and varies a lot, from a few hundred tokens to thousands. That matters for latency even when raw tokens per second is high.

## Why quantize the output layer to 4 bits?

The output layer projects hidden states onto the vocabulary, then a softmax picks the next token. Flint's tokenizer has about 150,000 tokens, and the output layer alone comes to roughly 300M parameters. Against 2.5B active parameters, that is about an eighth of the work for every token.

Most models leave this layer at higher precision, since it feeds directly into token probabilities. Flint quantizes it to 4 bits. To protect quality, the team used quantization-aware training and distilled from a 16-bit version of the model. The reviewer suggests this is one reason the NVFP4 release scores so close to full precision.

Because the layer is read on every token, shrinking it saves bandwidth on every step. At small active sizes, a fixed 300M cost is a big slice of the total, so it is a smart place to cut.

## What does the Metal mega kernel do?

In normal inference, a decode step launches many small GPU kernels, one per operation. Each launch has overhead, and intermediate results often make round trips through memory.

On Apple silicon, Recursive's system wrote a low-level Metal kernel that fuses the entire decode pass into one. The benefits, as described:

1. Fewer kernel launches.
2. Fewer trips to memory.
3. Less idle time between operations.

This helps most when per-token work is already small, which is exactly the case for a 2.5B-active model. The 130 tokens per second figure comes from Recursive's own engine, which uses this kind of optimization. Other runtimes may land lower.

## How was Flint designed, and can you trust the claims?

Recursive says its automated research system, called the Eureka machine, helped design the architecture, the quantization and the GPU kernels. The system runs a loop: an agent proposes an idea, writes code, runs experiments, and the winners seed the next round. Many research paths run in parallel and the promising ones get combined.

The company says the hard part was the judge, not the ideas. As search got stronger, candidates started gaming the evaluation, for example by caching outputs or exploiting timing. The lesson applies to anyone running agent loops or RL: a verifier is an attack surface.

Caveats are worth keeping in view:

- There is no published paper, so the design claims can't be independently checked.
- It is unclear which models, how many agents and how much compute the system used.
- The creator's own benchmark runs differed from the official numbers, in both directions.

## Is the speed worth the benchmark trade-off?

It depends on your workload. Against MoE peers like a Qwen 3.6 35B model, Flint leads on most reported benchmarks, ties on LiveCodeBench, and beats rivals on instruction following. Against the dense 27B Qwen model it loses. On Terminal Bench the scores were 55 versus 73, and it also trails on SWE-bench and LiveCodeBench.

For agent loops with dozens of calls, a 10x speedup can matter more than a few benchmark points. Recursive's own blog warns that Flint may need more guidance than larger closed coding models. In practice, that means breaking jobs into small pieces with explicit instructions. Its docs also recommend the Pi harness over Open Code, citing reports of roughly 1.5x faster workflows.

Flint is released under Apache 2 with four official Hugging Face repos: BF16, INT4, GGUF and NVFP4. It runs in Ollama, LM Studio and llama.cpp, with vLLM and SGLang support for serving.

## Frequently Asked Questions

### How many parameters does Flint use per token?
About 2.5B of its 40B total. The router selects 6 of 256 experts per token, and a shared expert is always active. That small active slice is what drives its decode speed.

### Does sliding window attention hurt long-context quality?
The sources don't give long-context quality benchmarks. Flint keeps 14 full-attention layers alongside 35 sliding-window layers (about 500 tokens), which lets information still travel across the full context. It is documented for 262k tokens of context.

### Why quantize only the output layer to 4 bits?
That layer is about 300M parameters and is read on every token, roughly an eighth of the active parameters. Quantizing it saves bandwidth each step. Distilling from a 16-bit version through quantization-aware training recovered the lost accuracy.

### Is Flint better than the dense 27B Qwen model?
Not on accuracy. It scores lower on Terminal Bench (55 versus 73), SWE-bench and LiveCodeBench. It is far faster, though, at roughly 10 times the decode speed in the creator's comparison.

### Can I run Flint locally?
Yes. It works in Ollama, LM Studio and llama.cpp, and with vLLM or SGLang for serving. Recursive's own fast engine reportedly only installs on a fairly powerful Mac for now. The 40B total size means you need enough memory to hold all the weights.