---
title: "How Is Ripple-65B Built? Dense Thin-Deep vs Mixture of Experts"
url: "https://goremy.ai/blog/articles/ripple-65b-thin-deep-dense-architecture-explained"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-06T00:00:11.390Z"
updated: "2026-11-06T00:00:11.390Z"
topic: AI Model Architectures
description: Ripple-65B is a dense 80-layer transformer where all 65B parameters run per token. How it differs from MoE, plus its attention tweaks and training.
---

# How Is Ripple-65B Built? Dense Thin-Deep vs Mixture of Experts

> Ripple-65B is a dense 80-layer transformer where all 65B parameters run per token. How it differs from MoE, plus its attention tweaks and training.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-06

## How is Ripple-65B built?

Ripple-65B is a dense decoder transformer from Recursive Intelligence. All 65 billion parameters work on every token, with no expert routing. It is built thin and deep: 80 layers with a narrow width. The team adopted a Gemma-style layout for stable training, added more attention heads to help reasoning, and used more sliding-window layers with a smaller window to cut cost. It is released under Apache 2.0 as a preview with light post-training.

## TL;DR

- Ripple-65B is **fully dense**, so every one of its 65B parameters is active for every token, unlike the mixture-of-experts (MoE) designs most large labs now ship.
- The shape is **thin and deep**: 80 layers and a narrow hidden width, chosen with training stability and reasoning in mind.
- Attention changes include **more heads, more sliding-window layers with a smaller window**, plus gated attention and a value residual.
- Pretraining ran on **12 trillion tokens**, mostly English text and code, with synthetic data deliberately held back for later stages.
- Post-training is a **five-step chain** ending in reinforcement learning, but this preview got only light post-training focused on agentic coding.
- Self-hosting is feasible but not turnkey. One hands-on tester ran a **quantized build in roughly 75 GB of VRAM** and had to rewrite the provided inference code for his GPU.

## What is the difference between a dense model and a mixture of experts?

In a dense transformer, every token passes through every weight. In an MoE model, each layer holds many "expert" feed-forward blocks, and a router sends each token to only a few of them. Only a slice of the model wakes up per token.

That creates two different numbers for MoE models: total parameters (what you store) and active parameters (what you compute with per token). For a dense model the two are the same. For Ripple, both are 65B.

The general tradeoffs are well known:

| | Dense | Mixture of experts |
|---|---|---|
| Parameters used per token | All | A small subset |
| Compute per token | Scales with total size | Scales with active size |
| Memory footprint | Total size | Total size (all experts must be loaded) |
| Training complexity | Simpler | Needs routing, load balancing |
| Serving complexity | Simpler | Expert placement and routing matter |

The source describes Ripple's dense choice as a way to keep the design simple enough for a very small team to train from scratch. That is a reasonable reading of the tradeoff. You give up per-token efficiency and gain a plainer system with fewer moving parts.

## Why build a thin and deep model?

Ripple uses 80 layers with a narrow width. Depth gives the network more sequential steps to transform a token's representation, which is commonly associated with multi-step reasoning. Narrow width keeps each layer cheaper, so the parameter budget goes into more layers rather than fatter ones.

The cost of depth is training stability. Very deep stacks can be harder to optimize, which is likely why the team moved to a Gemma-style layout. The source says only that this was done "for stable training" and does not spell out which components were borrowed. In the Gemma family, the layout is generally associated with extra normalization and a mix of local and global attention, but check the model card for what Ripple actually adopts.

## What attention changes does Ripple make?

The team listed four changes.

1. **More attention heads.** The stated reason is to help reasoning. More heads let a layer track more distinct relationships at once.
2. **More sliding-window layers.** In sliding-window attention, a token only attends to a fixed window of recent tokens instead of the whole context. Cost in those layers grows with window size rather than full sequence length.
3. **A smaller window.** Shrinking the window cuts cost further. Ripple handles fairly long context, so presumably a minority of layers still see the full sequence, though the source does not give the ratio.
4. **Gated attention and value residual.** These are extras layered on top. Gated attention applies a learned gate to the attention output. A value residual carries value information forward across layers. The source names both but does not detail how Ripple implements them.

The theme is clear even without the fine print. Spend compute on depth and heads, and claw back cost through cheaper local attention in many layers.

## How is Ripple-65B trained?

Pretraining is from scratch on 12 trillion tokens, mostly English text and code. The lab kept this data clean and mostly natural. It held synthetic data back for the next stage so the team could iterate on it quickly without blocking pretraining.

Post-training then runs as a chain:

1. Mid-training
2. A long-context stage
3. Supervised fine-tuning
4. A long-context fine-tuning pass
5. Reinforcement learning across mixed domains

The team says this preview received relatively light post-training, with the focus on agentic coding. Treat it as an early look, not the finished model.

## How does Ripple-65B perform?

The shared benchmark charts, as reviewed by one outside tester, put Ripple at the top on a terminal-based agentic benchmark and a harder software engineering benchmark, ahead of models many times its size. On a deep-research style benchmark it sat near the top group, with Meta's model clearly ahead. It trailed on math, safety, and automation, where Meta's and Reflection's models led. I have not independently verified these numbers, and no scores are given here because the source only described the charts.

The same tester ran hands-on checks:

- **Bug hunt in a Dockerized full-stack app.** The model found a planted bug on its own: a withdrawal rate stored as 4 but used as 0.04, which made every result 100 times too small. It fixed it, and the app produced sensible numbers afterward. This ran through an agent harness against a self-hosted copy, after hosted providers returned max-token errors.
- **Browser physics and graphics demo.** A doner kebab simulation with no external libraries or assets. The geometry, burner, and dripping fat worked well. The meat layers and browning were weaker.
- **Complex SQL.** A Databricks query over a six-table schema with arrays, structs, and a map, with traps like double-counted money across joins. The model summed items and refunds separately before joining and followed the stated rules.

These are single runs by one person, so they show promise rather than proof.

## Can you run Ripple-65B locally?

Possibly, with effort. The release is Apache 2.0, and the source says it can fit on a DGX Spark or similar device. In the hands-on test, a quantized build used about 75 GB of VRAM on a machine with 96 GB (an RTX Pro 6000 Blackwell).

The setup was not smooth. The provided inference code targeted a different GPU and had to be adjusted, and the tester built a small server around it. Expect rough edges with a preview release. Dense models also load all 65B parameters and compute with all of them on each token, so memory and throughput planning is simpler to reason about than with MoE, but not cheaper.

## Is a dense model the right choice?

It depends on what you value. If you want a straightforward architecture, predictable behavior, and a permissive license, a dense 65B is attractive. If per-token serving cost at scale is your main concern, MoE designs are built for that. Ripple's coding results suggest a dense model of this size can compete in agentic work, but the gaps in math and safety, and the light post-training, mean it is worth testing on your own workload before committing.

## Frequently Asked Questions

### Is Ripple-65B a mixture-of-experts model?
No. It is a dense transformer, so all 65 billion parameters are active for every token. This is a deliberate contrast with the MoE approach many large labs use.

### How many layers does Ripple-65B have?
It has 80 layers with a narrow width, which is what "thin and deep" refers to. The team paired this with a Gemma-style layout to keep training stable.

### What license is Ripple-65B released under?
Apache 2.0, a permissive open-source license. It is published as a preview, with only light post-training so far.

### What data was Ripple-65B trained on?
Pretraining used 12 trillion tokens, mostly English text and code. Synthetic data was held back for later stages. Post-training added mid-training, long-context stages, supervised fine-tuning, and reinforcement learning.

### What is sliding-window attention and why does Ripple use it?
Sliding-window attention limits each token to a nearby window of context rather than the full sequence. Ripple uses more of these layers, with a smaller window, to keep compute cost down in a deep model.