What Are Looped Transformers in AI?

A looped transformer reuses the same small block of layers multiple times in a single forward pass, instead of running data through a long chain of unique layers. Run a k-layer block L times and you get an effective depth of k times L, without paying for k times L worth of stored parameters.¹² A model gets to act deeper and smarter without getting physically bigger.

The idea has jumped from academic papers to the center of an AI safety argument in the last few weeks, because reporting from The Information says OpenAI's unreleased Astra model uses exactly this technique.³⁴

What is a looped transformer, in plain English?

Picture a standard transformer as an assembly line. Data goes in one end, passes through layer 1, then layer 2, then layer 3, each layer doing its own job with its own trained weights, until a finished answer comes out the other end. More layers generally means more capability, but also more parameters to store and more memory bandwidth to move around.

A looped transformer swaps the assembly line for a small workshop and a conveyor belt that keeps circling back through it. The same block of layers, with the same weights, processes the input, then processes its own output again, and again, for some number of loops, before the model commits to a final answer.² The block doesn't get bigger. It just gets used more.

The result is called "recurrent depth": you increase how much computation a model does at run time without increasing how many parameters it has to store.¹ Researchers have been formally building on this for a few years, with papers like "Looped Transformers as Programmable Computers" (2023) and "Looped Transformers Are Better at Learning Learning Algorithms" (2024) treating it as a real architectural direction, not a gimmick.¹

How is this different from chain-of-thought reasoning?

Today's reasoning models, the ones that write out "let me think step by step" before answering, do their extra thinking in plain text. Every intermediate step is a token you can read. That visible chain of thought (CoT) is also what safety researchers currently lean on to check whether a model is reasoning honestly or scheming toward something else.

Looped transformers do something structurally different. Instead of writing out intermediate reasoning as words, the model iterates on a continuous, numerical hidden-state vector across each loop, refining it internally before it ever generates a token.⁵¹ Researchers have started calling this style of internal computation "neuralese": thinking that happens in a compressed, mathematical form that never gets translated into human language.² Lilian Weng's technical rundown of the approach compares it to a conditioned diffusion process, where the original input keeps getting reintroduced as an internal state vector gets updated step after step.⁵

A useful analogy: a musician rehearsing a piece silently in their head, running through it over and over, produces a better performance without ever narrating the rehearsal out loud. The improvement is real. The narration just isn't there for anyone to check.

The research lineage: from Universal Transformers to Ouro

Looped transformers aren't a new invention dressed up for a headline. The core idea, sharing parameters across steps and adapting how much computation to spend, traces back to the 2019 Universal Transformer.¹ The technique got a serious scaling test in 2025, when researchers led by Jonas Geiping trained a 3.5-billion-parameter proof-of-concept model, nicknamed Huginn, on 800 billion tokens.⁶ Its performance on reasoning benchmarks kept improving up to a compute load equivalent to a 50-billion-parameter standard model, a big jump in effective capability for a comparatively small footprint.⁶

ByteDance Seed pushed the idea further with its Ouro models: 1.4-billion and 2.6-billion-parameter looped transformers, each looped four times and trained on 7.7 trillion tokens, that matched the benchmark performance of standard, non-looped transformers up to 12 billion parameters. The team attributed the gains to better knowledge composition, not extra memorized facts, which is a meaningfully different kind of improvement than just cramming in more training data.

Separately, a theoretical paper by Saunshi, Dikkala, Li, Kumar, and Reddi (ICLR 2025) proved that many reasoning problems need depth but not necessarily more parameters, and that a k-layer transformer looped L times can implicitly simulate T steps of chain-of-thought using T loops.⁷ That's the formal justification for why looping can substitute for writing reasoning out loud.

Why is OpenAI's Astra the trigger for this conversation?

Search interest in this topic spiked because of a specific report: The Information says OpenAI's unreleased Astra model uses recurrent depth, running the same layers repeatedly, which can make a smaller model act like a much larger one while cutting memory and bandwidth costs.⁴³

OpenAI's Chief Research Officer, Jakub Pachocki, pushed back publicly on how that reporting was framed. He said the computation-graph depth of frontier OpenAI models, Astra included, is "within a factor of two of GPT-4," and that OpenAI has worked to preserve chain-of-thought monitoring since its first reasoning models.⁸ He also conceded, in the same statement, that monitorability is "fragile and unfortunately trending in a negative direction."⁸ Other reporting says OpenAI has deliberately limited how much recurrent depth Astra uses specifically to keep chain-of-thought readable, and plans additional CoT-monitoring safeguards as a mitigation.⁹

Safety researcher Ryan Greenblatt of Redwood Research took the sharper view, calling the shift toward this kind of opaque reasoning architecture potentially one of the worst developments for AI safety to date, because it removes the legible trace researchers use to catch a model lying or pursuing a goal it wasn't given.⁴⁸

Why are looped transformers cheaper to run?

The efficiency case is the part that will actually matter to anyone deploying these models, not just the safety researchers arguing about them. A looped model spends compute at inference time, running more loops, instead of spending parameters at training time by adding more unique layers. That trade matters for a few practical reasons:

• Fewer parameters to store. Ouro's 1.4B and 2.6B models matched the benchmarks of standard transformers up to 12B parameters, meaning a fraction of the weight storage for comparable capability.
• Less memory bandwidth per token. Because the same block of layers is reused, you're not constantly loading fresh weights for each layer, which is a real cost at inference scale.³⁴ This is the same bottleneck that any inference engine is built to manage, and it's why architecture choices like looping matter as much as raw parameter count once you're serving real traffic.
• A tunable depth dial. Unlike a fixed-depth model, a looped model can spend more or fewer loops depending on how hard the problem is, closer to test-time compute than a static architecture.⁶

This is also why the framing matters for anyone evaluating vendors rather than building models. If you're picking an AI model for internal tools, this is one more axis beyond the usual benchmark score, and it's worth reading past leaderboard numbers the way our guide to choosing an AI model for internal tools argues you should generally. And as falling API prices push everyone to compete on cost, cutting parameter count without cutting capability is exactly the kind of lever that decides who's actually running efficiently.

Why is this harder to audit?

This is the governance problem, and it's the reason this architecture question landed on a safety beat and not just an ML engineering one. Chain-of-thought monitoring, reading a model's own stated reasoning to catch when it's about to do something wrong, only works if that reasoning is written in words a human or another model can read. Looped transformers, by design, can push a chunk of that reasoning into a hidden numerical state that never gets rendered as text.⁵²

OpenAI's own acknowledgment that monitorability is "trending in a negative direction" is notable precisely because it comes from the company building the model in question, not an outside critic.⁸ For companies thinking seriously about shadow AI and AI governance inside their own organizations, the underlying tension is the same: the more capability you push into systems that operate outside a legible, inspectable trace, the harder it gets to audit what's actually happening, whether that's a model's hidden reasoning or an employee-built tool nobody signed off on.

Is latent reasoning a permanent black box?

Not necessarily, and this is the part of the story that gets skipped in the more alarmed takes. A research sprint from BlueDot Impact applied logit-lens-style probing, a technique for decoding what a model's internal states "mean" at intermediate steps, to Huginn-3.5B, the Geiping team's recurrent-depth model.¹⁰ They found that decoded latent representations sometimes progressively shift from an incorrect answer to a correct one across the recurrent iterations, meaning the model's internal "thinking" left a trace that could be partially read out.¹⁰ Their conclusion: latent reasoning may be more tractable to interpretability tools than initially feared, not a total black box.¹⁰

That's a genuine counterpoint, not a full rebuttal. It shows interpretability researchers can sometimes still see into the loop. It doesn't mean today's monitoring tools, built around reading plain-English CoT, will work unmodified on this new kind of hidden reasoning.

What this means going forward

Looped transformers are a real, multi-year research direction, not a rumor invented around one company's roadmap. The architecture trades stored parameters for runtime compute, and the tradeoff is genuinely attractive: smaller models that punch above their weight class, at lower memory cost.⁶ But the same mechanism that makes this efficient, reasoning in a hidden numerical space instead of visible text, is also what makes today's chain-of-thought safety monitoring harder to rely on, by OpenAI's own admission.⁸

If you evaluate or deploy AI models for your team, ask vendors directly how much of a model's reasoning happens in a legible trace you could actually inspect, versus in a loop you can't see into. Depth as a runtime dial, not just a training-time architecture choice, is going to keep showing up in model cards and technical reports whether or not the word "looped" ever makes it into the marketing.