--- title: "Is NCM Faster Than an LLM? Next Concept Modeling Tested" url: "https://goremy.ai/blog/articles/next-concept-modeling-ncm-stanford-speed-tests" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-10-29T22:00:07.517Z" updated: "2026-10-29T22:00:07.517Z" topic: AI Model Architectures description: "Stanford's NCM writes text coarse to fine. Independent DGX Spark tests show about 4x speed for one user, slower batching, and a Sudoku result with a caveat." --- # Is NCM Faster Than an LLM? Next Concept Modeling Tested > Stanford's NCM writes text coarse to fine. Independent DGX Spark tests show about 4x speed for one user, slower batching, and a Sudoku result with a caveat. Edited by Luis Chavez-Mattos, Director, Product ยท Published 2026-10-29 ## Is NCM faster than a normal LLM? It depends on how many requests you send. In independent tests on a DGX Spark, Stanford's Next Concept Modeling (NCM) ran about four times faster than a same-size autoregressive model on translation when serving one request at a time. With 64 requests batched together, the ordinary model won. The paper's 31x speed claim was measured only against diffusion models, never against a standard token-by-token LLM. ## TL;DR - NCM writes text **coarse to fine**: it sketches the whole answer with a few codes, then refines level by level, instead of going left to right. - The paper's headline **31x speedup is against diffusion models**, not against autoregressive LLMs. - In local single-request tests, NCM was **about 4x faster on translation** and close to 2x faster on news summaries than a same-size autoregressive model. - With **64 simultaneous requests, the autoregressive model was faster**, which matters for servers and multi-agent setups. - The 86% hard Sudoku score looks like it comes **largely from cell-fill order**, since a diffusion model with a confidence-first rule got close and reached 95% with more passes. - The released models are small (under 1 billion parameters), code is evaluation only, and nothing is readable until the last pass finishes. ## Why are normal LLMs slow in the first place? Autoregressive models write one token, append it, then predict the next. A thousand-token answer takes a thousand sequential steps. The expensive part is memory, not math. On every step the GPU has to read the model's full weights to produce a single token. That is why local generation speed is mostly set by memory bandwidth. If your local model writes slower than you can read, this is the main reason. ## How do diffusion models try to fix that? Diffusion language models start with a page of blanks, guess the whole answer at once, and then revise the full page repeatedly. Each pass touches every position. Good quality usually needs 64 passes or more, and cutting passes hurts quality quickly. You get parallel writing, but you pay by redrawing the whole page many times. ## How does NCM work? NCM borrows the coarse-to-fine idea from image generation, where a model draws a blurry whole picture first and adds sharper layers on top. It has two parts. **The compressor.** It stores a passage as a stack of levels made of codes. The top level has only a handful of codes holding a blurry outline of the whole passage. Each lower level is longer and adds missing detail. The bottom level has one code per word. Add the levels together and the text can be rebuilt almost perfectly. **The generator.** This is an ordinary transformer. Instead of predicting the next word, it predicts every code in a level at once, in one pass, then moves to the next level down. The "concept" in the name is not a human idea. Each code comes from a fixed list of about 4,000 learned codes, and the paper calls each one a concept. A summary model with nine levels needs nine passes whether the summary is short or long. The translation model in the demo has 11 levels, so it takes 11 passes. The difference from diffusion is cost per pass. The top levels are tiny, so early passes are nearly free, and each level only adds the detail it owns. ## What did the paper claim, and what did it leave out? Against diffusion models on a single H100, the authors report that NCM matched the strongest diffusion model on news summarization while running 31 times faster. On hard Sudoku, a same-size autoregressive model solved about 1% of boards, while a tiny NCM model solved 86%. Two gaps stand out: - Every speed comparison in the paper is against diffusion models. None is against a token-by-token LLM. - On real text tasks such as translation and open-ended writing, the plain autoregressive model still wins on quality in the authors' own tables. ## Is NCM actually faster than an autoregressive LLM? A creator running the released models on a DGX Spark built an autoregressive model of the same size and shape, gave both the same prompts, and asked for the same output length. These are small models and simple experiments, so treat the numbers as directional. | Scenario | Result | |---|---| | One request, translation | NCM about 4x faster | | One request, news summary (long prompt) | NCM close to 2x faster | | 64 requests at once | Autoregressive model faster | The flip follows from the memory-versus-math split. With one request, the GPU mostly waits on memory, so finishing in nine or 11 passes is a big win. With a batch of 64, the weights load once for the whole batch and the GPU becomes math-bound. NCM does more math there because across all levels it processes roughly four times more positions than the words it writes. The release code also reruns the whole prompt through the model on every pass. That hurt most on the long news article. The built-in prompt-caching option appeared broken in these tests. For a single user on a local GPU, NCM is a real speedup. For a busy server, or for agent workloads running many sessions in parallel, it slowed down in these experiments. ## Does the Sudoku result prove a better architecture? Not by itself. The tester first reproduced the paper: almost exactly 86% on hard Sudoku. Printing the board after every pass showed that NCM does not fill squares one at a time. It redraws the whole grid each pass, even getting the starting clues wrong early before snapping into place at the end. Then came the control. A diffusion model of the same size, trained on the same puzzles and tested on the same hard boards, solved about a quarter of them when it filled cells in random order, in line with the paper. When the decoding rule changed so the model filled the cell it was most confident about first, it got close to NCM. With more passes, it reached 95%. The reading from that test is that most of the Sudoku gap comes from the order cells are committed, not from the NCM architecture. A plain autoregressive model fails because it cannot go back after committing to a wrong digit early. NCM still has a real advantage here. It gets coarse-to-fine behavior built in, while the diffusion model needed an extra decoding rule. The tester's suggestion was that combining the two ideas may be worth exploring. ## What are NCM's limitations? - **No streaming.** Nothing is readable until the final pass finishes, so you cannot stream tokens the way autoregressive chat does. - **Fixed-size output box.** Even a one-line answer runs through every level, so you pay the full cost. - **Names and numbers.** The compressor sometimes swaps them when converting text to codes and back. - **Unproven scale.** The largest model the authors trained is under 1 billion parameters. Behavior at 27B or hundreds of billions is unknown. - **Limited release.** The code is for evaluation only. There is no training code. ## Should you try NCM? If you run models on your own GPU for one user and care about latency, it is worth testing. The idea is neat and the single-request speedup held up in independent runs. If you serve many concurrent users or run parallel agents, current evidence says a standard LLM stays faster. Quality on natural text also still favors the autoregressive baseline in the paper's tables. ## Frequently Asked Questions ### Is NCM a diffusion model? No. Diffusion models redraw every position on every pass. NCM predicts a hierarchy of compressed codes from coarse to fine, so early passes touch only a few codes and later passes add detail. ### Does NCM replace autoregressive LLMs? The evidence so far says no. It is faster for single requests in small-model tests, but slower under batching, weaker on text quality in the authors' tables, and untested beyond sub-billion-parameter sizes. ### What does "concept" mean in Next Concept Modeling? It means a learned compression code drawn from a fixed list of about 4,000 entries. It is not a human-readable idea or sentence. ### Why did NCM score 86% on hard Sudoku? Part of it is the architecture, since it refines the whole board at once and can fix early mistakes. But a same-size diffusion model that filled the most confident cell first came close, and reached 95% with more passes. Fill order explains much of the gap. ### Where does the 31x speed figure come from? It comes from the paper's comparison with the strongest diffusion model on news summarization, run on a single H100. It is not a comparison with autoregressive LLMs.