The 5 Best Small AI Models for Local Inference in 2026

What are the best small AI models for local inference right now?

Five models answer that question well: Jev, Julia-1, Ember-1, Bonsai 2, and OrcaSAQ2. All shipped within a single month, and each either runs entirely on hardware you already own or costs pennies per thousand calls when hosted. Between them they cover the two jobs most internal apps actually need: fast, bounded decisions (classify this ticket, route this request, score this answer) and lightweight reasoning (summarize, plan, write code), with no GPU cluster or metered API bill standing in the way.

Most teams are paying SaaS classification and routing vendors for work a much smaller, much cheaper model can now do on a laptop. Below: what each of these five models actually is, what it costs to run, and where it fits in a stack you own instead of rent.

Why "small" doesn't mean generic anymore

Until recently, "small model" meant a scaled-down chatbot: worse than GPT-4 at everything, useful for nothing in particular. The September 2026 crop is different because each model was built for one narrow job and tuned hard against it. Three categories are worth separating:

• Decision-only models. Jev and Julia-1 don't write text. They take a state and a set of possible answers and return a typed choice, a score, or a yes/no, with a calibrated probability attached.¹²
• Compressed reasoning models. Bonsai 2 and OrcaSAQ2 take a real 27-billion-parameter reasoning model and quantize it down to a few gigabytes, trading a small amount of accuracy for the ability to run on a single consumer GPU.³⁴
• Token-efficient distillations. Ember-1 keeps the full size and quality of its base model but is trained to reach the same answer using far fewer reasoning tokens, which matters when you're paying by the token even on hosted infrastructure.⁵

Each category answers a different stack question: does this task need a model at all, does it need to run on-device, or does it just need to stop burning tokens?

1. Jev: the decision-only model that can't hallucinate

TypeSafe AI calls Jev a "System One" model. It never generates free text; it only returns a typed choice, score, or yes/no answer ("noul" in their terminology), trained with a method the company calls Reinforcement Learning for Calibrated Decisions.¹⁶ Because the output space is constrained, there's no room for the model to invent an answer that isn't on the list.

The economics are the headline: $0.042 per million input tokens with output free, and TypeSafe claims 40 to 200x faster response times than frontier LLMs on its own workflow evaluations, with end-to-end latency of 70 to 500ms.¹ Independent testing bears this out at a smaller but still meaningful scale: in an OpenRouter benchmark on a 60-ticket support-triage task, Jev cost $0.0248 per 1,000 requests at 194ms median latency, versus $2.8802 per 1,000 requests at 1,957ms for Claude Opus 5, while matching its accuracy closely.⁷

The catch for a "local inference" list: Jev is hosted-only. There are no downloadable weights, so it can't run on your own hardware.⁶ TypeSafe points to open alternatives like Laya and kev, which implement a similar typed-decision interface for teams that want to self-host the pattern.¹⁶ Jev earns its spot here as the benchmark the rest of this list gets measured against, not as something you'll deploy on-prem.

2. Julia-1: a 144M-parameter classifier that runs on a laptop CPU

Julia-1 is what happens when someone takes Jev's idea and makes it genuinely ownable. Supersonic Labs built it on JHU CLSP's mmBERT-small encoder, released it under Apache 2.0, and shipped it at 144.3 million parameters and 550.5 MiB of FP32 weights.²⁸ It handles the same job as Jev, classification, ordered scoring, and yes/no decisions over a list of candidate answers, entirely on CPU.

Its benchmark results are a mixed but telling picture. Julia-1 beats the Jev reference numbers on AG News (94% vs 91%) and DAIR Emotion (86% vs 48%), but falls well behind on the 72-label Banking77 task (64% vs 87%), and lands at 73.15% on a 2,000-question typed-decisions benchmark against Jev's 72.70%.² It also reaches 71.50% macro accuracy across 52 languages on the MASSIVE scenario-classification benchmark, which matters if your routing logic needs to work outside English.²⁸

The number that actually sells this model is the latency and the training bill. Julia-1 runs a decision in a median of 33.15ms on an Apple M4 and 107.83ms on an Intel Core i5-1235U laptop chip, and the whole thing was trained for roughly $104 in cloud GPU spend.⁸ That's the clearest example on this list of "own it, don't rent it": a model cheap enough to train yourself, small enough to fork, and fast enough to embed in a request path without anyone noticing.

3. Ember-1: a reasoning model trained to think in fewer tokens

Ember-1 is a different kind of small. Fireworks AI took Moonshot AI's Kimi K3, an open-weight, 2.8-trillion-parameter mixture-of-experts model, and post-trained it to hit the same quality bar using about 40% fewer reasoning tokens.⁵ Across seven benchmarks and two live customer A/B tests, the reduction held at 35 to 50% fewer tokens per task without giving up accuracy.⁵⁹

It belongs on a "small model" list not because it's physically small, but because it makes an oversized model behave like a cheap one. Kimi K3 itself requires roughly 1.4TB of storage and 18 or more enterprise GPUs just to load, so almost nobody is self-hosting the base model regardless of its open-weight status.¹⁰ Ember-1 is API-only, offered as a research preview on Fireworks' serverless platform at the same per-token price as Kimi K3, $3 per million input tokens and $15 per million output tokens, and Fireworks hasn't released weights or training code.⁵⁹ It's not local inference in the strict sense, but if your reasoning workload already runs on Kimi-class models through an API, swapping to Ember-1 is a straightforward way to cut the bill without touching accuracy.

4. Bonsai 2 27B: a 27B reasoning model compressed to 5.9GB

Bonsai 2, from Prism ML, takes Alibaba's Qwen3.8-27B and compresses it from roughly 54GB in FP16 down to 5.95GB using ternary weight quantization, an effective 1.72 bits per weight.³ Despite that compression, it retains 98.2% of the full-precision model's score across 14 thinking-mode benchmarks, 84.78 versus 86.32.³ That gap is small enough to be invisible in most internal use cases.

What sets it apart from a conventional low-bit quantization is quality at the same size: a standard 2-bit build of the same base model scores 72.59 on the same benchmark suite, well behind Bonsai 2's 84.78, and Bonsai 2 comes within 0.4 points of a 4-bit version roughly three times its size.³ It's genuinely fast on consumer hardware too, running around 130 tokens per second on an RTX 5090, 30 tokens per second on a 72-watt L4, and 47 tokens per second on an Apple M5 Max laptop through a custom llama.cpp and MLX fork.³¹¹ Licensed Apache 2.0, this is the pick for a team that wants frontier-adjacent reasoning running fully on a machine it owns, with no cloud dependency at all. We've covered the real math behind running it on one GPU in more depth elsewhere.

5. OrcaSAQ2 27B: fitting a 27B agent model on a single 16GB GPU

OrcaSAQ2, built by OrcaRouter on the same Qwen3.8-27B base as Bonsai 2, takes a different compression approach: sensitivity-aware mixed-precision quantization averaging 3.21 bits per weight, shrinking the checkpoint to 12.3GB, a 77% reduction from the 54GB original.⁴¹² The fidelity numbers are tight: perplexity within 0.02% of the BF16 reference and 93.2% top-1 token agreement, meaning roughly one token in 15 differs from what the full-precision model would have chosen.⁴¹²

The practical payoff is that it fits and runs on a single 16GB consumer GPU, the RTX 4080/5080 class, through vLLM at 65.3 tokens per second single-stream, or 90.1 tokens per second with multi-token prediction enabled.⁴ On coding-agent benchmarks it scores 70.0% on SWE-bench Verified and 58.4% on Terminal-Bench 2.1, competitive with some much larger frontier models on those specific leaderboards, though reviewers note those scores depend heavily on the agent scaffold used and aren't a controlled apples-to-apples comparison.⁴ For teams evaluating it alongside other local coding models, that caveat is worth keeping in mind.

There's a real deployment cost, though. OrcaSAQ2 currently needs a custom vLLM kernel and doesn't yet run in llama.cpp, Ollama, or LM Studio, and practical context on 16GB hardware sits closer to 32K tokens than the architecture's full 262K.⁴¹² It's the most capable model on this list, but also the one that asks the most of your infrastructure team.

When does a local model replace a SaaS classification subscription?

The math here is less about model quality and more about volume and task shape. Industry cost analyses put the self-hosting break-even point around 1 to 2 billion tokens a month of steady traffic; below that, per-token API pricing usually beats owning a GPU once idle time and maintenance are counted.¹³ But that break-even line moves sharply for classification, extraction, and routing tasks, which are repeatedly flagged as work where a small model is "good enough" regardless of volume.¹³

A concrete example: routing 700,000 classification requests a month through a small open model costs $28.49, versus $1,037.40 for hardcoding a larger hosted model like Claude Sonnet 4.6 to do the same job, a roughly 36x difference.¹⁴ The OpenRouter triage benchmark tells the same story at a different scale: $0.0248 per 1,000 requests for Jev versus $0.0921 for a small hosted LLM and $2.88 for a frontier model.⁷ None of these numbers require the local model to be smarter than the API. They just require the task to be bounded enough that a tiny model doesn't need to be smarter.

A simple framework for deciding:

1. Is the output space bounded? If the answer is one of a known set of labels, scores, or yes/no calls, you're in decision-model territory, and a SaaS classification API is almost certainly overpriced for the job.
2. Does the task need context beyond a few thousand tokens? If yes, a compressed reasoning model like Bonsai 2 or OrcaSAQ2 fits better than a pure classifier.
3. Is the workload steady and high-volume, or bursty and occasional? Below roughly a billion tokens a month, a hosted small model or token-efficient distillation like Ember-1 may still beat the operational cost of running your own GPU.¹³

If your app is doing more than one of these jobs, you don't need to pick a single model. A router paired with a few specialists handles this better than betting everything on one model, and it's the same pattern covered in our look at running AI locally for enterprise, where the local router becomes the piece of infrastructure you own rather than the subscription you renew. Teams wiring this together without writing custom orchestration code have also started reaching for builders like Remy to assemble the router and specialist calls into an internal app without touching Python.

Which model fits your stack?

Match the model to the shape of the task, not the size of the benchmark:

• Pure classification or routing, high volume, need CPU-only deployment: Julia-1. It's free to run, Apache 2.0 licensed, and fast enough on a laptop CPU that you'll never think about it again.
• Pure classification or routing, want a managed API without infrastructure work: Jev. Cheapest hosted option per decision, but plan for it living outside your firewall.
• On-device reasoning agent, want the widest hardware support: Bonsai 2. It runs on everything from an RTX 5090 to a laptop-class Apple chip and needs no special serving stack.
• On-device reasoning agent, have a spare 16GB GPU and can tolerate a custom vLLM setup: OrcaSAQ2. Highest capability per gigabyte on this list, at the cost of tooling maturity.
• Token-cost governance on a hosted reasoning workload you're not moving off the cloud: Ember-1. Same accuracy, fewer tokens, same price per token as the base model it replaces.

None of these five models is trying to be everything. That's the point. The SaaS classification and routing tools this list is closing the gap on were never selling intelligence. They were selling a shortcut around building this yourself, and in 2026 that shortcut costs less to build than to rent.