Is Nanbeige4.3 33B Claude-Level? Benchmarks vs Real Tests

Is Nanbeige4.3 33B actually Claude-level?

Not based on the evidence available so far. On the lab's own published benchmarks, Nanbeige4.3 33B beats Qwen3.8-27B, Muse-Glimmer-30B and Gemma4-31B on most agentic coding and cowork tests. But in two hands-on runs inside Hermes Agent (a real bug fix and a single-file HTML animation), it thought in circles for an hour or more and never delivered. Strong charts, failed practical tests, no proof of Claude-level behavior.

What is Nanbeige4.3 33B?

Nanbeige4.3-33B is the flagship of the Nanbeige4.3 series. The lab says it built a high-capacity teacher agent first, which will anchor distillation of agentic skills into smaller models such as a planned Nanbeige4.3-3B. There is also a base model, and the released model is fine-tuned from it.

Key specifics from the model card:

• Context length of up to 524,288 tokens (512K).
• A configurable chat template with enable_thinking and preserve_thinking switches.
• Support for Transformers, SGLang and vLLM.
• A recommendation to use tool_call_format="xml" for the best tool-calling results.

The lab has long argued that its small models can beat larger ones. This release extends that pitch to agentic coding and professional "cowork" tasks.

How good are the published Nanbeige4.3 benchmarks?

On paper, very good. The lab compares against three similarly sized models, all at their highest thinking effort, with the same inference hyperparameters and harness per benchmark. Here is a selection from its table. For every result, see the full 20-benchmark comparison.

The gaps are largest on hard terminal and software-engineering tasks, where some rivals barely register. The close races are SWE-Bench Pro and GDPval, where Qwen3.8-27B is slightly ahead, and GPQA Diamond, where Gemma4-31B leads.

The lab also shows a per-occupation breakdown on JobBench across 35 occupations, from IT and engineering to finance, healthcare and law. It says Nanbeige4.3-33B beats the baseline in most of them. In the chart as described by the reviewer, it roughly doubles Qwen's score for roles like web administrators and lawyers, while Qwen still wins a few, such as healthcare managers and personal financial advisers.

One footnote matters: Nanbeige's own numbers use preserve_thinking=true, and each benchmark has a specific harness (Terminus 2, mini-swe-agent v2.4.6, Claude Code and others). Results are tied to those setups.

What happened in the hands-on tests?

A reviewer ran the model locally through vLLM and Hermes Agent, where it was described as a Claude-level model. He wanted to see how close that was.

Test 1: fix a real bug. The app was a restaurant kitchen costing system for exotic meats. The bug: the costing engine ignored trim and yield loss. A wallaby shank with a 46% yield showed a 220 g plate but only 101 g to buy, when the purchase weight should be larger than the serving. That made food cost look like an "excellent" 18.9% and every margin bar green. The model was asked to find the bug, fix it and test thoroughly.

Results:

• After about 20 minutes it was still thinking heavily and compacting context, advancing a little at a time.
• After more than an hour it had flagged 14 or 15 issues. The app had one bug, and the model never identified it.
• The reviewer cancelled the run and called it a fail.

Test 2: single-file HTML animation. The prompt asked for a string of Mexican banners hanging across a plaza, with bright colors and doves, swaying gently. This is not a hard simulation. After more than 45 minutes the model hit its max output tokens and the full context length without finishing the HTML, CSS or JavaScript.

The reviewer's overall verdict was that it is not production ready, and that even Claude Haiku and Muse-Glimmer looked better in how they reasoned. He did not actually run the bug test on Haiku, so that comparison is impression, not data.

Why would benchmarks and real use disagree?

Several explanations fit, and the evidence doesn't separate them.

• Harness fit. Benchmark scores come from specific scaffolds with tuned settings. A different agent harness may trigger different behavior, such as the context compaction seen here.
• Settings. The card recommends temperature 1.0, top_p 0.95 and 65,536 max new tokens for agentic work, plus preserve_thinking=true and XML tool calls. It isn't clear from the test whether all of these matched.
• Open-ended tasks. Benchmarks usually have a bounded, checkable target. A "find the one hidden bug" task rewards restraint, and a model that over-thinks can chase false leads indefinitely.
• Thinking budget. The model runs at high reasoning effort in the lab's comparisons. Without a cap, long thinking can consume the whole output budget.
• Small sample. Two tests by one reviewer is anecdote, not a benchmark.

None of this proves the published numbers are wrong. It does suggest that a lead on Terminal-Bench or DeepSWE does not guarantee smooth behavior in your own agent loop.

What does it take to run Nanbeige4.3 33B?

The reviewer served it with vLLM and saw just under 88 GB of VRAM in use at his chosen context length. He estimated that even with a shorter context, full precision would still need around 62 GB. The model card's example launch commands use four-way tensor parallelism.

vllm serve ${MODEL_PATH} \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.8 \
  --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

This is not a laptop model at full precision. No quantized benchmark results were provided in the sources.

Should you use it?

Treat it as a promising model to evaluate, not a Claude replacement. If you build agents, test it on your own tasks with the card's recommended settings and a hard cap on thinking and output. Watch for loops, runaway token use and false-positive bug reports. The benchmark lead over Qwen3.8-27B is large enough to justify a trial. The failed practical runs are a good reason not to ship it unattended.