How to Run Spark-X2.6-4B Locally: Ollama, llama.cpp, MLX, vLLM
Run Spark-X2.6-4B, an Apache 2.0 4B on-device agent model with 1M-token context, via Ollama, llama.cpp, LM Studio, MLX, vLLM or SGLang.
How do I run Spark-X2.6-4B locally?
The fastest route is Ollama: run ollama run SparkLLM/Spark-X2.6-4B. Other supported routes are LM Studio (0.4.0+), llama.cpp (build b10828+), Unsloth Studio, the Spark-MLX-LLM project for Apple silicon, and the Transformers library. For serving an OpenAI-compatible API, the model card documents SGLang and vLLM. The weights are Apache 2.0, and the model has 4B parameters.
TL;DR
- Spark-X2.6-4B is a 4B-parameter, Apache 2.0 model aimed at on-device agents, with the same architecture as Spark-X2.5-4B and gains that come from better post-training.
- The model card claims best-in-class results among similarly sized models, and says it matches or beats some 9B models on several benchmarks. Those results are in charts, so check them yourself.
- You can run it through Ollama, LM Studio, llama.cpp, Unsloth Studio, MLX, Transformers, SGLang or vLLM, each with a minimum version or specific flags noted below.
- SGLang is configured for a 1,048,576-token context window, but that needs a lot of device memory, so lower
--context-lengthif you run out. - The card lists three thinking levels (none, low, high). The only switch it shows in code is an
enable_thinkingon/off flag, and thinking is on by default. - Recommended sampling in the card's examples is temperature 1.0 and top_p 0.95, with top_k disabled.
What is Spark-X2.6-4B?
Spark-X2.6-4B is a compact general-purpose language model from the SparkLLM team, published on Hugging Face as XHToken/Spark-X2.6-4B. It follows Spark-X2.5-4B by a little over a month. The architecture is identical. The team says the improvements come from post-training, and it names coding, agent tasks, long context, multilingual use and reasoning as the areas that moved.
The pitch is agentic work on small hardware: complex tool use, multi-step planning and software engineering. The card compares it against Qwen3.5-4B and Gemma-E4B on inference efficiency and says it is on par or better, while being more capable. The repository holds the model weights and configuration files for the post-trained model in Hugging Face Transformers format.
Which local runtime should I pick?
It depends on whether you want a chat app, a command-line tool or an API server.
| Route | Best for | Version or requirement from the card |
|---|---|---|
| Ollama | Quick local chat and a simple API | v0.34.1 or later |
| LM Studio | GUI users, plus the lms CLI | 0.4.0 or later |
| llama.cpp | GGUF, CLI and OpenAI-compatible server | b10828 or later |
| Unsloth Studio | GGUF inference in a desktop app | Latest version |
| Spark-MLX-LLM | Apple silicon GPU, Linux CPU, CUDA | No GGUF conversion needed |
| Transformers | Python scripting and experiments | transformers==4.57.1 |
| SGLang | Serving with tool and reasoning parsers | Pre-built Docker image |
| vLLM | Serving with prefix caching | Docker image, Spark plugin on Ascend |
How do I run it with Ollama, LM Studio or llama.cpp?
These are the lowest-friction options.
Ollama. Install v0.34.1 or later from ollama.com, then:
ollama run SparkLLM/Spark-X2.6-4B
LM Studio. Install 0.4.0 or later, search for "Spark-X2.6", download the model, load it and chat. The CLI route is:
lms ls
lms chat spark-x2.6
llama.cpp. Native spark2_5 support requires build b10828 or later. Older builds will not recognize the architecture. The card gives these commands, pointing at a GGUF version of the model:
llama cli -m /path/to/Spark-X2.6-4B-GGUF
llama serve -m /path/to/Spark-X2.6-4B-GGUF
The second command starts an OpenAI-compatible API server. Your install may expose the binaries under different names (such as llama-cli and llama-server), so use whatever your build provides.
Unsloth. The card says the latest Unsloth Studio supports native Spark-X2.6 GGUF inference. It gives no further setup steps.
How do I run it on a Mac with MLX?
Spark-MLX-LLM runs the original Hugging Face checkpoints directly, so you skip GGUF conversion. It supports Apple silicon GPU, Linux CPU and NVIDIA CUDA on Linux.
git clone https://github.com/XHToken/Spark-MLX-LLM.git
cd Spark-MLX-LLM
python3 -m venv .venv
source .venv/bin/activate
# Apple silicon
python -m pip install -e .
Linux users swap the last line for '.[cpu]', '.[cuda12]' or '.[cuda13]'. Then generate:
spark-mlx-generate \
--device gpu \
--dtype bfloat16 \
--model XHToken/Spark-X2.6-4B \
--prompt "What is the capital of Anhui Province?" \
--max-tokens 512 \
--temp 0
How do I serve it with SGLang or vLLM?
Both expose an OpenAI-compatible endpoint. Set MODEL_PATH to the absolute path of your downloaded checkpoint first.
SGLang (NVIDIA). The card points to a pre-built nightly image, lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1. The key flags are:
python -m sglang.launch_server \
--model-path /root/Spark-X2.6-4B \
--served-model-name spark2.6 \
--tool-call-parser spark25 \
--reasoning-parser qwen3 \
--tp-size 1 \
--mem-fraction-static 0.8 \
--context-length 1048576 \
--chat-template /root/Spark-X2.6-4B/chat_template.jinja
The spark25 tool-call parser matters if you are building agents. The qwen3 reasoning parser separates thinking output from the final answer. Ascend NPU images are listed too.
vLLM. The card uses the official vllm/vllm-openai:latest image with --trust-remote-code, --enable-prefix-caching, --gpu-memory-utilization 0.7 and the model's own chat template. For Ascend A2, A3 and 950DT hardware, you run a vllm-ascend container and install the Spark plugin from github.com/XHToken/Spark-plugin inside it. Note that the card's examples serve the model under the name spark25, so your client's model field must match whatever you pass to --served-model-name.
Transformers. For scripts, install transformers==4.57.1 and load with trust_remote_code=True. The example uses apply_chat_template with enable_thinking=True, then samples with temperature 1.0, top_p 0.95 and top_k 0.
How does the thinking budget work?
The card advertises three thinking levels: none, low and high. The goal is to let an app trade reasoning quality against latency and cost per request.
What the card actually shows in code is a simpler switch. Thinking is on by default in both the chat template and the Qwen3 reasoning parser. To turn it off for one request on an SGLang server, pass this in the request body:
"chat_template_kwargs": {"enable_thinking": false}
The excerpt of the card does not show how to select low versus high. If you need that, read the chat_template.jinja file shipped with the weights or check the full repository before assuming a parameter name.
A practical pattern: turn thinking off for routing, extraction and short lookups, and leave it on for planning and code changes. Also set a sensible max_tokens. The card's examples use 131072, which is a ceiling, not a target.
How much memory does it need?
The card does not publish VRAM or RAM figures, so there is nothing authoritative to quote. Two things are clear. First, 4B parameters is small enough that desktop and laptop hardware is the stated target. Second, the 1,048,576-token context setting in the SGLang example "requires sufficient device memory," and the card tells you to reduce --context-length when needed. Long context is where memory grows, so start with a modest window and raise it while watching usage.
Is Spark-X2.6-4B worth trying?
If you want a small, permissively licensed model for local agents, yes, and the setup cost is low. Ollama or LM Studio gets you running in minutes. The comparison claims (against Qwen3.5-4B, Gemma-E4B and some 9B models) come from the model's own authors, in charts, so test it on your own tool-calling and coding tasks before committing. The card recommends Llama-Factory for fine-tuning if you want to adapt it.
Frequently Asked Questions
What license does Spark-X2.6-4B use?
The Spark-X2.6 model series is released under the Apache 2.0 license. That permits commercial use and modification under the license's usual conditions.
How long is the context window?
The SGLang example is configured for a maximum context of 1,048,576 tokens. The card warns that this needs enough device memory and tells you to lower --context-length if necessary.
Can I turn off thinking?
Yes. Thinking is on by default. On an SGLang server, send "chat_template_kwargs": {"enable_thinking": false} with the request. In Transformers, the chat template call takes an enable_thinking argument.
Does it work on Apple silicon?
Yes. Spark-MLX-LLM runs the Hugging Face checkpoints on the Apple silicon GPU without GGUF conversion. Ollama, LM Studio and llama.cpp are also listed routes, though the card does not single out Mac performance for them.
What is the difference from Spark-X2.5-4B?
The architecture is the same. The card says the gains in coding, agent tasks, long context, multilingual use and reasoning come from improved post-training.
How we source, write, and correct our articles. Editorial standards
