How to Run StarCoder2 15B Locally: Hardware and Commands
Run StarCoder2 15B locally with Ollama or llama.cpp. Install commands, approximate quantized sizes, and the VRAM or unified memory each setup needs.
For Engineering and IT
How do you run StarCoder2 15B locally?
The quickest route is Ollama: install it, then run ollama run starcoder2:15b. The default download is a 4-bit quantized build of roughly 9 to 10 GB, which fits a 12 GB GPU or a Mac with 16 GB of unified memory. If you want more control over quantization and context length, use a GGUF file with llama.cpp. Full-precision weights through Transformers need around 30 GB, so most people quantize.
TL;DR
- StarCoder2 comes in 3B, 7B, and 15B sizes, and the 15B is the largest and the one this guide covers.
- Ollama is the fastest setup, and it works on macOS, Windows, and Linux with one install and one run command.
- llama.cpp gives you control over quant level, context size, and GPU offload, and it serves an OpenAI-style HTTP API.
- A 4-bit build needs roughly 9 to 10 GB for weights alone, so budget 12 GB of VRAM or a 16 GB Mac.
- Unquantized 16-bit weights need about 30 GB or more, which means a 32 GB+ card or multiple GPUs.
- This is a code completion model, so expect autocomplete and fill-in-the-middle behavior rather than polished chat.
What is StarCoder2 15B?
StarCoder2 is a family of code models from the BigCode project, released in 3B, 7B, and 15B parameter sizes. NVIDIA's launch writeup says the family supports over 600 programming languages and that the 15B model has a 16,000-token context window. The weights are released under the BigCode OpenRAIL-M license, which allows royalty-free distribution but carries use restrictions. Read the license before shipping anything commercial.
The Hugging Face repository bigcode/starcoder2-15b hosts the original safetensors weights. Those are meant for Transformers and vLLM. For Ollama and llama.cpp you want a quantized build, usually in GGUF format.
How much memory does StarCoder2 15B need?
The weights dominate. A rule of thumb: parameters times bytes per weight. The table below is my arithmetic from a 15B parameter count, not measured file sizes. Real GGUF files vary by a gigabyte or so depending on the quant recipe, so check the file listing before you download.
| Format | Approx. weight size | Fits in |
|---|---|---|
| FP16 / BF16 | ~30 GB | 32 GB+ GPU, or multi-GPU |
| 8-bit (Q8_0) | ~16 GB | 24 GB GPU, 32 GB Mac |
| 5-bit (Q5_K_M) | ~11 GB | 16 GB GPU, 16 GB Mac (tight) |
| 4-bit (Q4_K_M) | ~9 to 10 GB | 12 GB GPU, 16 GB Mac |
| 3-bit | ~7 GB | 8 GB GPU, quality drops |
Add memory for the KV cache, which grows with context length. Plan on at least a gigabyte or two on top of the weights at moderate context, and more if you push toward the full 16K window. That overhead is why a 4-bit build is comfortable on 12 GB but a 5-bit build is not.
On Apple silicon, the GPU shares unified memory with the OS and every other app, and macOS caps how much of it the GPU can use. A 16 GB Mac can run a 4-bit build, but close your browser tabs first. On 8 GB machines, use the 3B or 7B model instead.
How do you run it with Ollama?
Install Ollama. On macOS and Windows, download the installer from ollama.com. On Linux:
curl -fsSL https://ollama.com/install.sh | sh
Then pull and run the 15B model:
ollama run starcoder2:15b
The first run downloads the weights. Afterward, Ollama also exposes a local API on port 11434:
curl http://localhost:11434/api/generate -d '{
"model": "starcoder2:15b",
"prompt": "def fibonacci(n):",
"stream": false
}'
If you want the smaller siblings, swap the tag for starcoder2:3b or starcoder2:7b. Run ollama list to see what you have downloaded and how large each model is. That is the most reliable way to confirm the actual size on your disk.
How do you run it with llama.cpp?
llama.cpp suits you if you want to pick the exact quant, tune context, or run a server. Install it through a package manager on macOS or Linux:
brew install llama.cpp
Or build from source. Metal acceleration is on by default on Macs. For NVIDIA GPUs, enable CUDA:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
Next, get a GGUF file. Search Hugging Face for "starcoder2-15b GGUF" and pick a community conversion that lists the quant levels you want. Q4_K_M is the usual starting point. Download it:
huggingface-cli download <repo-name> <file-name>.gguf --local-dir models
Run a completion:
./build/bin/llama-cli -m models/<file-name>.gguf -ngl 99 -c 8192 -p "def fibonacci(n):"
The -ngl 99 flag offloads all layers to the GPU. If you run out of VRAM, lower that number to keep some layers on the CPU, or drop the context with -c. To serve an HTTP API for editor plugins:
./build/bin/llama-server -m models/<file-name>.gguf -ngl 99 -c 8192 --port 8080
Community GGUF conversions are not official, so check the repo's description and quant list before trusting one.
What about Transformers or vLLM?
Hugging Face's model page shows the standard Transformers load, using AutoModelForCausalLM.from_pretrained("bigcode/starcoder2-15b", device_map="auto"), and lists vLLM as a local app option. Use these if you have a large GPU or want to fine-tune or batch requests. Without quantization, you need the roughly 30 GB from the table above. Loading a 4-bit build through a library such as bitsandbytes cuts that sharply, at some cost in quality.
Which setup fits your hardware?
| Your machine | Best setup | What to expect |
|---|---|---|
| Laptop, 8 GB RAM | StarCoder2 3B or 7B in Ollama | The 15B will not fit comfortably |
| MacBook, 16 GB unified | Ollama, Q4 | Works, with little headroom. Keep context modest |
| MacBook, 32 GB+ unified | Ollama or llama.cpp, Q5 or Q8 | Room for higher quality and longer context |
| Desktop, 12 GB GPU | llama.cpp or Ollama, Q4 | Full GPU offload at moderate context |
| Desktop, 24 GB GPU | llama.cpp, Q8 | Near-full quality quant with long context |
| Workstation, 32 GB+ | Transformers or vLLM, FP16 | Unquantized, suited to batching and fine-tuning |
Expect a laptop to feel slower than a discrete GPU, since a model this size is limited by memory bandwidth. Generation on a Mac is usable for interactive completion, but a desktop GPU responds noticeably faster. I am not quoting tokens-per-second figures because they vary widely with quant, context, and chip.
Is StarCoder2 15B a chat model?
No. The base StarCoder2 models are trained for code completion, not instruction following. Give it a function signature, a comment, or the start of a file and it continues. It also supports fill-in-the-middle, where you provide the code before and after a gap, using special tokens such as <fim_prefix>, <fim_suffix>, and <fim_middle>. That is what editor autocomplete plugins use. If you ask it a question in plain English, results will be uneven. For chat-style coding help, pick an instruction-tuned model instead.
Frequently Asked Questions
Can I run StarCoder2 15B on a 8 GB GPU?
Only with an aggressive 3-bit quant, and quality suffers. The 7B model at 4-bit is a better fit for 8 GB, and the 3B model is lighter still. Partial offload to the CPU also works but slows generation.
Is StarCoder2 15B free for commercial use?
It is released under the BigCode OpenRAIL-M license, which permits royalty-free use and distribution but includes use-based restrictions. Read the license text on the Hugging Face model page for your case.
What is the context window?
NVIDIA's overview describes a 16,000-token window for the 15B model. Locally, a longer context costs more KV-cache memory, so many people run 4K to 8K to stay inside their VRAM.
Should I use Ollama or llama.cpp?
Start with Ollama if you want a working model in minutes. Switch to llama.cpp when you need a specific quant, finer control over GPU offload and context, or a server tuned to your editor setup.
Topics
Useful for
How we source, write, and correct our articles. Editorial standards
