How to Run Recursive Flint Locally: SGLang, GPUs and Setup

What hardware do you need to run Recursive Flint locally?

You need one NVIDIA B300 or H200 GPU per replica, plus about 80 GB of disk for the BF16 checkpoint. Recursive ships an SGLang serving kit with the weights, so the standard route is to download the model, build a Docker image from the kit, and launch it with the serve script that matches your GPU. The server exposes an OpenAI-compatible API on port 30000. Quantized NVIDIA and Apple Silicon variants are prepared separately.

What is Recursive Flint?

Flint is a decoder-only mixture-of-experts Transformer from Recursive. It targets agentic coding work: investigating codebases, fixing bugs, implementing features and writing tests. It takes text in, produces text out, supports reasoning, and handles multi-turn tool use.

The model card reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2 and 74.2% on SWE-bench Verified. Treat those as vendor-run numbers. The card notes that evaluation setups vary across sources, and that some comparison scores are developer-reported rather than measured by Recursive.

Why does an 80 GB model fit on one GPU?

Flint has 39.8B total parameters. At BF16, two bytes each, that lands near the 80 GB download. An H200 has 141 GB of memory, which leaves room for the weights plus cache and runtime overhead. The B300 has more still. I would not try the BF16 checkpoint on a single 80 GB card; the card names only the B300 and H200.

Only about 2.5B parameters are active per token (the active body alone is about 2.2B). Each of the 47 MoE layers has 256 routed experts, with 6 selected per token plus an always-active shared expert. That sparsity helps per-token compute. It does not shrink the memory you need, because every expert must be resident.

Attention design helps with context memory. Of 49 layers, 35 use a 513-token sliding window and 14 use full attention. Four key/value heads serve 32 query heads. The documented context configuration is 262,144 tokens, prompt and output combined.

How do you set up Flint with SGLang?

The model card gives these steps. Run them from a machine with Docker and the NVIDIA container runtime.

pip install -U huggingface_hub openai
hf download RecursiveSI/flint --local-dir flint && cd flint   # about 80 GB
docker build -t flint-sglang sglang
docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model flint-sglang bash /model/sglang/serve_b300.sh   # H200: serve_h200.sh
until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done      # progress: docker logs -f flint
python3 sglang/example_client.py --base-url http://localhost:30000/v1              # reasoning and a tool call

A few details matter:

1. Pick the right script. The example uses serve_b300.sh. On an H200, swap in serve_h200.sh.
2. Pin the GPU. --gpus '"device=0"' assigns the container to the first GPU.
3. Stay local. The port binds to 127.0.0.1, so nothing is exposed to the network unless you change it.
4. Wait for health. The until loop polls /health_generate. Loading 80 GB takes a while, and docker logs -f flint shows progress.
5. Run the example client. It exercises both reasoning and a tool call, which makes it a quick sanity check.

The sglang/README.md in the repo covers requirements, serve-script options, multi-GPU deployments and API details. Transformers modeling code is not included yet, so SGLang is the supported runtime for the BF16 weights.

How do you call the server?

Any OpenAI-compatible client works. Point it at the local base URL and use flint as the model name.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")
response = client.chat.completions.create(
    model="flint",
    messages=[{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
    max_tokens=32768,
)
print(response.choices[0].message.content)

Flint supports OpenAI-compatible tool calls, so existing agent harnesses that speak that protocol should connect without a custom adapter. Two rules from the card matter for agent loops. Use the checkpoint's own chat template, and preserve the returned reasoning_content in conversation history on later turns.

What sampling settings does Flint recommend?

The server applies these defaults when a request leaves them out, so you only need to set them if you want to override. The benchmark results were also produced with these values.

The output budget deserves attention. Flint reasons before it answers, and reasoning tokens count toward the limit. If max_tokens is too low, the model can run out of budget mid-thought and return an incomplete answer. Prompt and output share one context window, so a long repo prompt shrinks the room left for generation.

Are there quantized versions?

Yes, but not in this repository. The card says quantized variants are prepared separately for NVIDIA and Apple Silicon runtimes. Its benchmark chart compares BF16 and NVFP4 checkpoints, which suggests an NVIDIA FP4 build exists. The card also links a separate local engine on GitHub. Those details, including memory needs and supported hardware, live outside the source material here, so check the Recursive Hugging Face organization before planning around them.

The card also warns that results can vary with checkpoint precision and inference runtime. If you move from BF16 to a quantized build, rerun your own evaluations.

Is Flint free to use commercially?

The model card says Flint is released under the Apache 2.0 license. Recursive states that hosted API terms and data practices are separate from the weight license. Self-hosting means the weight license applies. Using their hosted service means the API terms do.

What should you watch out for when self-hosting?

• Safety boundaries are yours to build. The card says structured tool-call grammar does not stop the model from choosing an unsafe or incorrect action. Enforce permissions and sandboxing in your agent harness.
• Review generated code. Test it and review consequential changes before applying them.
• Long reasoning costs latency. Hard tasks can consume a large share of the token budget.
• Context is capacity, not a quality promise. The 262,144-token configuration does not guarantee uniform accuracy across lengths.
• Report issues properly. For serving problems, Recursive asks for the checkpoint revision, runtime version, hardware, launch command and a minimal reproduction.