Topic

Open Model Run Guides

Run guides for specific named open-weight models: how to run each one locally, with the right runtime (Ollama, llama.cpp, vLLM, SGLang, MLX), quantization, VRAM and context needs. General setup, tooling and hardware coverage stays in Local & Open-Weight Models.

15 articles·Updated Oct 23, 2026

Diagram-style illustration of a mixture-of-experts coding model, with a large block of parameters routed so only a few active experts load into a local GPU and memory assembly that produces code output.

Oct 23, 2026

How Do You Run Recursive Flint Locally? Hardware and Setup

Run Recursive Flint, a 40B MoE coding model with 2.5B active parameters, locally with llama.cpp or Rec. Hardware, GGUF size, settings and OpenCode fix.

Recursive Flintrun Flint locallyFlint llama.cpp

Exploded cutaway of a GPU card with a large model compressed into a small quantized file that fits into its memory, with three runtime modules attached, illustrating running a 40B model locally.

Oct 23, 2026

How to Run Flint 40B Locally: Ollama, GGUF, SGLang and VRAM Needs

Install Flint 40B locally with Ollama, llama.cpp or SGLang. Covers VRAM needs (about 32GB for the quant), GGUF Q4, NVFP4, MLX and BF16 on H200 or B300.

Flint 40Brun Flint locallyRecursive Flint Ollama

A pipeline diagram-style illustration of a model's context window of token blocks passing through a scaling stage that doubles its length, fed by two launch units, representing serving Sage-A1.5 at 512K context with YaRN.

Oct 23, 2026

How Do You Serve Sage-A1.5 With SGLang or vLLM at 512K Context?

Serve Sage-A1.5 with SGLang or vLLM: standard and MTP launch commands, parsers, and the YaRN factor 2.0 setup that extends 256K context to 512K.

Sage-A1.5 SGLangSage-A1.5 vLLMSage-A1.5 512K context

A cutaway graphics card with a compact grid of mixture-of-experts cells in which only a few are raised, next to a thin stacked KV cache and a densely packed quantized weight cartridge.

Oct 23, 2026

How to Run Raon-1 mini Locally: NVFP4, RTX 5090 and Context Needs

Raon-1 mini is a 28.8B MoE with 4.3B active parameters. See what NVFP4, MXFP8, DFlash2 and an 8 KiB per token KV cache mean for local hardware.

Raon-1 miniKRAFTON RaonRaon-1 mini RTX 5090

A cutaway graphics card with a stack of model slabs beside it, a few active expert slabs sliding into its memory, and a long ribbon of context tiles feeding in, illustrating a mixture-of-experts model running on one GPU.

Oct 23, 2026

What is Raon-1 mini and can it run on one RTX 5090?

Raon-1 mini preview is KRAFTON AI's Apache 2.0 30B-A4B MoE with hybrid attention and 262K context. Specs, NVFP4 on a 32GB RTX 5090, and benchmarks.

Raon-1 miniKRAFTON RaonRaon-1 mini NVFP4

A graphics card with a partly filled bank of video memory and a slotted stack of quantized mixture-of-experts layers, with a few active experts lifted out and a speech waveform extending from it, representing running a text-to-speech model locally.

Oct 23, 2026

Can You Run Fish Audio Drama 3 Locally? VRAM and Setup Guide

Fish Audio Drama 3 is an open-weight MoE text-to-speech model. Here are the VRAM needs, the 4-bit quant, bracket directions, and how it ranks.

Fish Audio Drama 3run Drama 3 locallyopen-weight text to speech

Exploded cutaway of a GPU accelerator module showing a mixture-of-experts model's layers, weight shards loading into memory, and a containerized inference server for self-hosting a coding model.

Oct 22, 2026

How to Run Recursive Flint Locally: SGLang, GPUs and Setup

Self-host Recursive Flint, a 39.8B MoE coding model, with SGLang on one H200 or B300. Covers the 80 GB download, Docker steps, sampling settings and licence.

Recursive Flintrun Flint locallyFlint SGLang

Diagram-style illustration of a large language model compressed into a quantized GGUF file, loaded across three GPUs with a VRAM fill gauge, and its output split into reasoning and final answer streams.

Oct 22, 2026

How to Run Ornith-1.5-122B Locally with llama.cpp and vLLM

Run Ornith-1.5-122B locally: llama.cpp GGUF setup, VRAM and context trade-offs, runtime versions to check, and how to handle thinking-mode output.

Ornith 1.5 122Brun Ornith locallyOrnith llama.cpp GGUF

Illustration of a large accelerator card serving a mixture-of-experts model, with a few active tiles lifted from a grid of dormant ones, feeding a memory store for Mem0 or MemOS.

Oct 22, 2026

How to Run Hy-Memory-30B-A3B Locally with vLLM for Mem0 and MemOS

Serve Tencent's Hy-Memory-30B-A3B with vLLM as hy-memory, call it via the OpenAI-compatible API, and use it as the write path in Mem0 or MemOS.

Hy-Memory-30B-A3B vLLMrun Hy-Memory locallyMem0 local memory model

A small chip module with a long unspooled ribbon of layered slices representing a 1M-token context, connected by six differently shaped sockets representing the ways to run the Spark-X2.6-4B model locally.

Oct 22, 2026

How to Run Spark-X2.6-4B Locally: Ollama, llama.cpp, MLX, vLLM

Run Spark-X2.6-4B, an Apache 2.0 4B on-device agent model with 1M-token context, via Ollama, llama.cpp, LM Studio, MLX, vLLM or SGLang.

Spark-X2.6-4Brun Spark X2.6 locallyon-device agent model

Cutaway diagram of a four-layer language model with a loop-back path, shown beside a tall full KV cache column and a short last-loop-only cache column on a unified-memory chip.

Oct 22, 2026

How to Run Nemotron Dory 1.0 4B Locally and Pick a Loop Count

Run NVIDIA Dory1.0-4B locally with vLLM or llama.cpp on Apple Silicon, choose 1, 2 or 4 loops, and cut KV cache memory with last-loop caching.

run Dory 4B locallyDory llama.cppDory vLLM

A large model-weights block plugged into a dock with an empty, accented slot where a chat template would sit, a token ribbon feeding straight in, and a long context ruler behind it.

Oct 22, 2026

How to Run DeepSeek-V4.1-Flash-1017 Without a Jinja Chat Template

DeepSeek-V4.1-Flash-1017 ships MIT-licensed with no Jinja template. See sampling settings, 1M context, the encoding folder and deepseek-recipe.

DeepSeek-V4.1-Flash-1017run DeepSeek V4.1 Flash locallyDeepSeek V4.1 prompt encoding

A large layered cube of data is shown compressed into a much smaller, denser cube that fits into a blank hardware slot, illustrating model quantization for local inference.

Oct 21, 2026

How to Run Distilled AI Models Locally (DeepSeek-R1, Ollama)

Learn how to pick, quantize, and run distilled models like DeepSeek-R1 locally with Ollama or llama.cpp, and when self-hosting beats API costs.

run distilled AI models locallyDeepSeek-R1 distilled localOllama

Diagram-like scene of a huge grid of dormant model-expert cubes on a storage slab with a small stream of cubes rising through a memory tier to a compact compute block, illustrating streaming inference of a large mixture-of-experts model from SSD.

Oct 21, 2026

How to Run DeepSeek R1 671B Locally by Streaming Experts Off SSD

Learn how to run 671B MoE models like DeepSeek R1 locally with llama.cpp, KTransformers and Unsloth quants, with hardware tiers and tokens per second.

run LLM locallyDeepSeek R1 671B localllama.cpp n-cpu-moe

A hardware schematic arrangement showing a massive parameter matrix routing directly through a primary CPU into expansive memory banks, bypassing any secondary coprocessors.

Aug 31, 2026

How to Run GLM-5.3-Flash Locally on CPU (No GPU Required)

How to run the 320B-parameter GLM-5.3-Flash model on CPU with 128GB RAM using Unsloth's dynamic GGUF quants, plus the hardware math and exact commands.

run GLM-5.3-Flash locallyGLM-5.3-Flash CPUUnsloth GGUF quantization