Blog
vLLM Semantic Routersemantic router tutorialLLM routing local models

How to Set Up vLLM Semantic Router for Local Models

Install vLLM Semantic Router, serve two models on one GPU, and set routing rules in the dashboard or YAML so easy and hard prompts reach different backends.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Engineering and IT

A small figurine stands beside an oversized network router with two cables leading out of it to two separate devices.

What is vLLM Semantic Router?

vLLM Semantic Router is a routing layer that gives an AI application one endpoint for many models. Your app or agent sends every request to that single address. The router then picks a backend for each call based on a policy you write, choosing among local, private and cloud models. vLLM, an open-source engine for serving large language models, runs the models. The router decides which one gets each request.

TL;DR

  • One endpoint, many models. Your agent talks to the router, and the router chooses the backend per request.
  • vLLM serves, the router decides. They are separate tools that work together.
  • The install is short. Create a uv environment, install vLLM, then run the router's official launcher, which sets up its own isolated environment.
  • Two small models fit on one GPU. In the demo, two models of roughly 7 and 8 billion parameters used under 20 GB of VRAM on a 48 GB card.
  • Routing is a rule plus a signal. A "decision" is the rule that picks a model. A "signal" is something the router detects in a request that a decision can match.
  • The dashboard and the YAML file edit the same setup. Use the dashboard to learn, then the YAML file when you want precise control.

What do you need before you start?

You need a machine that can serve a model, ideally with a GPU. The walkthrough this article draws on used an Ubuntu box with a 48 GB RTX 6000. The presenter noted you can also run on CPU if the model allows it, but expect it to be slower.

You also need:

  • uv, a Python package and environment manager.
  • Docker. The router's installer detected Docker and used it as its runtime.
  • Model weights for two models. Pick one smaller model for easy prompts and one for harder ones. Both need to fit in your VRAM together.

I am not giving exact version numbers because they change. Check the vLLM and Semantic Router documentation for current requirements.

How do you install vLLM?

Start with a clean virtual environment so vLLM does not collide with other Python projects.

uv venv
source .venv/bin/activate
uv pip install vllm

Then verify the install. Check the vLLM version, and confirm that it can see your CUDA GPU. If the GPU does not show up, fix your drivers before going further. Nothing downstream works without it.

How do you install the Semantic Router?

Use the project's official launcher, found in its documentation. Run it from the terminal and let it work.

Two things happen that are worth knowing about:

  1. It creates its own isolated Python environment. It does not touch your vLLM environment. In effect, you get an environment inside an environment, so the two tools cannot break each other's dependencies.
  2. It starts the router and opens a dashboard in your browser. The first screen has an "enter dashboard" button, and the next asks you to create an administrator with a username and password.

Once you are in, the dashboard walks you through three steps: connect your models, choose routing, then review and activate.

How do you serve two models on one GPU?

Before connecting anything, serve your models with vLLM. Open a terminal for each and run vllm serve with a model name and its own port.

vllm serve <first-model> --port 8000 --gpu-memory-utilization 0.4
vllm serve <second-model> --port 8001 --gpu-memory-utilization 0.4

The model names and numbers here are placeholders. The point is that each model needs a different port. When two models share a GPU, you also need to cap how much memory each one claims, which is what the --gpu-memory-utilization flag does. Add other flags, such as tool-use options, if your agent needs them.

In the demo, both models were fully loaded and the GPU was still under 20 GB of VRAM. Check your own usage with nvidia-smi before moving on.

How do you connect the models in the dashboard?

Go back to the dashboard. It shows fields for your first model. Fill them in like this:

FieldWhat to enter
Model IDThe exact name of the model you downloaded and served
ProvidervLLM (the dropdown also lists Ollama, OpenAI-compatible and Anthropic-compatible APIs)
Base URLlocalhost plus the port that model is running on
Endpoint label"primary" for the first model. Use "secondary" or a name you choose for the next one
Access keyNot needed for a local vLLM server

Click "add model" to get another set of fields and repeat for the second model. The demo stopped at two, but the presenter noted you can add as many as your GPU allows.

The provider dropdown matters. Because it works with Ollama, OpenAI-compatible APIs and any Anthropic-enabled API, the same setup can point at a private server or a cloud model, not only local ones.

How do routing decisions work?

After connecting models, you choose how routing starts. The dashboard offers ready-made starting points:

  • Balanced. Trades off cost, quality and safety across five model tiers. It is designed for more models than the two in this setup.
  • Security. A privacy-first mode that keeps sensitive requests on local models and restricts cloud egress (data leaving to cloud providers).
  • Scratch. Starts empty so you define everything yourself.

The demo used scratch to show every option. The review page then confirmed the setup was ready, with the router listening and both models attached. It also offers a JSON view of the config.

Two terms you will see:

  • A decision is a routing rule that picks which model handles a request.
  • A signal is something the router detects in a request, such as a keyword, that a decision can match on.

Activate the setup, and the dashboard returns with your models and routes listed. In the demo, a "hard task" route pointed at the roughly 7 billion parameter model and a "small talk" route pointed at the 8 billion one.

How do you test routing in the playground?

The dashboard has a playground at the top. Use it to send a prompt and see which model answered and which decision fired.

Two tests from the demo:

  1. A greeting. Typing "hello" matched the small-talk decision and went to the 8 billion parameter model. (If you get an error here, check that both vLLM servers are still running. The demo hit exactly this after a model had been stopped.)
  2. A reasoning task. Asking whether a number is prime matched the hard-task route and went to the roughly 7 billion parameter model. It took longer to finish because the model worked through the problem.

Run a handful of prompts of your own. Include borderline ones. The playground shows the decision name, so you can see whether your rules match your intent before any real traffic arrives.

What can you change in the YAML file?

You do not have to use the dashboard. A YAML file defines the whole setup, and you can edit it directly or work from the command line. The demo pointed out these parts:

  • Where the router listens. You can change the address.
  • Models and their providers. Swap a model or provider, restart the router, and it follows.
  • Signals. These include keyword lists you can edit.
  • Decisions. These tie signals to models.

That opens up practical rules. Add finance terms as a signal and send those queries to a finance-oriented model. Or send certain keywords to a model you fine-tuned on your company's data. The demo also mentioned semantic routing and safety routing as options.

The dashboard's system page shows a health check for the pieces running underneath, including an Envoy proxy, a Grafana-based front end, Redis and the backends. You do not need to manage these for a first test, but they show this is built with production deployments in mind.

What does this mean for you?

If you use AI at work but do not build infrastructure, this matters in three ways.

Cost. Easy requests such as greetings, short lookups and reformatting do not need your biggest model. Routing them to a small one is cheaper and faster.

Privacy. A rule that keeps sensitive requests on local models means private data does not have to leave your machines. Ask your IT or security team whether a privacy-first policy fits your data rules.

Control. Routing rules are plain text you can read and review. That is easier to audit than "we use one big model for everything."

If you are a manager or analyst, you probably will not run this yourself. You can still ask the right questions: which requests go to which model, who wrote the rules, and what stays on-premises. If you are technical enough to run Docker, a weekend with one GPU and two small models is enough to see it work.

Can you build tooling around a router like this?

A router tells you which model answered, but it does not give your team a place to work with the results. A small internal tool can fill that gap: a request form where colleagues submit prompts, or a dashboard that logs which route each prompt took and flags ones you want to review. If you would rather own a tool like that than stitch together spreadsheets, you can describe it in plain English to Remy, which builds and deploys full-stack apps with a database and sign-in included. Remy works with its own model catalog, so treat it as a companion for the team-facing side, not a replacement for the router.

Frequently Asked Questions

5 questions

Do I need a GPU to use vLLM Semantic Router?

A GPU is the smoother path, and the demo used one. The presenter said you can also run on CPU if the model allows it. Expect slower responses.

Can the router send requests to cloud models too?

Yes. The router can select among local, private and cloud backends. The provider dropdown includes OpenAI-compatible and Anthropic-compatible options, in addition to vLLM and Ollama.

What is the difference between vLLM and the Semantic Router?

vLLM is the engine that loads and serves a model. The Semantic Router sits in front of one or more served models and decides which receives each request. You need both for this setup.

Should I configure routing in the dashboard or the YAML file?

Start in the dashboard to connect models and test with the playground. Move to the YAML file when you want to edit keywords, swap models or manage the config as text. Both describe the same setup.

Does the router replace my agent?

No. Your agent keeps the loop, the tools and the task state. The router only chooses which model handles each call.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.