Blog
gpu passthrough kvmvfio gpu passthroughproxmox gpu passthrough

GPU Passthrough on KVM: The Setup Guide for Local AI Coding Agents

Step-by-step KVM/VFIO GPU passthrough guide: IOMMU, VFIO binding, libvirt/Proxmox config, and model serving for near-native local AI coding agents.

Portrait of Josh
Edited by Josh, Chief of Staff, GTM & Product··RSS

For Engineering and IT

A person is slotting a giant graphics card directly into a bare metal server motherboard on a home workbench.

How do you set up GPU passthrough on KVM for local AI coding agents?

GPU passthrough on KVM hands a physical GPU directly to a virtual machine, so the VM runs at 98 to 100 percent of native GPU performance instead of eating the usual virtualization tax. That's the technical foundation for running your own coding-agent infrastructure on hardware you own, with no cloud GPU bill and no code leaving the building.

Figure 1
How Close Passthrough Gets to Bare Metal
98%
Native GPU performance retained (low end)
100%
Native GPU performance retained (high end)
Illustrative figure summarizing the article's core performance claim for GPU passthrough versus bare metal; not tied to a specific benchmark source in the pool.
Illustrative figure. Constructed for explanation, not a measured source.

This is a build guide, not a home-lab pitch. If your team is spinning up cloud GPUs to run Ollama, vLLM, or a self-hosted Continue.dev backend for coding agents, passthrough is how you move that workload onto owned hardware without giving up the isolation a hypervisor gives you.

TL;DR

  • GPU passthrough gives a VM direct access to the card, so it keeps 98 to 100 percent of native GPU performance.
  • Clean passthrough depends on an isolated IOMMU group and on binding the GPU to vfio-pci before the host's native driver loads.
  • The VM should use the q35 machine type, OVMF (UEFI) firmware, and memory ballooning turned off.
  • At Q4_K_M quantization, plan on roughly 1.2GB of VRAM per billion parameters when choosing a card.
  • Owning a GPU beats renting once your team sustains about 4 to 6 hours of daily use over an 18 to 24 month horizon.
  • Unauthorized AI tool usage sits at 81 percent, so a fast, sanctioned local setup keeps code on your network and gives engineers an alternative.

Why Use Passthrough Instead of Bare Metal or Cloud Rental?

Three options exist for running a GPU-backed coding-agent stack: rent it, run it bare metal, or virtualize it with passthrough.

  • Renting means recurring cost, and for many popular coding tools, your source code transiting a vendor's infrastructure by default.
  • Bare metal gives you full GPU performance but ties the whole box to one workload. Want to also run a file server or a CI runner on the same machine? Now you're negotiating for GPU time.
  • Passthrough splits the difference. The hypervisor still runs other VMs and services, but the GPU VM gets near-native performance because the device is isolated at the hardware level via IOMMU, not shared through a software layer.
Figure 2
Rent, Bare Metal, or Passthrough: How the Three Approaches Compare
Rent, Bare Metal, or Passthrough: How the Three Approaches Compare
Cost ProfileGPU PerformanceCan Share Host With Other WorkloadsCode Stays On-NetworkSetup Complexity
Cloud Rentalbursty or occasional useRecurring hourly billHighYesNoLow
Bare Metalsingle dedicated GPU workloadHigh upfront, one workloadHighNoYesLow
RecommendedKVM Passthroughteams running agents most of the day on owned hardwareUpfront hardware, shared hostHighYesYesHigh
Ratings are relative across these options, not absolute. Based on the article's own framing of rent vs. bare metal vs. passthrough tradeoffs.
Illustrative figure. Constructed for explanation, not a measured source.

One Proxmox forum user summed up the actual motivation after setting this up on an RTX 5090: the goal was running AI agent pet projects "and have complete privacy on LLM interactions." That's the whole pitch in one sentence: near-bare-metal performance, plus a hypervisor you can still use for everything else.

What Do You Need Before You Start?

Passthrough has real prerequisites. Skip them and you'll burn an afternoon.

  • IOMMU-capable CPU and motherboard. Intel VT-d or AMD-Vi, usually present on hardware from the last decade but sometimes disabled in BIOS by default.
  • A GPU with an isolated IOMMU group. The GPU and its audio function need to sit in their own group, separate from your NIC or storage controller.
  • BIOS settings enabled. Above 4G Decoding and Resizable BAR, alongside IOMMU/virtualization support.
  • A second display path for the host. If you're passing through your only GPU, plan for headless host management via SSH, or use an integrated GPU for the host console.

How Do You Enable IOMMU and Check Your Groups?

Add intel_iommu=on (or amd_iommu=on) and iommu=pt to your kernel boot parameters, rebuild your bootloader config, and reboot. Then check the actual grouping with find /sys/kernel/iommu_groups -type l.

You're looking for a group that contains only the GPU and its audio device. If your GPU shares a group with unrelated devices like a network card or SATA controller, clean passthrough won't work without an ACS override, and that workaround carries its own stability tradeoffs. This is the single most common wall people hit, and it's as much a motherboard and PCIe topology issue as a software one.

How Do You Bind the GPU to VFIO Before Anything Else Loads?

Once IOMMU is confirmed, the GPU needs to be claimed by the vfio-pci driver before the host's native GPU driver ever touches it. The reliable way to do that is via the kernel command line and initramfs, not a runtime driver swap.

  1. Remove or blacklist the native driver (NVIDIA or AMDGPU) from loading at boot for that device.
  2. Load the VFIO modules (vfio, vfio_iommu_type1, vfio_pci) in initramfs.
  3. Bind the specific GPU by PCI ID using vfio-pci.ids= on the kernel command line.
  4. Verify with lspci -k that the kernel driver in use for the GPU is vfio-pci, not nvidia or amdgpu.

Doing this early, rather than swapping drivers after boot, is what stops the intermittent failures that show up in most passthrough troubleshooting threads.

How Do You Build the VM (libvirt or Proxmox)?

Whether you're using raw libvirt/QEMU or Proxmox's UI, the VM configuration follows the same pattern: q35 machine type, OVMF (UEFI) firmware instead of legacy BIOS, and memory ballooning turned off.

On Proxmox specifically, add the GPU as a PCI device to the VM with all four passthrough flags enabled: all functions, ROM-bar, primary GPU, and PCIe. On high-RAM hosts running multiple VMs, allocating static huge pages ahead of time avoids startup failures that show up as the GPU silently refusing to attach.

If you're running more than one GPU on the same host for multiple engineering teams, CPU pinning keeps each VM's vCPUs mapped to physical cores instead of floating, which matters more for consistency than raw throughput.

How Do You Install the Guest OS and GPU Drivers?

Boot the guest Linux install, then install GPU drivers exactly as you would on bare metal: NVIDIA proprietary drivers or the AMD ROCm stack, depending on your card. Run nvidia-smi or rocm-smi inside the guest. If the card shows full VRAM and correct model name, passthrough is working.

Two failure modes are worth knowing before you hit them:

  • Code 43 / driver refusal, common on older consumer NVIDIA cards that used to check for virtualization and refuse to initialize. Modern cards and drivers have mostly resolved this, but it still shows up on older hardware.
  • GPU reset bugs, where the card doesn't reset cleanly when a VM shuts down, forcing a host reboot to reclaim it. This is a known source of instability across hypervisors and worth testing before you put the setup in front of a team.

How Do You Deploy the Owned AI Coding Stack Inside the VM?

With drivers confirmed, install a model server, Ollama or vLLM, inside the guest and pull a coding-focused open-weight model sized to your VRAM. Point Continue.dev, or whatever IDE integration your team uses, at the local OpenAI-compatible endpoint instead of a cloud API. For the fuller picture of building a sandboxed, multi-developer version of this, the self-hosted agent factory guide covers the agent orchestration layer on top of what you build here.

Test actual usability, not just that it runs: sustained tokens per second for chat, and latency for autocomplete. A model that technically loads but returns completions after a three-second delay won't get adopted by your team, no matter how good the GPU utilization numbers look.

How Much GPU Do You Actually Need?

VRAM requirements scale predictably with model size and quantization. At Q4_K_M quantization, plan on roughly 1.2GB of VRAM per billion parameters.

  • 7B models (~5GB VRAM): fast autocomplete, lighter reasoning, fits comfortably on a mid-range consumer card.
  • 13B models (~9GB VRAM): a step up in coherence, still runs on common 16GB cards with headroom.
  • 32-34B models (~20-22GB VRAM): meaningfully stronger reasoning for chat and multi-file context, needs a 24GB-class card.
  • 70B models (~35-40GB VRAM): near the top of what a single consumer GPU can do, needs a 48GB+ card or the RTX PRO 6000 Blackwell's 96GB.
Figure 3
VRAM Needed by Model Size (Q4_K_M Quantization)
70B4032-34B2113B97B5
Illustrative figure. Constructed for explanation, not a measured source.

For a deeper look at matching specific compact models to specific GPU budgets, the Bonsai 2 and Qwen3.8 math walks through the tradeoffs.

When Does Owning the Hardware Actually Beat Renting?

Buying a GPU only pays off past a certain usage threshold. Most cost analyses put the crossover point around 4 to 6 hours of daily sustained use over an 18 to 24 month horizon, once you factor in electricity and depreciation.

Figure 4
The Break-Even Point for Buying a GPU
4 hrs
Minimum daily hours of use to break even
6 hrs
Typical daily hours of use to break even
Break-even window assumes an 18 to 24 month ownership horizon.
Illustrative figure. Constructed for explanation, not a measured source.

Below that threshold, renting wins outright. An A100 80GB runs about $1.09 an hour on services like Thunder Compute, while an RTX 5090 costs over $5,500 to own outright. The math shifts further if your VRAM needs exceed a single consumer card: the RTX PRO 6000 Blackwell, at 96GB, is effectively the ceiling for individual GPU ownership, and its price has climbed 87 percent to roughly $16,000 in 16 months amid a GDDR7 shortage.

If your team runs coding agents against a shared GPU most of the working day, five days a week, passthrough on owned hardware pays for itself. If usage is bursty or occasional, keep renting and revisit the math as usage grows. The broader case for when this tradeoff flips is laid out in the 2026 local-versus-cloud cost breakdown.

Figure 5
Owning vs. Renting GPU Capacity
$1.1/hr
A100 80GB rental rate
$5,500
RTX 5090 upfront cost
$16,000
RTX PRO 6000 Blackwell upfront cost
Illustrative figure. Constructed for explanation, not a measured source.

Why Does Keeping Code Inside Your Network Matter Beyond Cost?

Cost is only half the argument. Popular AI coding assistants route your code through third-party infrastructure by default. Cursor sends completion requests through its own AWS infrastructure with no HIPAA BAA or FedRAMP certification. Claude Code sends code context to Anthropic's servers on every query. GitHub Copilot routes through Microsoft Azure. For a 20-developer team running 50 queries a day each, that's roughly 1,000 codebase fragments leaving the network daily.

Figure 6
Daily Code Exposure From Cloud AI Coding Tools
1,000
Codebase fragments leaving the network daily (20 devs, 50 queries/day each)
Illustrative figure. Constructed for explanation, not a measured source.

That exposure compounds with shadow AI. Independent surveys put unauthorized AI tool usage among employees at 81 percent, with security leaders admitting similarly high awareness gaps. Banning tools outright just pushes usage underground. The better fix is giving engineers a sanctioned, local alternative that's actually fast enough to use. A passthrough GPU running Ollama or vLLM behind Continue.dev is that alternative, and it's the kind of infrastructure a platform like Remy is built to help teams inventory and govern once it's running, rather than leaving it as one more unmanaged box in the corner.

Figure 7
Shadow AI Usage Is Already Widespread
Security leaders admitting awareness gaps88%Employees using unauthorized AI tools81%
Illustrative figure. Constructed for explanation, not a measured source.

What Are the Common Pitfalls to Check When Troubleshooting?

Most failed passthrough setups trace back to a short list of causes:

  1. Bad IOMMU grouping. Your GPU shares a group with your NIC or storage controller, blocking clean isolation without an ACS override.
  2. Driver binding order. The native GPU driver claimed the device before VFIO could, usually from missing initramfs config rather than a bad blacklist.
  3. Reset bugs. The GPU doesn't reset cleanly between VM shutdown and restart, forcing a host reboot.
  4. Huge page misconfiguration. On high-RAM multi-VM hosts, missing static huge pages causes intermittent VM startup failures.
  5. Wrong machine type or firmware. Passthrough GPUs generally need q35 and OVMF/UEFI, not the older i440fx and SeaBIOS defaults.

None of these are exotic. They're the same five things that show up across most passthrough guides and forum threads, and working through them once, methodically, is the difference between a flaky setup and one your team trusts enough to use for daily coding work. Once it's stable, the economics and the privacy case both hold, and you've turned a recurring cloud bill into infrastructure you own outright.

Frequently Asked Questions

5 questions

Does GPU passthrough on KVM actually match bare-metal performance?

Yes. Peer-reviewed benchmarking across CUDA and OpenCL workloads shows KVM GPU passthrough hitting 98 to 100 percent of bare-metal performance on modern hardware, edging out Xen and VMware ESXi, which land at 96 to 99 percent.

What's the most common reason GPU passthrough setups fail?

Bad IOMMU grouping. If your GPU shares an IOMMU group with an unrelated device like a NIC or storage controller, passthrough can't cleanly isolate it, and you're stuck with an ACS override workaround that adds instability.

Do I need a second GPU to set up passthrough?

Not strictly, but it makes host management much easier. If you're passing through your only GPU, plan on managing the host headlessly over SSH, or use an integrated GPU for the console.

How much VRAM do I need to run a coding model locally?

At Q4_K_M quantization, budget about 1.2GB of VRAM per billion parameters. A 7B model needs roughly 5GB, a 32B model needs 20 to 22GB, and a 70B model needs 35 to 40GB.

When does buying a GPU for local AI actually beat renting cloud GPUs?

Most cost frameworks put the break-even point around 4 to 6 hours of daily sustained GPU use over an 18 to 24 month horizon, once electricity and depreciation are factored in. Below that usage level, renting is cheaper.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.