What Is an LLM Proxy? How to Set One Up for Local AI Agents

What is an LLM proxy and how do you set one up for local AI agents?

An LLM proxy sits between your developers' AI agents and the models they call. It gives IT one place to see every prompt, every tool call, and every dollar spent, even when the agent runs entirely on a laptop. That single choke point is exactly what a company loses the moment engineers swap a SaaS coding assistant for a free local stack. Here's why that gap matters and how to close it without taking anyone's tools away.

Why are free local AI agents on your network a problem?

Somewhere on your company's laptops, a developer has pulled Ollama, wired it into Continue or Cline, and pointed a coding agent at a model running entirely on their own machine. No subscription, no procurement ticket, no IT review. It's fast, it's private by their own definition, and it costs nothing beyond electricity and disk space.

This is the same pattern we've covered in how employees build their own software stacks without telling anyone, and it's not a fringe behavior. UpGuard's global survey of security leaders and employees found that 81% of employees and 88% of security leaders admit to using unapproved AI tools themselves, and 45% of workers will actively find a workaround when a tool gets blocked. When developers had to use a SaaS AI product, IT at least had a vendor contract, a login, and usage logs. Local agents erase all three.

Why is "it's local, so it's safe" a myth?

The intuitive argument: a model running on a developer's own laptop can't leak anything, because nothing leaves the building. Wrong, on two counts.

First, local coding agents don't just generate text. They execute shell commands, read files, and make network calls with the full OS-level permissions of whoever is logged in. Tools like Claude Code, Gemini CLI, and Codex CLI store API tokens and session data in predictable, unencrypted config directories, the kind any other process on the same machine can read. Sysdig's Threat Research Team captured five iterations of a single agent's tool-use loop in just 10 seconds, generating 64 execve syscalls and multiple outbound HTTPS connections, all behind one visible chat response. That's how much invisible activity a local agent produces before anyone glances at a log.

Second, this isn't theoretical. MITRE ATLAS has documented a real attack in which a malicious MCP server triggered prompt injection in Cursor that ran a base64-encoded shell command to exfiltrate credentials straight out of a local config file. Microsoft's DART team separately found malware using the OpenAI Assistants API itself as a covert command-and-control channel. OWASP's 2025 Top 10 for LLM Applications now ranks "Excessive Agency" as a top risk precisely because agentic tools give models the ability to take real actions, not just answer questions. Running the model locally changes where the compute happens. It does nothing to change what the agent is allowed to do once it's running.

What do you lose without an LLM proxy?

Without a governed entry point, IT is blind in four specific ways:

1. Which models and tools are in use. You don't know if someone's agent is calling a 7B local model or routing through an unapproved hosted API behind the scenes.
2. What data leaves the device. Prompts, file contents, and tool outputs can travel to a provider's servers with zero record of what went out or when.
3. Spend and duplicate licensing. Teams quietly pay for the same capability three different ways: one dev on a paid subscription, another on a free local setup, a third expensing an API key nobody approved.
4. An audit trail for compliance. When an incident happens, or an auditor asks what model touched a given dataset, there's nothing to show them.

These blind spots compound. Freshworks research found 86% of IT leaders reported at least one negative incident tied to unapproved AI use in the past year, and nearly a quarter had more than three. A BlackFog study found that of the roughly half of workers using unapproved AI, 58% were on free-tier tools that never had enterprise-grade data governance in the first place. We've written before about what that pattern means for persistent AI compute running from the CLI; local agents just make the problem free to adopt and harder to spot.

What is an LLM proxy?

An LLM proxy, sometimes called an AI gateway, is middleware that sits between an application or agent and the model providers it talks to. Instead of a developer's tool calling OpenAI, Anthropic, or a local Ollama endpoint directly, it calls the proxy, which forwards the request, logs it, and returns the response. Providers from OpenAI and Gemini to Bedrock and OpenAI-compatible local setups like Ollama all get abstracted into one interface.

The level of proxy matters. A basic proxy just forwards requests and does light authentication or rate limiting. An enterprise-grade gateway goes further: it enforces policy, inspects prompt content, redacts secrets, applies model allowlists, sets per-team quotas, and produces audit logs a SIEM or compliance team can actually use. As one framing puts it, a basic proxy may pass traffic, but it usually can't answer which model processed which data, whether a tool call was allowed, or how retention was enforced. For governing local agents, build the gateway, not the forwarder.

Why use an LLM proxy for local AI agents?

Routing local agent traffic through a proxy, local model or hosted, gets you:

• Centralized visibility. Every call shows up in one log, regardless of whether the developer picked Cline, Continue, or something else entirely.
• Real cost and usage tracking. Per-developer and per-team spend becomes visible instead of buried in personal API bills or invisible because it's free and local.
• Consistent guardrails. Secrets detection, PII redaction, and model allowlists apply the same way no matter which agent or model a developer chose, closing the exact "excessive agency" gap OWASP flags.
• Policy without blocking productivity. You can say yes to local tools while still enforcing what data they're allowed to touch.
• One point for incident response. When something goes wrong, there's a single system to pull logs from instead of forty laptops.

The data backs the approach: Gartner's 2025 AI in Software Engineering Survey found 34% of top-performing organizations building AI-powered solutions use an AI gateway, compared to just 8% of lower performers. Gateway adoption correlates with outcomes, not just comfort.

How do you implement an LLM proxy?

1. Pick a proxy. LiteLLM is the most widely adopted open-source option. It exposes a single OpenAI-compatible endpoint across more than 100 providers, including local setups, and supports virtual keys, budgets, and detailed usage logs out of the box.
2. Stand it up centrally. Deploy it on infrastructure your team controls, not a developer's own machine, so it's one place everyone routes through.
3. Repoint local agents at it. Reconfigure Continue, Cline, and any Ollama-based setup to call the proxy's OpenAI-compatible endpoint instead of a provider directly. This usually requires no code changes, just a config URL swap.
4. Issue virtual keys per developer or team. This is what turns an anonymous stream of requests into an attributable one.
5. Turn on logging, budgets, rate limits, and guardrails. Set spend caps per key, enable secrets and PII detection, and make sure logs land somewhere your security team actually reviews.
6. Roll it out through config management or MDM. If developers can simply edit a config file back to a direct provider call, the proxy is optional. Push the endpoint setting centrally so it can't be quietly bypassed.

How do you govern local AI tools without banning them?

The instinct to block local AI tools outright is understandable, and it almost always backfires. Samsung's 2023 ban on ChatGPT and similar tools is the canonical example: it followed engineers pasting sensitive source code and leaking internal meeting transcripts into ChatGPT three times in about 20 days. Samsung's own internal survey found that 65% of participants already believed generative AI tools carried a security risk, and kept using them anyway. Awareness of risk didn't stop the behavior. A ban just removes the ability to see it. A proxy lets you say yes, but observed, which is the only version of that answer developers will actually stick with.

What operational risks should you plan for when self-hosting a proxy?

A proxy is new infrastructure, and infrastructure needs maintaining like anything else you'd rather own than rent carelessly. In March 2026, attackers compromised LiteLLM's build pipeline and published two malicious PyPI versions containing a three-stage payload that harvested SSH keys, cloud credentials, and Kubernetes secrets. PyPI quarantined the package within 46 minutes, but the incident is a reminder that whatever proxy you choose becomes a high-value target the moment it holds every API key in your organization. Plan for patch discipline, isolate the credentials the proxy itself holds, and decide deliberately whether a managed or self-hosted deployment fits your team's capacity to maintain it. This mirrors the same lesson we've laid out about why fences, not sandboxes, are the right model for local LLM risk: containment has to be real, not assumed.

Where does this fit into owning your stack?

A governed proxy is exactly the kind of software a team should own rather than quietly inherit. The choke point doesn't have to apply only to coding agents. Internal tools and other AI applications can sit behind the same gateway, logged and budgeted the same way, so the governance you build for local AI agents doesn't become a one-off project that only covers half of what your developers actually run.

Adoption of AI agents is already outpacing the ability to monitor them: 79% of organizations report agent adoption, but fewer than 10% have scaled agents in any single function, and 81% of respondents cite data security and privacy as an active concern. A proxy is the piece of infrastructure that closes that gap, not by slowing adoption down, but by making it something you can actually see.