Exfiltrate Your Weights: The Push for Mission-Critical Local AI

Running AI models locally means installing an open-weight model like Llama, Qwen, or GLM on infrastructure you own or lease outright, instead of routing every request through a vendor's API. For enterprises, the trade is control and cost predictability in exchange for the operational work of running your own inference stack. In 2026, a widening set of banks, hospitals, and governments are deciding that trade is worth making.

The API honeymoon is ending

For three years, the default enterprise AI architecture was simple: pick a frontier model from OpenAI, Anthropic, or Google, wire it up over an API, and let someone else worry about GPUs. That model still dominates spending. Anthropic, OpenAI, and Google together capture 88% of enterprise LLM API usage, and Anthropic alone now earns 40% of that spend, up from 24% a year earlier.¹

But a shadow question now hangs over that architecture: what happens when the vendor changes the rules, raises the price, or gets cut off entirely? In early 2026 that question stopped being hypothetical, and it's reshaping how technical leaders think about the model layer.

What does 'exfiltrating your weights' actually mean?

The phrase sounds dramatic. The mechanics are mundane. Open-weight models publish their parameters so anyone can download them and run inference on hardware they control, using serving engines like vLLM to handle the request queue, batching, and GPU scheduling that a commercial API normally hides from you. Once the weights sit on infrastructure you control, the model stops being a metered service and becomes a capital asset with a fixed cost.

That shift from renting to owning is the same logic driving software teams to self-host their AI stack instead of stacking API subscriptions. It's also why treating AI infrastructure as owned property, tracked and budgeted like any other capital asset, has become its own discipline. Some teams now manage that infrastructure the way they'd manage any owned software asset, with tools like Remy built specifically to track what a company owns instead of rents.

The case for local: sovereignty, cost, and compliance

Three motivations show up over and over in enterprise deployments. None of them are new, but they've gotten sharper.

• Data privacy and IP protection. Sending proprietary code, patient records, or financial data to a third-party API creates a permanent exposure surface. Local deployment keeps that data inside infrastructure the enterprise controls end to end.
• Cost predictability at volume. API pricing scales with usage in a way that can blow up a budget overnight. Owned infrastructure converts a variable cost into a fixed one, which finance teams tend to prefer once volume justifies the upfront spend.
• Vendor independence and compliance. Regulated industries need auditability a black-box API can't offer. Owning the stack means no single vendor's policy change can take down production.

Siemens built this case in practice. The company stood up its own GPU fleet, running NVIDIA L40S and H200 chips, and built an open-source serving stack around vLLM, Kong, and Prometheus/Grafana to run Llama, Qwen, DeepSeek, and Mistral for its internal developer base.² Siemens' own framing is blunt: AI sovereignty means owning the entire stack, and vendor independence comes from open weights on your own infrastructure, full stop.²

a16z's 2025 survey of 100 enterprise CIOs found the same pattern at scale: open-source model adoption concentrates at larger enterprises, driven by a preference for on-prem deployment tied to data security, compliance, and the ability to fine-tune models internally. Those same CIOs expect their AI budgets to grow roughly 75% over the next year.³

The Pentagon-Anthropic wake-up call

If one event turned vendor dependency from an abstract risk into a board-level one, it happened in March 2026. The Department of Defense applied its 'supply chain risk' designation to Anthropic, the first time that classification, normally reserved for foreign adversaries, was applied to a U.S. company.⁴ Defense contractors running Claude had no graceful migration path. Within hours of the ban becoming known, OpenAI signed a replacement deal with the Pentagon on terms Sam Altman himself called 'definitely rushed.'⁴

That episode is the clearest illustration yet of a risk that has nothing to do with model quality: when your production AI depends entirely on one proprietary vendor, a policy decision you had no say in can take your capability offline overnight. It's the same structural fragility covered in why owned infrastructure doesn't go down the way rented endpoints do, just applied to defense procurement instead of an outage.

When does running AI locally actually pay off?

The 'local always wins' story doesn't survive contact with the actual math. A Carnegie Mellon study of 54 deployment scenarios is the best evidence available on where the line falls.⁵

• Small models (24B-32B parameters) can break even against premium commercial APIs in as little as 0.3 months.⁵
• Medium models (70B-120B parameters) break even in a range of roughly 2.3 to 34 months, depending on usage volume and which API you're comparing against.⁵
• Large models (200B+ parameters) can take up to 69.3 months, nearly six years, to break even against a cost-competitive commercial API like Gemini 2.5 Pro.⁵

That spread matters. The sovereignty pitch is strongest at the small-to-medium end, where hardware and electricity costs are modest and inference volume is high enough to amortize them fast, a threshold laid out in more detail in the real break-even math on local models versus API cost. At the frontier end, where a single large model needs a rack of expensive GPUs to serve, the math often still favors renting, at least until usage climbs into the millions of tokens per day.

The broader data backs this up directionally. Enterprise AI inference performed on-premises or at the edge jumped from 12% in 2023 to 55% today, and on-premise deployments already hold 51.85% of the LLM market, led by banks, hospitals, and government agencies prioritizing sovereignty and latency control.⁶

Who's actually doing this

Beyond Siemens, the clearest proof points sit in regulated industries with the most to lose from a data leak.

Healthcare is a standout case. A Harvard Medical School study found that Meta's open-weight Llama 3.1 matched or outperformed GPT-4 on complex diagnostic cases, hitting 70% accuracy on established cases versus GPT-4's 64%.⁷ The study's authors put the implication plainly: institutions may be able to deploy high-performing custom models that run locally without sacrificing data privacy or flexibility.⁷ For a hospital bound by HIPAA, that's not a marginal improvement. It's the difference between using AI on patient data at all and not.

The cautionary tale that kicked off this whole shift is Samsung's. In 2023, Samsung banned employee use of ChatGPT and other generative AI tools company-wide after engineers leaked sensitive source code and internal meeting notes into the chatbot on three separate occasions within 20 days.⁸ That incident is now the textbook example of shadow AI turning into a governance crisis, and it's a big reason enterprises now prefer locking AI usage down to infrastructure they can audit end to end rather than trusting employees to self-police on external APIs.

Sovereignty's fine print: you don't fully escape dependency

Here's the part vendors selling 'sovereign AI' packages don't put on the slide: owning your weights doesn't mean owning your supply chain. Stanford HAI's review of commercial AI sovereignty offerings finds that even enterprises and governments running open-weight models locally remain dependent on Nvidia chips, TSMC manufacturing, and a narrow set of foreign cloud and hardware providers.⁹ In practice, sovereignty recalibrates dependency rather than eliminating it, a dynamic critics have started calling 'sovereignty washing.'⁹

Nvidia is, unsurprisingly, happy to sell into this. Its 'AI factory' sovereign infrastructure program, built around open-weight Nemotron models, has been deployed or supported in at least 25 countries, and sovereign AI already accounts for roughly $30 billion, or 14%, of Nvidia's revenue, a figure projected to reach $200 billion.⁹ The company that makes the chips underneath every 'independent' local deployment is also the biggest financial winner of the independence movement.

The enterprise verdict: hybrid, not all-in

Despite everything above, most enterprises haven't walked away from closed APIs. Menlo Ventures' late-2025 survey found enterprise open-source LLM market share actually fell from 19% to 11% year over year, even as sovereignty concerns grew louder.¹¹⁰ Closed models still command the large majority of a typical enterprise's roughly $7 million annual LLM budget, out of a total enterprise generative AI market worth $37 billion in 2025.¹¹⁰

The honest read isn't that local AI is winning. It's that enterprises are building two lanes at once, the same quiet split described in why the real shift in enterprise AI is happening off the cloud: a frontier-capability lane on rented APIs for the hardest reasoning tasks, and a growing local lane for the high-volume, compliance-heavy, or cost-sensitive workloads where owning the weights already pencils out. Boards that used to ask which API to standardize on are now asking a second question: which workloads should never depend on someone else's policy decision. That second question is what's pushing weights out of the cloud and onto infrastructure enterprises actually own.