The Hidden Bloat in Your AI Tooling Stack

LLM inference cost optimization means cutting what you spend to run and serve model predictions without cutting the intelligence you actually use. In practice: cache repeated computation, batch requests, quantize models, route to the cheapest model that can do the job, and match GPU capacity to real demand instead of paying for idle silicon.

That definition sounds abstract until you look at what's actually installed on your machine right now.

Why is the ChatGPT desktop app 1.7GB of stuff you didn't ask for?

In September 2026, developer Simon Willison went digging through OpenAI's Codex/ChatGPT desktop app and found a folder called codex-primary-runtime sitting at 1.7GB.¹ Inside it: a full Python installation, a full Node.js installation, native binaries for Poppler and git, and 429.7MB dedicated to libreoffice-headless, a complete copy of the open source office suite.¹ None of this is disclosed anywhere in the app's interface.

The reaction on Hacker News was blunt. One commenter confirmed a 422.9MB LibreOfficeDev.app buried in their own install and called it "peak software bloat."² Nobody asked ChatGPT for a spreadsheet engine. It's there because bundling a working office suite is the fastest way to ship agentic file-editing features, and disk space is cheap until you're the one auditing it.

Why do 'free' AI tools carry hidden weight?

This isn't an isolated bug. It's a pattern. Vendors racing to ship agentic capabilities are bundling heavyweight runtimes and redundant dependencies directly into consumer apps rather than calling out to lightweight, purpose-built libraries.¹² The tradeoff is real:

• Disk footprint. Gigabytes of duplicate runtimes across every AI app you install, most doing the same job five different ways.
• Attack surface. Every bundled binary is code that has to be patched, and most users have no idea it's running.
• Update overhead. A full Node.js and Python stack shipped inside a chat app means two more update cycles your IT team never signed up to track.

None of this shows up on an invoice. It shows up on disks, in security audits, and eventually in an incident report nobody wanted to write.

What does your AI subscription stack actually cost?

The bloat inside the app is a rounding error next to the bloat in the subscription line. Roughly 50% of all enterprise software licenses go unused, and SaaS specifically sees 53% of applications sitting idle or underused, costing organizations around $21 million a year at scale.³ Companies wasted an average of $18 million on unused SaaS licenses in 2023 alone, a 7% jump from the year before.⁴ That waste pattern predates AI. AI just added a second, faster-moving layer on top of it.

Enterprise generative AI spend went from $11.5 billion in 2024 to $37 billion in 2025, a 3.2x jump in a single year.⁵ And the composition of that spend is shifting. Enterprise LLM API spend more than doubled in six months of 2025, from $3.5 billion to $8.4 billion.⁶ The money used to go mostly toward training models. Now 74% of startups and 49% of large enterprises say the majority of their AI compute is inference, running models in production, up from 48% and 29% a year earlier.⁶ The bill you're paying today is overwhelmingly a bill for serving tokens, not building models. That's exactly where the markups on frontier model APIs do the most damage, because you're paying that markup on every request, forever.

Shadow AI: the tools employees add that nobody audits

Layer shadow AI on top and the picture gets worse. Multiple surveys put unauthorized AI tool use at 59% to 78% of employees, meaning most organizations have subscriptions and inference spend running through tools IT never approved and can't see on a dashboard.⁷ That's not just a governance problem. It's untracked cost, compounding the same way unused SaaS seats compound, except nobody's finance team even knows the line item exists. We've written before about what's sitting on employees' desktops, and the exposure is bigger than most security reviews assume.

The core levers: quantization, batching, caching, and routing

Once you accept that inference is where the money goes, the fix is technical, not just financial. A handful of levers do most of the work:

• KV caching. Avoids recomputing attention tensors for tokens the model has already seen during decoding, cutting redundant compute on every generation step.⁸
• Continuous batching. Groups incoming requests dynamically instead of processing them one at a time, dramatically improving GPU throughput per dollar.
• Quantization. Runs models at lower numerical precision (8-bit or 4-bit instead of 16-bit), shrinking memory footprint and speeding up inference with minimal accuracy loss.
• Prompt and semantic caching. Stores and reuses output for repeated or near-duplicate prompts instead of paying for a fresh generation every time.
• Model routing. Sends easy queries to cheap, fast models and reserves expensive frontier models for the requests that actually need them.
• Speculative decoding. Uses a small draft model to propose tokens that a larger model verifies in parallel, cutting latency and compute per output token.⁸

Combined, these techniques can cut serving costs 3-4x or more without buying new hardware.⁹ Running Llama 3.1 70B on a single H100 costs $0.60 to $0.80 per million tokens unbatched, versus $0.15 to $0.25 with continuous batching.⁹ Same GPU. Same model. A fourth of the cost.

Case study: how prompt caching cut one company's LLM bill by 59-70%

ProjectDiscovery's agentic security system, Neo, is a clean real-world proof point. By restructuring prompts to maximize cache reuse, the team raised their cache hit rate from 7% to 84% and served 9.8 billion tokens from cache instead of regenerating them.¹⁰ The result: a 59% overall reduction in LLM costs, improving to 66% post-optimization and 70% in their most recent measured period.¹⁰ No new model. No new hardware. Just an architecture change that stopped paying for work already done.

Why do most GPUs sit idle, and what does that cost you?

Here's the number that should worry every finance team funding an AI roadmap: average GPU utilization across production Kubernetes inference fleets sits at just 5%.⁹ The best cluster in Cast AI's dataset, a 136-node H200 deployment, still only hit 49%.⁹ The typical enterprise is paying full price for GPUs that spend 95% of their time doing nothing. It's the SaaS license-waste problem wearing a different badge. Fairgen cut GPU infrastructure costs by 70% simply by deploying automated right-sizing and autoscaling, without touching their model or serving framework.⁹

If prices are falling, why do inference bills keep rising?

LLM inference prices have fallen 9x to 900x per year depending on the task, according to Epoch AI.¹¹ Yet enterprise bills keep climbing, because falling per-token prices don't translate into falling total spend. Instead of switching to cheaper, older models and pocketing the savings, most enterprises move en masse to the newest, priciest frontier model available, and usage growth outpaces the price drops.¹¹⁶ Cheaper tokens just mean more tokens get spent, on more expensive models, more often. That's why enterprise API spend more than doubled in six months even as unit prices fell.⁶

When does it make sense to self-host instead of paying API markup?

None of this argues against using frontier APIs when workloads are unpredictable or low-volume. It does argue for a threshold. Once you're running high-volume, predictable inference workloads, the break-even math on owning quantized models usually favors self-hosting over paying API markup indefinitely. The same logic that has companies collapsing five-database stacks into one Postgres instance applies here: rent while volume is uncertain, own once the volume is proven and the math is real. Platforms like Remy exist for exactly this transition, turning ad hoc inference spend into infrastructure you actually control.

A practical audit checklist for your AI tooling stack

Treat your AI tooling the way a sharp finance team treats any SaaS renewal: assume it's carrying more weight than it needs to, and prove otherwise.

1. Inventory every desktop AI app's actual footprint. Check disk usage per app, not just the download size. Look for bundled runtimes you didn't request.
2. Audit license utilization quarterly. Unused and underused SaaS seats already cost organizations tens of millions a year.³⁴ AI seats decay the same way.
3. Measure your cache hit rate. If it's near ProjectDiscovery's original 7%, you're leaving 50%+ of your bill on the table.¹⁰
4. Check GPU utilization, not just GPU count. If your fleet is anywhere near the 5% average, you're funding idle hardware, not intelligence.⁹
5. Map shadow AI usage. Survey employees directly. Assume the real number sits in the 59-78% range reported industry-wide.⁷
6. Route by task, not by habit. Reserve frontier models for the queries that need them and route everything else to cheaper models.

The fix for a bloated app and a bloated inference bill is the same discipline: stop assuming the vendor's defaults are lean, and check.