How to Reduce LLM API Costs in Development

Stop sending every request to your most expensive model. Route simple, high-frequency tasks like file reads, boilerplate generation, and formatting to cheap models, reserve the frontier model for genuine reasoning, and layer prompt caching on top. Teams doing this report cost cuts of 40% to 90%, with no new seat license and no platform rebuild.¹²³

That's the short answer. The longer answer is why this became urgent, and how real engineering teams are doing it.

Why is AI coding suddenly the biggest line item in the budget?

Gartner predicts that by 2028, AI coding costs will surpass the average developer's salary, driven by rising token consumption and a shift from seat-based to consumption-based pricing.⁴ That's not hypothetical. Interviews across 15 companies found token spend rising roughly 10x in six months at multiple large organizations, with individual developers burning $500 to $1,400 a day on Claude Code.⁵ Spotify's own engineering team found a quarter of engineering leaders already spend $200 to $500 per developer per month on tokens, with some organizations well past $2,000.¹

This is a new kind of budget line. It doesn't look like a SaaS renewal you can cancel. It scales with usage, and usage scales with how good the tools get, so the bill keeps climbing even when your team's size doesn't change. We covered the mechanics of this shift in The Race to the Bottom: unit prices for tokens keep falling, but total spend keeps rising because usage grows faster than price drops.

Why per-seat pricing is already dead

GitHub's own numbers make the case. The company moved Copilot from a premium-request model to full usage-based billing, effective June 1, 2026, because a quick chat question and a multi-hour autonomous coding session were burning wildly different amounts of compute for the same nominal price.⁶ GitHub had been absorbing the difference. It stopped being sustainable.

That single change signals where the whole market is headed. Agentic coding sessions, the kind that read a codebase, plan, write, and iterate, consume far more tokens than a one-off chat answer. Vendors can't flatten that into a $19-a-month seat fee anymore. The cost lands on whoever runs the agent, which puts the incentive to control it on engineering teams, not vendors.

What an orchestration layer actually does

An orchestration layer, sometimes called an AI gateway or LLM router, sits between your developers and the model APIs. Instead of every request going straight to Claude or GPT-5, it inspects the task and decides which model should handle it.

The core techniques are simple:

• Model tiering. Send routine, high-volume tasks to a cheap, fast model. Reserve the expensive model for tasks that actually require deep reasoning.
• Prompt caching. Store repeated prompt prefixes so the model doesn't reprocess the same context on every call.
• Context compression. Limit how much of a codebase or document gets read into a request, instead of dumping entire files.
• Batch processing. Group similar requests instead of firing them one at a time.
• Centralized routing infrastructure. A gateway that enforces these rules automatically, rather than relying on every developer to remember to use the cheap model.

The research behind this is consistent: 60 to 80 percent of typical LLM requests in production don't need frontier-level capability, and routing them to smaller models cuts costs by 40 to 70 percent with no measurable quality drop.² Anthropic's own prompt caching can cut input costs by up to 90% and latency by up to 85% on long, repeated prompts, and it takes almost no engineering effort to turn on.³

Case study: Spotify's shunt plugin

Spotify's Portal, built on their internal developer platform Backstage, ships a Claude Code plugin called shunt with declarative rules called AiKA Modes.¹ The idea is narrow: bulk file reads and boilerplate code generation get delegated to a cheaper worker model, Gemini 2.5 Flash, while Claude keeps doing the actual reasoning.

Tested against a Java monorepo across four scenarios, the mean savings on bulk-read tasks came out to roughly 90%.¹ The mechanism is straightforward: reading through a large file to find a class definition doesn't require frontier reasoning. It requires pattern matching and retrieval. A cheap model does that job just as well, then hands the relevant snippet back to Claude to reason over.

This is the version of the story that matters for smaller teams: it's a plugin, not a platform. No new infrastructure team, no vendor contract. Just a routing rule bolted onto an existing tool.

Case study: Kong AI Gateway at platform scale

Spotify also runs a second, heavier layer underneath all of this. In May 2025 they deployed Kong's AI Gateway to unify LLM integrations that had been fragmented across hundreds of internal teams, each solving authentication, rate limiting, cost tracking, and failover independently.⁷ More than 1,000 internal services have since onboarded, and integration time for a new team dropped from weeks to an afternoon config change.⁷ The gateway gives Spotify unified, token-level telemetry and routes across multiple providers, including OpenAI, Vertex AI, and first-party models.⁷

The payoff shows up in adoption numbers. More than 99% of Spotify's engineers use AI coding tools weekly, 94% report increased productivity, and pull request frequency is up 76%, much of it driven by their internal background coding agent.⁸ None of that scale is sustainable if every request defaults to the most expensive model available.

Five levers teams are actually using

1. Model tiering and routing. Classify tasks by complexity and route accordingly. This is the single biggest lever, delivering 40 to 70 percent savings on its own.²
2. Prompt caching. Turn on caching for any prompt with a repeated prefix, especially large system prompts or codebase context. Up to 90% cheaper on the cached portion.³
3. Context limiting. Don't let an agent read an entire repository when it needs three files. Constrain the search before the model ever sees the tokens.
4. Batch processing. Group non-urgent requests, like nightly test generation or documentation updates, instead of running them one at a time in real time.
5. Self-hosted gateways. Tools like LiteLLM, an open-source proxy with over 56,000 GitHub stars, add multi-provider routing, budgets, and fallback without markup and without building anything from scratch.

Ramp's experience backs this up outside the coding-tool context: their internally built router processes more than 2.75 trillion tokens a month and cut internal LLM costs by 30% with no drop in performance.⁹ These gains aren't theoretical. They show up in production billing, which is the same pattern we've traced in The Hidden Costs of AI Agents: the markup on frontier API calls is real, and routing is how you claw it back.

Do you need a platform team, or just a plugin?

Most teams don't need to rebuild what Spotify built. Spotify's shunt plugin proves the low-cost end works: a declarative routing rule on top of an existing tool, no new headcount. The Kong gateway is the other end, a genuine platform investment that makes sense once you have hundreds of teams and thousands of services generating fragmented, ungoverned LLM traffic.

Most engineering orgs sit somewhere in between. A handful of teams and a handful of AI tools means start with a lightweight router like LiteLLM or a plugin-level rule set. Token spend fragmented across dozens of teams with no shared visibility means you've outgrown the plugin and need the gateway. It's the same build-versus-buy calculus we laid out in AI Software Factory vs. SaaS: the right answer scales with your actual usage, not your ambitions.

Routing as a check on shadow AI spend

There's a governance dividend here too. More than 80% of workers, including nearly 90% of security professionals, already use unapproved AI tools at work.¹⁰ Without a routing layer, that shadow usage is invisible until the invoice arrives. A gateway sitting between developers and model providers gives you token-level telemetry by default: who is calling what, how often, and at what cost. That's not a side benefit. It's the difference between discovering a runaway bill after the fact and catching an overrun in real time.

A practical checklist to start

1. Measure token spend per task type. You can't route intelligently until you know which tasks burn the most tokens.
2. Tier your models. Pick a cheap, fast model for retrieval and boilerplate, and keep the frontier model for planning and complex reasoning.
3. Turn on prompt caching wherever prompts share a long, repeated prefix.
4. Delegate I/O-heavy work (bulk reads, file scans, formatting) to the cheap tier by default, not as an afterthought.
5. Set budgets and alerts at the gateway level, not just the vendor billing dashboard, so overruns get caught inside the sprint instead of at month end.

None of this requires a platform rebuild. It requires treating token spend the way you'd treat any other infrastructure cost: measured, tiered, and governed, not just paid.

For teams weighing whether to own more of this stack outright rather than keep routing around rented tools indefinitely, a platform like Remy is built around that same principle: keep the parts of your software you actually control, and stop paying rent on the parts you don't.