How to Reduce LLM Token Usage With Local Indexing

How do you reduce LLM token usage with local indexing?

You reduce LLM token usage with local indexing by moving retrieval and tool selection out of the prompt and into a local store, then handing the model only the small, ranked slice it actually needs for a given query. Instead of re-sending the same code, docs, or tool definitions on every call, you query an index once at runtime and pass a few thousand tokens instead of tens of thousands. Done well, this cuts token spend by 40% to over 90%, depending on the workload.

Why does your LLM bill keep climbing?

Most teams don't notice where the tokens go. Every tool call re-sends the full tool schema. Every RAG lookup re-reads the same files. Every agent loop re-pastes context the model already saw three turns ago. None of this shows up as a single bad decision. It shows up as a slow accumulation of billable input tokens.

One production system that exposed all 70 of its tool definitions on every query was burning an average of 7,244 input tokens per query before anyone touched the retrieval logic. That's not a RAG problem or a prompt-engineering problem. It's a re-reading problem: the same tool metadata, over and over, priced per token.

The pricing structure makes this worse at scale. Flagship models typically charge $2-3 per million input tokens and $10-15 per million output tokens, a 4-5x output premium that punishes verbose context even before you factor in volume. Redis found that simply swapping a flagship model for a budget-tier model dropped a 1-million-conversation-per-month chatbot's bill from $3,250 to $195. Model tier matters. But what you send into the context window on every single call matters more, because it compounds with every tool, every file, and every query you add.

What does local indexing actually mean?

Local indexing is the practice of pre-processing your code, docs, or tool metadata once, offline, into a structure you control, then querying that structure at runtime instead of pasting raw material into the prompt. Three variants show up in practice:

• Vector databases. You chunk and embed your corpus once, store the vectors locally, and retrieve the top-k relevant chunks per query.
• AST or grep-based search indexes. You build a structural index of your codebase (symbols, call graphs, file paths) and let the model search it on demand, similar to how a human developer greps a repo.
• Semantic tool routers. You embed tool or MCP schemas once and retrieve only the handful relevant to the current task, instead of dumping the full tool catalog into every call.

All three share the same mechanic: pay the processing cost once, pay a tiny retrieval cost per query, and stop paying to re-read the same material at full price every time.

How do you index what you'd otherwise paste into every prompt?

If you're currently pasting files, docs, or conversation history straight into context, start here.

1. Chunk with structure in mind. Use AST-aware chunking for code (function and class boundaries) or semantic chunking for prose, rather than fixed-size splits that cut mid-thought.
2. Embed with a self-hosted model. Options like BGE, E5, or MiniLM run through sentence-transformers or Ollama produce embeddings without a per-token API bill. The self-hosting math favors this earlier than most teams assume: self-hosted embeddings beat OpenAI's text-embedding-3-large pricing once monthly volume crosses roughly 1.5 billion tokens, and beat text-embedding-3-small pricing at around 10 billion tokens. At 50 billion tokens a month, self-hosting runs about $199 total against the API equivalent, a 97% cost cut.
3. Store it locally. Chroma, FAISS, Qdrant, and LanceDB all run on your own infrastructure, with no vendor metering your query volume.
4. Retrieve a ranked slice at query time. The model sees the top handful of relevant chunks, not the whole corpus.

The payoff is measurable. One benchmark of real coding-assistant tasks found that a RAG-based local code index cut average token usage from 38,779 tokens per query to 7,427, an 80.8% reduction, which translated to an estimated monthly cost drop from $194 to $37 at 200 coding sessions a month. If you want the mechanics of building this without a heavyweight framework, a plain-Python RAG pipeline covers the chunk-embed-search-ask loop end to end.

How do you replace bloated tool definitions with semantic tool selection?

If your agent calls out to a dozen or more tools or MCP servers, the schema overhead alone may be your biggest line item.

1. Embed tool descriptions once, offline, the same way you embed documents.
2. Retrieve only the top-k relevant tools per query, instead of sending every schema on every call.
3. Add a gateway layer (something like search_tools, get_tool_schema, invoke_tool) so the model discovers tools on demand rather than having them all pre-loaded.

These are the most dramatic numbers in the dataset. The semantic tool-selection system mentioned earlier dropped average input tokens from 7,244 to 198 per query, a 91.5% reduction and roughly 49% cost reduction per query. At 10,000 daily queries, that's an estimated 24 billion tokens saved per year. A gateway approach that consolidates many exposed tool definitions behind a few meta-tools cut tool-definition tokens by 86.8% in one benchmark. Speakeasy's independently built "Dynamic Toolset" pattern, tested across toolsets of 40 to 400 tools, reduced total token usage by 96.4% on simple tasks and 90.7% on complex ones while holding a 100% task success rate, at the cost of roughly 2-3x more tool calls and about 50% more execution time.

That tradeoff, more calls for far fewer tokens, is the recurring pattern in this whole approach. Some agent platforms build this routing layer in by default rather than leaving it to each team to reinvent.

Why retrieve just-in-time instead of just-in-case?

Anthropic's own guidance on context engineering argues for loading data into context only when needed, keeping lightweight references like file paths and stored queries instead of pre-loading everything up front, because every added token depletes a finite "attention budget" and oversized contexts suffer measurable quality loss, what Anthropic calls context rot.

Anthropic tested RAG with a local vector database in early Claude Code and dropped it. Engineer Boris Cherny said agentic search, grepping and reading files on demand, "outperformed [RAG] by a lot," which the team found surprising enough to rebuild around filesystem search and sub-agents instead. That's not a universal verdict against vector indexing. Zilliz's Milvus-backed "Claude Context" MCP server, tested against grep-only search, cut token consumption by 39.4% on average and tool calls by 36.1%, with reductions as high as 93% on specific bugs. A separate 2026 study measured roughly a 26x per-query token-cost gap between reading full source documents and querying a pre-built index, and found that over-stuffed context can actively hurt answer quality: 48K well-chosen tokens outperformed 117K tokens of full context by 13 F1 points in a related test.

The takeaway: both agentic grep search and vector RAG beat dumping full context, and which one wins depends on your corpus. Codebases with strong structural search (grep, AST) often don't need embeddings at all. Large, unstructured document sets usually do.

How much does local indexing actually save?

Lined up together, the case studies point to a consistent range rather than a single magic number:

• Semantic tool selection vs. sending full tool catalogs: 91.5% token reduction.
• RAG-based code index vs. full-file reads: 80.8% token reduction.
• MCP gateway consolidation vs. direct tool exposure: 86.8% token reduction.
• Dynamic toolset routing vs. static toolsets: up to 96.4% total token reduction.
• Vector-indexed code retrieval vs. grep-only: 39.4% average token reduction.
• Full-document reads vs. a pre-built index: roughly 26x the per-query token cost.
• Semantic caching on repetitive workloads (a complementary technique, not retrieval indexing): up to 73% cost reduction.

If you're estimating your own savings, start from where your tokens actually go, tool schemas, repeated file reads, or repeated conversation context, and match the technique to the bottleneck rather than assuming one pattern fixes everything.

When doesn't local indexing help?

Indexing is engineering work with real infrastructure cost. It isn't free of tradeoffs.

• Small corpora don't justify the overhead. If your tool catalog is five tools or your docs are a single README, the build and maintenance cost of an index exceeds the tokens you'd save.
• Freshness becomes your problem. An index is a snapshot. Code and docs that change hourly need a re-indexing pipeline, or your retrieval starts returning stale answers.
• Semantic search misses renamed symbols and exact-match lookups. Vector similarity is bad at finding getUserByID right after someone renamed it to fetchUserById. Grep and AST search often win here precisely because they don't guess.
• One-off and real-time tasks don't benefit. A single ad hoc query, or a task that needs a live API result rather than historical data, is often cheaper and simpler to just send straight to the model or a live MCP call. LlamaIndex makes a related point about federated, non-indexed queries across live sources: without a unified index, you lose consistent relevance ranking and inherit the weakest connected source's search quality.
• Extra tool calls add latency. Every pattern above trades fewer tokens for more round trips. If your workload is latency-sensitive rather than cost-sensitive, that tradeoff can bite.

What does a reference build you can actually own look like?

For a developer who wants to stop renting tokens on their own tools and codebase, a minimal stack looks like this: a self-hosted embedding model (BGE or E5 through Ollama or sentence-transformers), a local vector store (Qdrant or LanceDB are reasonable defaults for solo and small-team use), and a thin tool-search layer in front of your MCP servers that exposes search_tools and invoke_tool instead of the full schema list. None of it requires a vendor subscription, and all of it runs on hardware you already have or can buy outright. For the broader case on replacing ad hoc AI tool sprawl with infrastructure you actually control, an AI-native workflow built on owned tooling walks through the same philosophy at the team level, and a rundown of current open-source AI infrastructure covers the specific tools worth running in production.

The pattern across every number in this piece is the same: you're not paying for intelligence, you're paying for re-reading. Local indexing is how you stop renting the same context twice.