Billion Context Benchmarks: Does Compression Cut Token Costs?

Does Billion Context actually cut token costs?

Yes, according to the reported benchmarks. On Context Bench, a coding-agent test over real repos, Billion Context scored F1 0.31 against 0.29 for full context and cost $4.61 instead of $11.05, about 58% less. On LongMemEval it scored 79.2% versus 83.2% for full context. The cost is exact recall: on a needle-in-a-haystack test, accuracy fell from 69% to 27%.

What is Billion Context and how does it work?

Billion Context is a small proxy that sits between a coding agent and the model API. You point the agent's base URL at the proxy, and every request passes through it. The proxy reads the conversation, compresses the parts that are finished, and forwards the result to the real model. It supports Anthropic-style and OpenAI-style APIs, so it can sit in front of most agents.

The unusual part is who decides what to compress. The proxy gives the model four tools:

So the model manages its own memory. It compresses finished work, searches when it needs something back, and decompresses when a summary is not enough.

How is this different from built-in compaction?

Most agents handle a full window by compacting everything into one summary, or they fall over. Billion Context differs in three ways, according to the project:

1. Incremental. It summarizes small ranges, not the whole session at once.
2. Reversible. Compressed ranges can be decompressed, so detail is not permanently lost.
3. Cache-friendly. The approach is designed to keep the prompt cache intact.

That third point drives the economics. Every turn of a long agent session resends the history, so you pay for the same tokens repeatedly. Cached input tokens are far cheaper than fresh ones. The project says a healthy session sits at a 95 to 97% cache hit rate, with compression itself costing 2% or less. A summarizer that rewrites the start of the prompt breaks the cache and gives that saving back. Compressing finished ranges in place is meant to avoid that.

What do the benchmark numbers show?

The figures below come from a third-party benchmark run on DeepSeek V4.1 Flash, as reported in the coverage of the project.

Some context on each row:

• Context Bench tests a coding agent exploring real repositories. Billion Context ranked first. A 0.02 F1 edge is small, so read it as "no loss in quality" rather than a clear win. The 58% cost reduction is the stronger result.
• LongMemEval tests memory over long conversations. Full context still wins by four points. The project reports 71% lower cost on this test, though the source does not say which baseline that figure is measured against.
• The 1.48M-token test is less a quality comparison than a survival test. That input exceeds what a normal window can hold, so full context failed every time. Billion Context stayed alive with about 30x compression.

The author also published a paper in the repo covering 4.5 months of real use: 174,000 model calls, 18.76 billion input tokens, and zero context window overflows on 200k models. That is self-reported usage data, not an independent benchmark, but it speaks to long-run stability.

What does compression cost in exact recall?

This is the trade you should weigh most. On a needle-in-a-haystack test, where the model must return one exact sentence buried in a long context, accuracy fell from 69% to 27%.

That result is consistent with how the system works. A summary keeps the meaning of earlier work (what was tried, what changed, what was decided). It does not keep every literal string. If the model needs a verbatim error message, a specific line of config, or an exact quote from 300 turns ago, it has to notice that the summary is insufficient and call decompress or search. When it does not notice, it answers from the summary and may be wrong.

For many coding sessions this is acceptable. Agents mostly need to know what has been done and why, and they can re-read files from disk. For workloads where exact wording matters, such as legal text, audit trails, or quoting source documents, a 42-point drop on this kind of test is a serious warning.

Does it beat RAG?

On LongMemEval, yes by the reported numbers: 79.2% against 62.6% for the best RAG setup. The mechanisms differ. RAG retrieves chunks that look relevant to the current query and can miss context that matters but does not match the query well. Billion Context keeps a running, compressed account of the whole session and lets the model pull specific ranges back. That fits agent work, where the history is one continuous thread. It is a weaker argument for large static document collections, which the benchmarks here do not cover.

What are the practical limits and risks?

The project is young and moving quickly. Points worth knowing before you adopt it:

• Fast-moving versions. It was at version 0.121 in the coverage, updates itself by default, and the master branch can break at any time. Installing a tagged release from npm is the safer path.
• No authentication. If you expose the proxy on your network with the host flag, anyone who can reach it can use it. Only do that on a trusted network.
• Some clients bypass it. One desktop app's code tab skips the proxy entirely, so you get no compression there.
• Dense docs. The documentation is large and technical, and setup takes some reading.
• Recall drift. As covered above, verbatim lookups depend on the model choosing to decompress.

Setup is a global npm install that provides a billy command. It can install as a native plugin for some clients, launch a client through the proxy, or run as a standalone proxy whose address you put in front of your base URL. The proxy serves a local dashboard with sessions, live logs, and a config editor. In supported clients, a /acp cache command reports your cache hit rate, tokens saved per compression, and what broke the cache if the rate dropped. That report is the quickest way to check whether you are really saving money.

Is Billion Context worth trying?

It is worth a test if your agent sessions run long, resend large histories, and mostly need the gist of earlier work. The reported numbers suggest you can cut cost by more than half without hurting coding quality. Check your own cache hit rate before trusting the headline savings, since they depend on the cache staying warm. Skip it, or test hard, if your tasks depend on recalling exact text from deep in the past.