AI Model Size vs. Compute Effort: What Are You Actually Paying For?

What are you actually paying for: AI model size or compute effort?

Model size and compute effort are the two different things you pay for on every AI API call, and most invoices don't separate them. Model size is the number of parameters baked into a model during training, a fixed property that sets its ceiling on knowledge and skill. Compute effort is how much the model "thinks" at the moment you call it, a variable, per-request dial, often billed as reasoning tokens at output-token rates. You can hold size constant and watch your bill swing 6x just by turning effort up.

Most teams assume a bigger invoice means a bigger model. Often it just means the same model spent more tokens deliberating before it answered. Telling these two apart is the difference between cutting AI spend intelligently and cutting it blind.

What does AI model size actually mean?

A model's size is its parameter count: the weights learned during pretraining that encode everything it knows and can do. GPT-3's 175 billion parameters are the classic reference point. Once training ends, that number is locked. It doesn't change based on how you prompt the model or how hard a given question is.

Size isn't arbitrary. DeepMind's Chinchilla research found that for a fixed training compute budget, model size and training data should scale together, roughly doubling one means doubling the other to stay compute-optimal. That's why frontier labs don't just pile on parameters; they pair size with proportionally more training tokens. The result is a model's capability ceiling: the hardest class of problem it can reliably handle, regardless of how you query it later. Size is a training-time cost, paid once, before anyone sends an API request.

What does compute effort actually mean?

Compute effort, also called test-time compute, inference-time reasoning, or a "thinking budget," is what the model spends per call, after training is done. It's the number of reasoning tokens a model generates internally while working through a problem before it writes its final answer.

Every major lab now exposes this as a parameter you can set:

• OpenAI's reasoning models (o1, o3, o4-mini, the GPT-5 series) expose a reasoning.effort setting with levels like none, low, medium, high, and xhigh, trading thoroughness for tokens and latency.
• Anthropic's Claude models expose an effort parameter (low/medium/high/xhigh/max) and a legacy thinking-budget-in-tokens control; most Claude models default to high effort unless you dial it down.
• Google's Gemini 2.5 introduced an explicit "thinking budget," letting developers set how many tokens the model can spend deliberating, up to 24,576 tokens at launch.

Anthropic's own framing is useful here: Claude 3.7 Sonnet let users "toggle 'extended thinking mode' on or off, directing the model to think more deeply about trickier questions," with a budget to control precisely how long it spends on a problem. That's a dial you turn per request, on the exact same model, with no retraining involved.

Why can a smaller model with more effort beat a bigger model with none?

This is the part that actually changes how you should spend money. Research from UC Berkeley and Google DeepMind found that, FLOPs-matched, optimally scaling test-time compute on a smaller model can outperform a model roughly 14 times larger on math benchmarks. The same study found that a compute-optimal test-time strategy improved efficiency more than 4x compared to a brute-force best-of-N approach. Effort can substitute for size, up to a point. The hardest problems still reward more pretraining compute, not just more thinking time.

DeepSeek-R1 demonstrated a related idea from the training side: reasoning ability can be taught through reinforcement learning alone, producing a model that dynamically allocates more inference-time computation to harder problems on its own. OpenAI has described its own jump from o1 to o3 as roughly a 10x scale-up in reasoning compute, tracked as a distinct axis from the model's underlying training scale. Labs are explicitly managing size and effort as two separate levers, not one.

Why did your AI bill spike without changing models?

Reasoning tokens are generated by the model and then billed as output tokens, at the model's output rate. That's the mechanism behind some startling price swings on functionally the same request.

Google's Gemini 2.5 Flash made this unusually visible: output tokens cost $0.60 per million with thinking turned off, and $3.50 per million with thinking turned on, a roughly 6x difference, same model, same weights, different effort setting. Anthropic's effort parameter works the same way under the hood: it "affects all tokens in the response, including text responses, tool calls, and thinking," all billed as standard output tokens. OpenAI's reasoning_effort levels produce the same pattern on GPT-5-class models: more effort, more hidden reasoning tokens, more spend, without changing which model you called.

If your bill spiked and nobody changed the model field in the request, check the effort setting first. That's usually the real story.

Is mixture-of-experts a third variable hiding in "size"?

Just to complicate the picture, "size" itself isn't always one number. Mixture-of-experts (MoE) architectures split a model's parameter count into total parameters and active parameters. Mixtral 8x7B has 46.7 billion total parameters, its stored capability, but activates only about 12.9 billion of them per token during inference, which is its actual compute cost. A model can be large for capability purposes and comparatively cheap for per-token serving cost, because most of its weights sit idle on any given token.

This matters because dense models scale more predictably: inference speed for a dense transformer scales roughly with the inverse square root of its parameter count and the cube root of GPU memory bandwidth. MoE models break that clean relationship. When you're comparing prices across model families, you're not just comparing size and effort anymore. You're comparing total parameters, active parameters, and effort, three dials instead of two.

What is a practical framework for your AI budget?

Once you separate these levers, the optimization questions get a lot more concrete.

1. Match effort to task difficulty, not to habit. Reserve high reasoning effort for genuinely hard, high-stakes calls; classification, lookups, and routing rarely need it. If your workload has a mix of easy and hard requests, a per-request effort setting beats a single fixed one.
2. Pick model size for the task's ceiling, not its average. If the hardest 5% of your requests need frontier-level knowledge, size that case separately rather than paying frontier rates on every call. The piece on choosing an AI model without chasing leaderboard scores walks through how to match a model to your actual workload instead of a benchmark number.
3. Benchmark price-per-fixed-performance, not price-per-token. Epoch AI's tracking shows the price of hitting a fixed performance bar, like GPT-4-level scores on PhD-level science questions, has fallen about 40x per year on average, with some benchmarks falling as much as 900x per year. A model's sticker price per token tells you almost nothing about what it costs to hit the bar you actually need.
4. Know which dial you're paying for before you tune it. Dropping effort settings cuts reasoning-token spend immediately, on the same model, with no retraining and no migration. Dropping model size changes what the model can do at all. These are not the same decision, and they shouldn't be made with the same slider.

That distinction only pays off if someone on your team can actually set these dials per call instead of inheriting whatever defaults a vendor shipped. Teams building internal tools instead of renting another SaaS wrapper get that control by default: when you own the layer that calls the model, picking the model and the effort level per feature is part of the build, not an afterthought you negotiate with a vendor. That's one reason teams using a product agent like Remy to build and run their own internal software end up closer to the actual cost structure behind their AI calls, rather than paying a flat markup on top of it.

The parameter count was decided months ago, in a training run you'll never see a bill for. The effort setting is decided every time your code sends a request. Knowing which one moved is the first step to spending less without getting less.