How to Control LLM Costs in Production

You cannot cut what you cannot see. Most teams first see LLM cost as one line on a monthly invoice, which says nothing about which feature, team or customer produced it. NeuralTrust calls this the attribution problem and argues that it separates teams that can optimize from teams that are guessing (NeuralTrust, Token usage monitoring).

The minimum to log on every request, following NeuralTrust's list:

• Input and output tokens, separately. Output tokens cost more on most models, so a call that produces a long response is a different problem from one that sends a long prompt.
• The model. Without it you cannot compare efficiency or make routing decisions.
• Latency, at the median and the slow tail, per model and per feature.
• Cost, calculated at log time from token counts and the price in effect, so you never have to reconstruct it after prices change.
• Tags: feature name, a user or account identifier, environment and experiment ID.

{
  "ts": "2026-10-17T14:02:11Z",
  "feature": "support-reply",
  "account": "acct_8f2c",
  "env": "production",
  "model": "<model id from config>",
  "input_tokens": 3120,
  "cached_input_tokens": 2800,
  "output_tokens": 240,
  "latency_ms": 1840,
  "cost_usd": 0.0041,
  "task_id": "ticket_55193",
  "resolved": true
}

The most useful metric is cost per resolved task, not cost per call. BigData Boutique notes that an agent that solves a problem in one expensive call can be cheaper than one that grinds through ten cheap ones (BigData Boutique). Measure the unit the business cares about: cost per resolved ticket, per generated report or per active user.

Once tagged data exists, the first finding is usually concentration. BigData Boutique observes that a small slice of traffic tends to drive most of the bill: one chatty agent, one verbose endpoint or a batch job that re-embeds the same documents every night. Fix those first.

Model choice is usually the largest lever because prices between model tiers differ widely. BigData Boutique calls model selection "the highest-leverage decision in the whole stack" and names the default mistake as using a frontier model everywhere. Classification, extraction, routing, short answers and formatting rarely need the strongest model (BigData Boutique).

There are three patterns, in order of complexity:

1. Static routing. Each request type goes to a fixed model: a small model for classification, a mid-tier model for summaries and a frontier model for open-ended reasoning.
2. Dynamic routing. A router decides per request, sending only hard queries to the strong model.
3. Cascading. A cheap model answers first, its answer is scored for reliability and the request escalates only when confidence is low.

BigData Boutique advises starting with static routing, measuring the quality gap and adding cascades only where the savings justify the extra scoring step and the latency on escalation. Maxim's guide frames the same idea as matching each task to the cheapest capable model (Maxim, LLM cost optimization guide).

Routing is only safe with an evaluation set. Before moving a task to a smaller model, run that task's test cases against both models and confirm the cheaper one clears the quality bar. A cheaper wrong answer costs more than an expensive right one, especially if it produces a support ticket or a refund.

Keep model choices in configuration, not code, so you can change a route without a deploy and roll it back the same way.

Caching attacks the input side of the bill. There are three kinds, and they stack (BigData Boutique).

Provider prompt caching stores the processed state of a stable prompt prefix, such as the system prompt, tool definitions, examples or a long document, and bills reused tokens at a discount. BigData Boutique, citing Anthropic's documentation as read on 2026-10-17, describes cache reads at a tenth of the base input price, with a premium on the first write. The engineering rule is structural: put everything stable at the front of the prompt and everything that varies at the end, so the cacheable prefix is as long as possible. A timestamp or user name near the top of the prompt breaks the cache on every call.

Semantic response caching skips the model call entirely when an incoming question is close enough to one already answered. It works for repetitive traffic such as FAQs and support. Its risk is the false positive, two questions that look alike but need different answers, so BigData Boutique recommends tuning the similarity threshold conservatively and excluding anything where a near-miss answer is unacceptable.

KV caching applies when you host models yourself. Inference engines reuse attention computation across turns, and it is part of why a well-configured self-hosted endpoint reaches high throughput.

For agents, prompt caching matters most. The system prompt and tool definitions are re-sent on every step of the loop, so caching them reduces the cost of every call after the first.

Every token you do not send is a token you do not pay for. Start with the prompt:

• Trim the system prompt. Remove repeated rules, outdated instructions and long lists of edge cases.
• Retrieve precisely. BigData Boutique notes that better chunking, reranking and a tighter number of retrieved passages beat a bigger context window on both cost and accuracy (BigData Boutique).
• Compact agent history. Drop or shorten old tool outputs once the agent has used them. Morph recommends compacting context continuously rather than waiting until the context window is nearly full, because by then the agent has already paid full price for many bloated turns (Morph). Be careful with summaries: Morph points out that they can lose exact file paths, error codes and numbers, which the agent then spends tokens re-fetching.

Then control the output, which is usually priced higher than input:

• Always set a maximum output length. NeuralTrust's advice is blunt: set max_tokens, always. If a task extracts a small JSON object, it does not need thousands of output tokens. It suggests auditing average output length per feature and setting limits somewhat above it (NeuralTrust, LLM cost reduction).
• Ask for structured output so the model returns data instead of prose.
• Use stop sequences and concise instructions such as "answer in one sentence," which BigData Boutique describes as a cost control, not only a style note.

Not every request needs an answer in two seconds. Major providers offer asynchronous batch interfaces for work that can wait. BigData Boutique, read on 2026-10-17, describes the OpenAI and Anthropic batch offerings as processing requests within a 24-hour window at half the standard token price, and notes that the batch discount stacks with prompt caching (BigData Boutique).

NeuralTrust's guide sorts workloads by one question: is a person waiting for this answer right now? (NeuralTrust, LLM batching)

Batching needs a little extra engineering. NeuralTrust lists the things that break: you need per-request status checks, retry logic for failed items and downstream processing that is safe to run twice. Most production systems end up running both modes side by side, synchronous for live users and batch for the background layer.

Optimization decays without guardrails. A single bug in an agent loop can spend a month's budget overnight. BigData Boutique recommends per-key and per-team budgets, rate limits and alerts on cost spikes "so a runaway agent loop trips an alarm instead of an invoice," and attributing spend back to the teams that write the prompts (BigData Boutique).

Limits work best in layers:

• Per call: a maximum output length and a timeout.
• Per task: a maximum number of agent steps, tool calls and total tokens, after which the agent stops and reports what it has.
• Per user or account: daily or monthly token budgets. NeuralTrust notes that a small group of power users, scripts or test suites often drives a large share of consumption, and recommends per-user budgets with alerts (NeuralTrust, LLM cost reduction).
• Per team or key: budgets that block requests once spent. Maxim describes hierarchical budgets where a team and each key within it carry separate caps, so an exhausted budget at any level stops the request in real time (Maxim).
• Per environment: separate keys and lower limits for development and staging.

Prefer limits that stop spending over reports that describe it afterward. A gateway that bills from a prepaid balance, such as Quanta Inference, turns the budget into a hard ceiling rather than an invoice line.

A sensible order of work, from cheap and high-return to expensive:

1. Log and tag every call, then find the few features driving most spend.
2. Cap output length and trim system prompts.
3. Reorder prompts for caching and turn on provider caching.
4. Route simple tasks to smaller models, verified by evaluations.
5. Move background jobs to batch.
6. Add budgets, rate limits and agent step limits.
7. Consider semantic caching, cascades or self-hosting only once volume justifies the complexity.