What an Intent Classification Pipeline Costs per Thousand Queries

Every API call bills two streams: input tokens (system prompt, any few-shot examples, the user query) and output tokens (the label and any surrounding syntax). The price per million tokens differs by model and by direction, and cached input is cheaper than fresh input. Anthropic publishes a pricing table that shows base input, output, cache writes and cache reads as separate line items in its prompt caching docs, which is the cleanest structure to model.

Two properties of tokenization matter for forecasting. First, tokens are not words: short function words pack cheaply, and product codes or SKUs tokenize poorly. Second, because input dominates a classifier, the system prompt is the budget. A 400-token prompt sent a million times is 400 million input tokens, before the queries.

Here is the baseline assumption set used throughout this article. Keep the structure and swap the numbers for your own.

That yields 420,000 input tokens and 5,000 output tokens per thousand queries. Add any per-call overhead your SDK sends (role markers, JSON scaffolding) if your provider counts it, and round up rather than down.

The arithmetic is the whole method:

input_tokens  = queries * (system_tokens + query_tokens)
output_tokens = queries * label_tokens
cost = (input_tokens / 1,000,000) * price_in
     + (output_tokens / 1,000,000) * price_out

For 1,000 queries: (420,000 / 1,000,000) times the input price, plus (5,000 / 1,000,000) times the output price. The output side is small but not ignorable when output is priced many times higher than input, which is common. Using relative multipliers keeps the model portable across providers, and Anthropic's table makes the spread concrete: its Haiku-tier model lists input at $0.10 per million tokens and output at $0.50 per million for prompts under 100,000 tokens, a 5x output-to-input ratio from the caching docs. Plug in whatever your provider quotes and the shape holds.

A worked number with a made-up round price of $1 per million input tokens and $5 per million output tokens:

• Input: 420,000 divided by 1,000,000, times $1.00, equals $0.42
• Output: 5,000 divided by 1,000,000, times $5.00, equals $0.025
• Total per thousand: $0.445

That is the number to beat. Everything below is a discount applied to it, and the discounts do not add the way people assume.

Caching stores the processed prefix of a prompt so the provider can skip recomputing it on the next request with the same beginning. Anthropic's docs explain the mechanism: the system checks for a cached prefix up to a breakpoint, reuses it if found, and otherwise processes and caches the full prompt. Cached reads cost far less than base input, and in the published table cache hits and refreshes sit at a small fraction of base input per the same docs.

The catch is the cache lifetime. Anthropic's default cache lives five minutes and is refreshed on each use, and the clock starts at the beginning of the request, not when the response finishes per the caching docs. That single detail decides whether caching helps you: a steady query stream keeps the cache warm, while a bursty one pays the write price and then lets the entry expire.

What does it save? With your 400-token prompt cached, each call after the first pays the cache-read price for those tokens instead of the base input price. If the cached rate is a tenth of base input, the 400 tokens drop from 0.4 of a cent-equivalent unit to 0.04, and the per-thousand input cost falls from the baseline to roughly the query-only portion plus a tenth of the system portion. The first call in each cache window is a write, priced above base input, so short windows with sparse traffic can lose money.

Caching is not free money in every architecture. A production team measured an 85.8% cache-hit rate that cut input-token costs by 77%, but also found that a poorly ordered prompt with caching enabled hit only about 20% reported by Deriv. The lesson is about prefix stability: put every fixed instruction before every variable one, and never interpolate a timestamp or user ID into the front of the system prompt. A broader breakdown of provider caching mechanics and where the savings actually land is worth reading before you build per Flexera's analysis, and if you want a plainer definition of the cost unit, Infracost defines cached token pricing in its glossary.

Batch endpoints group requests into an asynchronous job and return results later, in exchange for a discount. OpenAI's Batch API and Anthropic's Message Batches API both offer 50% off versus synchronous calls, with results delivered within 24 hours, and batch rate limits use a separate pool from synchronous quota per NeuralTrust's guide. The same source notes Anthropic completes batches in under an hour typically, and that the batch discount stacks with prompt caching on Anthropic.

Apply 50% to the baseline and the per-thousand figure halves to about $0.22. Stack caching and it drops further, because caching shrinks the input base that the batch discount then applies to. In that order: cache first to remove repeated work, then batch to discount what remains. Doing it the other way is not wrong arithmetically, but it hides how much of the bill caching already removed.

Batching only fits some workloads. It suits classification, document processing, labeling and embedding generation, and it does not suit live chat or any flow where a human waits for the answer per NeuralTrust. For intent classification, the right split is usually easy to state: real-time routing of a single incoming query goes synchronous, and backfilling historical queries, re-labeling a dataset or scoring a nightly sitemap crawl goes to batch. If your labels feed a live search box, batch is a maintenance job, not the hot path.

The table below applies both levers in sequence to the baseline, using illustrative prices and the assumed discount rates above. Your provider's numbers will differ; the ordering and the magnitude of each step will not.

The last row is the one to carry into planning. A richer prompt with more examples can improve label quality, and it multiplies input cost directly if you are not caching. The decision rule: expand the prompt only after caching is in place, and measure the accuracy gain against the token bill rather than against a sample.

If you run classification across several model providers, a single gateway with one key and prepaid, dollar-denominated billing removes a chunk of the forecasting work, which is what Quanta Inference is set up to do. Not every team needs that, and none should add a dependency before the baseline math is done.

Forecasting is mostly discipline. Run through this before you commit to a number in a plan.

1. Capture the real token counts, not estimates. Log input and output tokens per request from the API response for a week.
2. Measure the query length distribution, not just the mean. The 95th percentile token count, not the median, sets your worst case.
3. Freeze the system prompt and version it. Any edit invalidates the cache and changes the baseline.
4. Order the prompt fixed-first. All static instructions, then examples, then the query.
5. Check your provider's cache lifetime against your traffic pattern in requests per minute.
6. Split the workload into synchronous and asynchronous lanes and price each separately.
7. Add a buffer for retries and validation failures, then reconcile against the invoice monthly.

A threshold on confidence is part of this. Confidence thresholds act as a decision boundary that routes high-confidence predictions to automated processing and low-confidence ones to human review defined by LlamaIndex. Each low-confidence query that gets a second model call is extra tokens, so set the threshold against the cost of a wrong label, not by default.

If the forecast work itself is the bottleneck, building the pipeline with an expert who has shipped classification systems is usually faster than learning the accounting from scratch; a consulting call is where that conversation starts. For teams that want the whole thing built for them, Quanta takes a description and turns it into a working system.