Batch Size and Parallelism: Latency and Cost

The numbers in this section are a worked example built on the token assumptions above. They are not vendor benchmarks. Re-measure with your own token counts and your own provider before committing.

Serial. One row per request, 1,000 requests, one after another. At a median of 4 seconds per round trip, the job runs about 4,000 seconds, roughly 67 minutes, before any retries.

Batched, 20 rows per call. Fifty requests. The shared instruction block is billed 50 times instead of 1,000. Per-request latency rises because the model reads and writes more, so assume 12 seconds with a wider spread: 50 calls back to back is about 10 minutes, less if a few are in flight.

Ten concurrent workers. The same 1,000 single-row requests with ten in flight. At 4 seconds median the theoretical floor is 1,000 divided by 10, times 4 seconds, about 400 seconds or 7 minutes. That floor holds only if the provider does not throttle you and retries stay rare. Providers meter usage per tier, and OpenAI's batch documentation states that each usage tier has an associated batch rate limit measured in enqueued input tokens the Batch API FAQ.

That table is the decision surface. Serial wins on traceability. Batching wins on token cost. Concurrency wins on wall clock but is the only pattern whose tail is set by someone else's limiter.

Two different techniques get called "batching" and they price differently.

Packing many rows into one synchronous request. The prompt asks the model to score 20 items and return a JSON array of 20 scores. Standard synchronous rates apply, but repeated instruction tokens vanish and request count drops twentyfold. The hard part is validation: a malformed array means all 20 rows need re-running together, so partial failure handling becomes the engineering.

Submitting to an asynchronous batch endpoint. OpenAI states that each model is offered at a 50 percent discount versus the synchronous APIs, that requests are processed within 24 hours, and that an expired batch cancels remaining work while completed work is still billed the Batch API FAQ. A batch can finish in minutes or take most of a day, because there is no minimum and no way to set the completion time within that window.

A worked example of the trade: 10,000 classification calls a day at 800 input and 100 output tokens each, priced at $0.75 per million input and $4.50 per million output tokens, costs about $10.50 a day synchronously and about half that through the batch endpoint. The same writeup estimates 8 to 12 hours of senior developer time to build the JSONL generation, polling, token tracking and partial failure recovery, putting break-even around 10 months at that volume the hidden costs analysis. At 1,000 rows the token saving is smaller still, which is why the call should be volume-based rather than dogmatic.

A second trap: community threads show users surprised when a batch bill does not match a hand calculation, usually because they assumed a per-request charge or mis-scoped the token counts the pricing confusion thread. Reconcile the provider's usage report against your own counter before extrapolating to a thousand rows.

Concurrency looks free until you measure the tail. Ten workers at a 4-second median gives a 400-second floor, but p95 is what a scheduler or a dashboard actually feels. Three forces push the tail out.

Rate limits. Providers meter requests and tokens per minute by tier. Cross a limit and you collect 429s, and a retry policy without backoff turns ten workers into ten workers plus a growing retry backlog.

Provider-side variance. Latency for the same prompt moves with load, so part of your p95 is theirs.

Retries on the slowest rows. The rows that fail are usually the long ones, and re-running them at the end stretches the final few minutes.

The practical fix is a bounded worker pool with a cap you derived from measurement. Keep the concurrency limit below the provider's request-per-minute allowance divided by your expected requests per second, and track enqueued tokens if you mix in batch submissions.

import asyncio, random

async def bounded(rows, limit=10, max_retries=3):
    sem = asyncio.Semaphore(limit)
    async def one(row):
        async with sem:
            for attempt in range(max_retries):
                try:
                    return await score(row)          # your API call
                except RateLimited:
                    await asyncio.sleep(2 ** attempt + random.random())
            raise RuntimeError(f"gave up on {row.id}")
    return await asyncio.gather(*(one(r) for r in rows))

The real decision is not "use concurrency." It is "set the limit from the p95 you can tolerate, then leave headroom for retries." A pool of ten that produces a 45-minute p95 through retry storms is worse than a pool of four that finishes in 25 minutes with a tight tail.

Parallel and batched execution change more than speed. They change what "the answer" means.

If the pipeline assigns row identifiers, maps results back by key, and retries only failures, order survives. If you stream completions, write to a shared queue, or read results from a batch endpoint, you need an explicit key. Every request in an OpenAI batch file carries a client-supplied custom_id for exactly this reason, alongside the method, URL path and full request body on each JSON line the hidden costs analysis. Treat that key as the contract and downstream joins survive any reordering.

Determinism is the other half. Scoring models are non-deterministic, and batched prompts can shift scores because the model sees neighbouring rows. If scores must be comparable across runs, pin the rubric, keep examples identical between patterns, and check agreement on a fixed sample. Chain-of-thought rationales, randomized ordering to counter position bias, and normalizing against a small human-labeled gold set are the calibration moves that make scores dependable this calibration guide. A cheaper bill with a biased judge is just a faster wrong answer.

Before scaling any pattern, run a regression check on a golden set with known acceptable outputs, exactly as you would for code. The workflow of building "before" and "after" datasets and testing semantic similarity, classification quality, length and tone is described step by step in this regression testing tutorial. Comparing model outputs against ground truth, or against each other when ground truth is missing, and tracking cost alongside accuracy is covered in this Label Studio walkthrough. A batching change that moves 3 percent of scores is a bug, not an optimization.

Run the three-pattern experiment on 1,000 representative rows, including the long and messy ones, then apply this rule in order.

1. Hard deadline under an hour? Use bounded concurrency at a limit you measured to be safe, plus batching of 20 if the prompt format allows it. Skip the asynchronous batch endpoint.
2. Can the job tolerate a multi-hour to 24-hour turnaround? Submit to the batch endpoint, take the discount, and reconcile the usage report. Build the retry and key-mapping layer first, because partial completion is normal and billable.
3. Small and infrequent, under a few thousand rows? Serial or batches of 20 are usually cheapest in total once engineering time is counted. The token saving will not repay the pipeline.
4. Large and recurring? Concurrency for the interactive slice, batch endpoint for the backlog, and one shared evaluation harness so both produce comparable scores.

Two numbers decide it: cost per thousand rows from the invoice, and p95 latency from your own timings. Track them per pattern, not per model, because the calling pattern moves both.

Pricing arithmetic is easy to get wrong under load, and providers change rates and modes. Verify every rate against the provider's current documentation rather than a blog post, including whether cache writes, long-context surcharges or tool calls apply, as illustrated in this pricing reference.

When the same job has to run across several model families, or you want one key and one prepaid balance instead of a separate integration per provider, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API (https://quantaintelligence.ai/inference). That removes per-provider plumbing from the experiment, leaving batch size and concurrency as the only variables.

You cannot compare patterns without identical measurements on each. Log per request: request ID, row count, input tokens, output tokens, start and end timestamps, HTTP status, retry count and model identifier. Aggregate into cost per thousand rows, median and p95 latency, retry rate, and the share of rows that landed in the wrong index or key.

Then check the aggregates against the provider's usage report. A gap between your counter and the billed usage is the most common source of budget surprise, and the fix is per-run reconciliation rather than a quarterly audit. The cost-aware selection work makes the same point: benchmarked quality without measured operational cost is an incomplete evaluation the cost-aware model selection study.

For scoring pipelines, add one more metric: agreement with a human-labeled gold set, computed identically for every pattern. A multidimensional rubric with per-question scoring and calibration against human judges is what turns raw model output into a signal you can act on, and the reported result was an RMS error below 0.5 on a 1 to 4 satisfaction scale, a twofold improvement over the uncalibrated baseline the calibrated rubric paper. None of that holds if the calling pattern quietly changes what the judge sees.

If you would rather have an expert run the measurement and hand back the cost and latency comparison for your own rows, Quanta Experts takes the brief and returns finished deliverables (https://quantaintelligence.ai/agents).