What an Acceptance Check Costs in Command Time and Tokens

Command time is the wall clock from process start to exit for everything the check executes: test runner, build, linter, browser automation, container startup. Measure it at the boundary, not inside the agent, so retries and setup count. DORA's test automation guidance is a useful benchmark: developers should get automated feedback in under ten minutes locally and from CI (DORA's test automation capability). Anything longer than that inside an agent loop is a cost problem before it is a quality problem.

Tokens are the model calls needed to interpret the check and decide. A clean pass usually needs one small call: feed the exit code and a summary, get a classification. A failure needs more: the agent reads the error, reasons about it, edits, and rechecks. Token accounting should follow the provider's own usage fields, which include prompt and completion counts plus cached-token and reasoning-token details (Braintrust's token tracking guide). For portability across providers, the OpenTelemetry GenAI conventions define attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so you can compare engines without rewriting dashboards (Braintrust).

The trap is that aggregate totals hide the cause. Braintrust separates three views: per-call prompt and completion tokens, context window utilization per request, and per-step tokens inside agent traces. Prompt bloat, context overflow, and runaway agent loops each raise totals but need different fixes (Braintrust). For an acceptance check, the third view matters most: the expensive step usually sits deep in the loop, not at the top.

You do not need a full observability stack to start. Wrap the check in a small runner that records start time, exit code, output size, and the token usage of any model call made to interpret the result. Here is a minimal Python sketch that works with any OpenAI-compatible endpoint:

import time, subprocess, json

def run_check(cmd, interpret):
    t0 = time.monotonic()
    proc = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    elapsed = time.monotonic() - t0
    verdict, usage = interpret(proc.returncode, proc.stdout[-4000:], proc.stderr[-4000:])
    return {
        "command": cmd,
        "exit_code": proc.returncode,
        "command_seconds": round(elapsed, 2),
        "output_chars": len(proc.stdout) + len(proc.stderr),
        "verdict": verdict,
        "tokens_in": usage.get("prompt_tokens", 0),
        "tokens_out": usage.get("completion_tokens", 0),
        "tokens_cached": usage.get("cached_tokens", 0),
    }

Two implementation notes. First, truncate output before sending it to the model; a failing test suite can emit megabytes, and output tokens are the expensive kind. Second, include cached-token counts, because caching stable prefixes is the cheapest optimization available and cache-read tokens are roughly 90 percent cheaper than fresh input (Glean). Log each record as one JSON line, then aggregate per check name. That gives you the exact two numbers the budget needs.

If you would rather not build the routing and metering yourself, one option is Quanta Inference, which puts text, image, video, audio, and embedding models behind a single OpenAI-compatible key with prepaid, dollar-denominated pricing, so the token side of the ledger is one number to track.

Take a mid-sized feature: add discount stacking to a checkout flow. The ticket says a 20 percent loyalty discount stacks with promotional discounts to a maximum of 35 percent, which is exactly the kind of requirement that converts cleanly into acceptance tests (VirtusLab's maturity matrix). Assume four acceptance checks: unit tests for the pricing rules, an API contract test, a browser test for the checkout page, and a linter.

The numbers are illustrative but the method is not: every cell comes from a logged record like the one above, and the run counts come from your trace, not from a guess. Read the table the way a budget analyst would. The browser test consumes 56 percent of command time from three runs, so its cost is dominated by wall clock, and the fix is to run it once on the final diff rather than after every edit. The API contract test consumes 27 percent of tokens from four runs, so its cost is dominated by re-reading its output, and the fix is to assert on a compact JSON diff instead of dumping the whole response.

Now compare with the edit loop. Suppose the agent makes nine edits across the task at roughly 6,000 input and 800 output tokens each (context plus the edit), about 61,000 tokens. The check loop at 22,760 tokens spends roughly 37 percent of the edit loop's tokens, comfortably inside the rule of thumb that checks should not exceed edits. If instead the browser test ran after all nine edits and re-read full page HTML each time, its token line would roughly triple and the check loop would overtake the edit loop. That is the crossover point where you change the check, not the budget.

The acceptance-test literature supports investing here: in an industrial case study, 60 percent of LLM-generated acceptance test cases were usable as generated, 8 percent needed minor fixes, and 24 percent needed regeneration with more input (the arXiv case study). Translation for budgeting: expect roughly one in three checks to cost a second interpretation round, and size the per-task ceiling to absorb that without letting a degenerate check run unbounded.

A per-task ceiling is two numbers, not one: a command-time cap and a token cap. Set them from the task's edit loop, measured on your own traces. A practical procedure:

1. Run ten representative tasks with instrumentation on, and record edit-loop command time and tokens per task.
2. Set the check ceiling at 100 percent of the median edit-loop cost, with a hard stop at 150 percent.
3. Allocate the ceiling across checks in proportion to their historical share, but cap any single check at 40 percent of the total.
4. On breach, do not raise the ceiling. Shrink the check: reduce scope, compress output, or move it to a final-only gate.
5. Review breaches weekly; a check that hits its cap repeatedly is a design signal.

Step 4 is the decision rule that matters. Raising a ceiling hides a check that is too broad, and broad checks are exactly what agents over-run. Anthropic's documented workflow separates exploration and planning from implementation, which is the same principle applied to checks: run the narrow, fast checks constantly and the broad ones once (SSOJet on Anthropic's guidance). Spec-driven practice points the same way: a good spec covers just enough nuance to guide the agent without overwhelming it, and large tasks should be broken into smaller ones rather than crammed into one prompt (Addy Osmani's spec guide).

A compact policy table keeps the rule enforceable in code:

The last row deserves emphasis. A retry loop that produces the same error is burning both currencies for no information, and per-step trace visibility is the only reliable way to catch it, since the expensive span usually sits deep inside the run (Braintrust).

Three patterns account for most check-loop overspend. First, interpretation by a frontier model for trivial verdicts: a passing exit code needs no reasoning, so route classification to a small model. OpenAI's own agent guidance recommends building with the most capable model to establish a baseline, then swapping in smaller models where accuracy holds, optimizing cost and latency only after accuracy targets are met (OpenAI's practical guide to agents). Second, undersized output hygiene: piping raw logs into the prompt is prompt bloat, one of the three named drivers of token spikes (Braintrust). Third, no per-step attribution: without it you cannot tell a retrieval fault from a retry loop, and per-agent cost attribution is precisely what surfaces those faults (Mintive).

A fourth pattern is architectural. The OpenDev terminal-agent paper describes workload-specialized model routing and adaptive context compaction, both of which cut the cost of every step that re-reads the conversation, and both are deliberate engineering choices rather than defaults (the OpenDev paper). The decision rule that ties it together: treat the check loop as a fixed budget attached to the task, sized from measured edit-loop cost, and treat any breach as evidence about check design. Checks that pass cleanly on the first run cost almost nothing to interpret; checks that fail repeatedly are the expensive ones, and they are also the ones most likely to be too broad to be useful.

For teams that want the loop staffed rather than hand-tuned, Quanta Experts runs each expert against your context and returns finished deliverables, which pairs naturally with a check budget set at the task level rather than the tool level.