What a Test List Costs in Tokens and Minutes

The token formula is boring, which is good, because boring formulas get used.

tokens per run ≈ (list tokens + diff tokens + instruction tokens) × passes
run cost ≈ (input tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price)

State the assumptions, then do the arithmetic. Take a mid-size repo: 40 items on the test list at roughly 30 tokens each, so 1,200 list tokens. A diff of 300 changed lines, with context lines, lands somewhere around 6,000 tokens once the patch is serialized. Instructions and the required output schema add another 800. Output is short: a verdict per item, call it 600 tokens. One pass.

That is about 8,000 input tokens and 600 output tokens per gate run. Multiply by 20 pull requests a day, and you get 160,000 input tokens and 12,000 output tokens daily, before any caching or batching. At the per-million pricing of a small fast model, that is a rounding error against a single engineer hour. Even at frontier-model pricing, a month of runs stays inside a plausible expense line rather than a budget conversation.

Three levers move this number, in order of impact:

1. Diff size. Token cost is linear in serialized diff length, and serialized length grows faster than line count because context lines and file headers come along. Capping diffs at a few hundred lines cuts this pool and every other pool at once.
2. List shape. A list of 200 loose items is mostly noise. A list of 40 atomic behaviors, each checkable on its own, costs less to send and produces better verdicts. Write behaviors as verb plus object plus condition, and keep them small enough that one focused test exists for each (Rhesis on turning user stories into behaviors and metrics).
3. Passes and caching. Repeat runs to measure flakiness multiply cost directly. Cache the static list and instructions so only the diff is re-sent.

There is a subtler point: models are unreliable judges of their own workflow adherence. An exploratory evaluation of TDD inside the agent loop found that agents following a disciplined workflow did not produce clearly better outcomes than agents that did not, and that agents routinely skipped the steps they were instructed to follow (Böckeler's TDD-in-the-agent-loop study on martinfowler.com). Translation for a gate: do not pay tokens to check process compliance you cannot verify. Pay for outcomes you can check deterministically.

Command minutes are the easiest pool to forecast and the easiest to accidentally blow up.

wall minutes per run = ceil(Σ check durations on affected files ÷ parallel workers) + setup
monthly minutes = runs × wall minutes

Numbers make the shape clear. A manifest with 30 checks, each averaging 2 seconds on the files a diff touches, is 60 seconds of serial work. With 8 workers it is about 8 seconds plus setup. Setup is the part that surprises people: hook environments get installed once and reused, but the first run of a new hook can take minutes because the tool builds an environment rather than just running (pre-commit's introduction).

The false-failure risk column is where command minutes and reviewer minutes couple. A formatter that rewrites a file and exits non-zero has done its job and produced a failure that costs the author seconds, not a reviewer minutes. A flaky unit test in the subset produces the same signal and costs a reviewer ten minutes of investigation. Both are "failures". Only one belongs in front of a human.

Two levers dominate here. First, run cheap deterministic checks before anything else, including before the model reads the diff. Deterministic static analysis catches mechanical issues at a fraction of the cost of model review and, crucially, does not consume context or produce a judgment call (Codacy on deterministic static analysis in AI coding workflows). Second, scope checks to affected files rather than the whole tree. pre-commit's default behavior is to run hooks on changed files during a commit, with an explicit all-files mode reserved for when you add a new hook (pre-commit usage docs).

Command minutes also do not stay flat. Every hook you add to the manifest is a permanent per-run tax, so a manifest that grows unchecked turns a 20-second gate into a two-minute one a year later, and a two-minute gate on a busy day pushes developers to bypass it locally.

The reviewer formula has a hard floor and a nasty multiplier.

reviewer minutes = runs × (base read time + failures × investigating/failure × (1 + dispute rate))

Base read time is what a reviewer spends scanning a green gate's output to confirm it actually ran and actually covered the list. Call it 30 seconds. Failures are the interesting term. A real failure costs a few minutes to confirm and route. A false failure costs that plus a decision about whether the gate can be trusted, and that decision is what burns senior time.

Use a worked month. Twenty gate runs a day, 30 days, so 600 runs. Base read time at 30 seconds is 300 minutes, or 5 hours monthly, which is tolerable. Now add a 5 percent failure rate, so 30 failures. If a fifth of those are false, that is 6 false failures. At 15 minutes each, that is 90 minutes, which is still small. Push the false-failure rate to 20 percent of 30 failures, so 6 real and 6 false, and the same arithmetic holds. The number only explodes when failures cluster: a flaky test that fails on a third of runs turns 600 runs into 200 failures, and at even 5 minutes each that is over 16 hours a month, more than the base read time by a factor of three.

This is the same bottleneck that shows up in AI-assisted review generally: reviewers lose trust within a week when a tool floods them with low-value comments, and the fix is severity thresholds and suppression rather than more comments (Dev.to on building an AI review layer in CI). An LLM reviewer running on every pull request can complete in parallel with CI checks and cost very little per PR, but the value comes from what it filters out, not from how much it says (LLM code review as a CI quality gate).

There is a second, quieter cost: review quality degrades on long diffs. Human review attention is finite, and generation tools increase PR volume and diff size at the same time. A gate that demands a careful read of every failure on every large diff is asking reviewers to do the thing they are worst at, repeatedly.

Here is the whole thing assembled, with every assumption stated so you can substitute your own.

Assumptions: 20 gate runs per working day, 22 working days, so 440 runs monthly. Average diff 300 changed lines. 40 list items. 30 manifest checks at 2 seconds each, 8 workers. Failure rate 8 percent, of which a quarter are false. Reviewer base read 30 seconds; real failure 4 minutes; false failure 12 minutes.

The decision rule falls out of the table: the reviewer pool is roughly six times the wall-clock pool and dwarfs the token pool, so effort spent shaving tokens is misallocated. Shave false failures first, then wall time, then tokens. Cutting the false-failure rate from 2 percent of runs to 1 percent saves about 53 reviewer minutes a month here, which is more than the entire command pool.

That rule has a boundary. On a repository with very large diffs and a very long list, token cost stops being a rounding error, and the diff cap becomes the dominant lever. The test is arithmetic, not taste: compute both pools with your own numbers before choosing what to optimize.

Cost per run is a vanity metric, because you can always make it lower by catching nothing. The number that survives a budget review is cost per regression caught.

cost per regression caught = total gate cost over a period ÷ regressions the gate caught in that period

"Caught" needs a definition you can actually count. A reasonable one: a regression is caught when the gate fails a change and the author changes the code to fix a behavior, rather than to satisfy the gate. That distinction separates the gate working from the gate being appeased.

Watch the denominator for gaming. A gate that fails on style produces failures that authors "fix" by reformatting, which inflates apparent catches without preventing anything. If your catch count is dominated by formatter nudges, the gate is mostly doing the work a pre-commit hook does for free (pre-commit-hooks collection).

A worked comparison. Gate A catches 12 real regressions a month at a total cost of 800 minutes, so about 67 minutes per catch. Gate B catches 14 a month at 1,400 minutes, about 100 minutes per catch. Gate B looks worse, and it is, unless the 14 include two that would have reached customers. Then the comparison is between minutes per catch of different severity, and the cheap-looking gate is the expensive one. Keep the severity in the numerator.

Track three counters alongside the budget: runs, real failures, false failures. The ratio of false to real failures is the single best predictor of whether the gate will still be enforced in six months.

The cheapest gate is the one the team does not route around. Three practices get you there.

Filter deterministically before you spend tokens. Order the gate so parse checks and cheap linters run first and stop the run. Deterministic checks resolve or eliminate the obvious cases without a model call, which reduces both token spend and the number of judgment calls handed to a reviewer (Codacy).

Cap the diff, not the list. A diff over a few hundred lines gets split or the gate gets a lighter list. Both are cheaper than a reviewer skimming a 900-line change.

Version-pin everything the gate runs. A hook whose version floats will change behavior without anyone editing the manifest, and every behavior change in a gate is a potential new false failure. pre-commit pins each hook to a revision by design, and that is the property worth preserving (pre-commit configuration).

If the list spans many models, image or document checks, or you want per-run cost visibility without wiring a separate billing path per provider, one key across model types keeps the token pool measurable: Quanta Inference offers text, image, video, audio and embedding models through a single OpenAI-compatible API. That is a small win against the reviewer pool, so do it after you have fixed false failures.

When the pool you actually need to cut is reviewer time, the lever is delegation rather than another dashboard. A scoped expert that reads the diff, applies the list and returns a verdict you can act on reduces the number of judgment calls that reach a person.