Preflight Checks: Printing Token Cost Before the Run

Before you can compare a count to a budget, you need a realistic budget. Published ranges are wide because "agentic task" covers everything from a single lookup to a multi-hour refactor.

For a browser agent, one project reports roughly 6,500 input tokens and 500 output tokens per test on early data, with about 1,000 input tokens per step (browser-agent discussion). Production coding agents solving software engineering tasks can cost $5 to $8 per task when unconstrained (agentic task complexity estimation).

The practical lesson is that your preflight budget should be set from measured runs, not from intuition, and it should include a multiplier. One budgeting reference recommends applying a 1.7 to 2.0x multiplier to estimated tokens to account for retries, system prompts, and context overhead (token usage guide). That multiplier is not padding. It is the difference between a budget that trips on a normal Tuesday and one that only trips when something is genuinely wrong.

A decision rule that works: set the preflight budget at roughly half of what a healthy run of that task type has consumed in your own logs after applying the multiplier. Files are the largest controllable input, so capping them early leaves room for the unpredictable part (model output and retries) without starving the run.

The preflight needs a list, not a directory. "Read the auth module" is not a list. A resolver turns it into one:

from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class Source:
    path: Path
    reason: str  # why the task wants this file

def resolve_sources(root: Path, patterns: list[str], deny: list[str]) -> list[Source]:
    seen: dict[Path, Source] = {}
    denied = {Path(root, d).resolve() for d in deny}
    for pattern in patterns:
        for p in sorted(root.glob(pattern)):
            resolved = p.resolve()
            if not resolved.is_file() or resolved in denied:
                continue
            seen.setdefault(resolved, Source(resolved, pattern))
    return list(seen.values())

Two details matter. Deduplicate on resolved paths, because glob patterns overlap and double counting inflates the estimate. And keep the glob pattern attached to each source as its "reason", because the report is far more actionable when it says which rule pulled a file in.

Deny lists are not optional in a real codebase. Lock files, minified bundles, generated clients and vendored dependencies are the usual suspects for a single file blowing an entire budget. Excluding them is cheaper than discovering them in the report.

A token is not a word. In English prose, one token is approximately four characters or about three quarters of a word, and the same text can produce different counts depending on the model and its encoding (OpenAI token guide). Characters divided by four is acceptable for capacity planning but breaks on code, JSON, non-English text and emoji, which is exactly the content an agent sends (counting LLM tokens and costs).

So count with the tokenizer of the model you are actually calling. For OpenAI models that means tiktoken with the encoding for the target model (OpenAI token guide). Anthropic exposes a count-tokens endpoint that is free to call subject to its rate limits, and provider docs describe the result as an estimate rather than an exact figure (counting LLM tokens and costs). Gemini has an equivalent counting call. For open models, use the model's own tokenizer.

If you want a tool rather than a library call, count-tokens wraps tiktoken and handles directories, JSON and CSV output, and a maximum-token threshold (count-tokens on PyPI). The token-count package does the same for strings, files and directories (token-count repo).

Here is the counting half, with a caching layer because you will call it on every run:

import tiktoken
from pathlib import Path

_ENC = tiktoken.get_encoding("cl100k_base")

def count_file(path: Path) -> int:
    text = path.read_text(encoding="utf-8", errors="replace")
    return len(_ENC.encode(text, disallowed_special=()))

The encoding name above is an example, not a recommendation. Pick the encoding that matches your target model, because a count taken with the wrong tokenizer is a guess (counting LLM tokens and costs). Note also that this count covers file text only. Message structure, role framing, tool schemas and image inputs all add tokens you are billed for, so the file total is a floor, not a ceiling (OpenAI token guide).

Now the comparison. The report is the product of this whole exercise: it is what the engineer reads when the run is blocked, and it should make the next action obvious.

from dataclasses import dataclass

class BudgetExceeded(Exception):
    def __init__(self, report: "PreflightReport"):
        super().__init__(f"preflight over budget by {report.overage} tokens")
        self.report = report

@dataclass
class LineItem:
    path: str
    tokens: int
    reason: str

@dataclass
class PreflightReport:
    items: list[LineItem]
    total: int
    budget: int

    @property
    def overage(self) -> int:
        return max(0, self.total - self.budget)

    def render(self) -> str:
        rows = sorted(self.items, key=lambda i: i.tokens, reverse=True)
        lines = [f"{'tokens':>8}  {'share':>6}  file"]
        for i in rows:
            share = i.tokens / max(1, self.total)
            lines.append(f"{i.tokens:>8}  {share:>5.0%}  {i.path}  ({i.reason})")
        head = f"total {self.total} / budget {self.budget}"
        tail = f"over by {self.overage}" if self.overage else "within budget"
        return "\n".join([head, *lines, tail])

def preflight(sources, budget: int) -> PreflightReport:
    items = [LineItem(str(s.path), count_file(s.path), s.reason) for s in sources]
    report = PreflightReport(items, sum(i.tokens for i in items), budget)
    if report.total > budget:
        raise BudgetExceeded(report)
    return report

Sorting by token count descending is the small design decision that carries the value. When a run is blocked, the top three rows usually explain the whole overage, and the fix is one glob pattern narrower or one lock file denied.

Worked example, using the published heuristics. Say the resolver returns 34 files totalling 410,000 characters of mostly Python and Markdown. Characters divided by four gives about 102,500 tokens. A real tokenizer on mixed source code and prose typically lands higher than that heuristic, since source code runs around 1.5 to 2.0 tokens per word versus roughly 1.33 for standard English (token usage guide). Suppose the actual count is 128,000 and the budget is 60,000. The report shows the top file (a generated client) at 61,000 tokens, which is 48 percent of the total and explains more than half the overage on its own. Denying that one path brings the run under budget without touching the task's real inputs. That is a decision you cannot make from a character count.

The check is worthless if it lives next to the run instead of in front of it. Wrap the call:

def guarded_run(task, run_fn):
    report = preflight(task.sources, task.budget)
    try:
        result = run_fn(task)
    except Exception as exc:
        log.warning("run failed after preflight: %s", exc)
        raise
    log.info("preflight %d tokens, run complete", report.total)
    return result

Three properties make this wrapper worth the code. It raises a distinct exception type, so callers can catch BudgetExceeded and either surface the report to a human or retry with a narrowed file list. It holds the report on the exception, so the caller never has to reconstruct what went wrong. And it logs the preflight total on success, which gives you the dataset you need to recalibrate budgets over time.

Keep the failure path side-effect free. A blocked preflight should not have written files, opened a pull request or sent a message. Because the wrapper runs before the model call, that is the default, but only if the file resolver is also read-only.

One more guard belongs here: floors as well as ceilings. A task that resolves to zero files is almost always a glob bug, and it will burn tokens discovering that fact. Treat an empty file list as a separate typed error.

A preflight check bounds the input. It does not bound output tokens, reasoning tokens, or the number of turns. Reasoning tokens are not visible in the answer but count toward output usage and are billed as output tokens, so a short visible answer can consume far more than its text suggests (OpenAI token guide). An input count does not predict how many output tokens the model will generate (same source).

The practical consequence: preflight is one of three controls, and it is the one that catches the class of errors you can see in advance (wrong glob, runaway directory, vendored dependency). The other two catch errors you cannot. An output cap per call bounds the tail. A turn limit or wall-clock timeout bounds the loop. Teams often find that a small number of scheduled, repetitive tasks dominate spend, so routing those to a cheaper tier matters as much as any guard (production cost discussion).

If you want the whole stack behind one API key rather than wiring per-provider token counters and routing rules yourself, that is what Quanta Inference is for, and it stays OpenAI-compatible.

Work through it top to bottom on one task type first, ideally a scheduled job where you can compare the preflight total against the actual usage reported by the provider afterward. The gap between those two numbers tells you how much the input count understates the real cost, and that gap is your multiplier.