Token Math for Long Files: Counting a Codebase Before You Load It

Before any tokenizer runs, you can get exact, reproducible counts for every file. Lines and bytes have no model dependency, no encoding question and no sampling error. They cost a directory walk.

On a Unix-like system, three tools cover most of it:

# Total lines and bytes per tracked file, sorted by size
git ls-files -z \
  | xargs -0 -I{} sh -c 'wc -lc "{}"' \
  | sort -k2 -n -r \
  | head -40

That gives you a ranked list of the files that will dominate any token estimate. In most repositories the top 40 files account for a large share of the total, which means the expensive part of the calculation is a short list, not the whole tree.

Two refinements matter. First, exclude what the agent will never read: vendored dependencies, generated code, lockfiles, binary assets, minified bundles. If those are in your repository, git ls-files already excludes ignored files, but generated directories that are committed need an explicit filter. Second, exclude files by extension. A dependency lockfile can be tens of thousands of lines of near-useless tokens; an image file has bytes but almost no meaningful text tokens.

The output of level one is a table: path, lines, bytes. That table is your inventory. Everything downstream refines it.

Token counts are model-specific, and the same text can produce different counts under different encodings and providers as OpenAI's own documentation notes. That means there is no single "correct" token number for a file, only a number for a specific tokenizer. This is the first place where precision has diminishing returns.

For a first pass, the standard heuristics are good enough: one token is roughly four characters or about three-quarters of a word for English text per the same OpenAI guidance. Those heuristics are calibrated on prose. Source code tokenizes differently: the reference table puts ordinary prose at about 1.33 tokens per word and source code at 1.5 to 2.0 tokens per word in the token usage guide. Using the prose ratio on a codebase systematically undercounts, sometimes by a third.

A worked example makes the gap concrete. Take a repository with 400,000 lines of code at an average of 40 characters per line. That is 16 million characters. At four characters per token, the prose heuristic predicts 4 million tokens. At the code-calibrated ratio of 1.75 tokens per word, with roughly six words per line at 40 characters per line, the estimate lands closer to 4.2 million. Those are not catastrophic differences, but they compound: if your agent makes 30 tool calls and re-reads context each turn, a 5 percent underestimate becomes a 5 percent underestimate of a very large number.

When you need a real count, use the tokenizer that belongs to the model you will call. The reference guide to counting tokens lists the exact route per provider: tiktoken for OpenAI, the count-tokens endpoint for Anthropic, countTokens for Gemini, and the published Hugging Face tokenizer for open models per this OpenReplay guide to token counting. It also flags an important trap: Anthropic's counting endpoint returns an estimate, and the billed figure comes from the response's usage fields. Anthropic's tokenizer has also changed across model generations, with later Claude models turning the same text into roughly 30 percent more tokens than earlier ones per the same guide. A count taken against the wrong generation is not a conservative estimate; it is a different number.

If you want a command-line tool that walks a directory and reports token counts, several exist. token-count wraps tiktoken and can count a string, file or directory as documented in its repository. The count-tokens package adds directory patterns, streaming for large files and a --max-tokens flag that warns when a file exceeds a limit per its PyPI page. Either one converts level one's inventory into level two's token table.

Exhaustive tokenization of a large repository is slow, and it produces a number more precise than you need for a decision. Sampling gives you the same decision at a fraction of the cost, provided you sample correctly.

The wrong way to sample is to grab the first 20 files returned by the directory walk. That sample is biased by whatever the filesystem returns first, which in practice means the root-level configuration files, not the code that dominates the byte count. The right way is to sample by size. Sort files by line count, then take a stratified sample: a handful of the largest files, a handful from the middle of the distribution, and a handful of the smallest. Tokenize each, compute tokens per line for each stratum, and weight back to the full population.

The sampling error is small because lines-to-tokens is a stable ratio within a file type. The predictable failure is mixing file types into one pool: a Markdown documentation file and a minified JavaScript bundle have very different tokens-per-line ratios, and pooling them produces a ratio that describes neither. Stratify by extension.

The rule that falls out of this table: sample by size and extension, then spend the exhaustive tokenizer run only on the files that survived the cut. If your top 40 files are 60 percent of the total, tokenizing those 40 exactly and estimating the tail gives you a number within a couple of percent for the price of 40 tokenizer calls.

One package worth knowing about exposes a simple programmatic interface for this: TokenCount takes a model name at construction and then counts strings, files or directories per the token-count repository. That makes the sample-and-weight loop a short script rather than a project. If you would rather press a button than write the loop, browser token counters accept a paste and report token, word and character counts locally, with the caveat that they run a reference tokenizer rather than every provider's per token-calculator.net.

Even an exact tokenizer count of every file is not the billed number. Several things sit between your count and the invoice.

The billed prompt is the whole request body: system prompt, role framing, tool schemas, and the full re-sent conversation history, not just the file contents you counted per the OpenReplay guide. Tool definitions alone can be substantial. A coding agent with a dozen tools and their JSON schemas adds tokens before it has read a single line of your repository.

Then there is the multiplier from multi-turn work. Context doubles with each turn because history accumulates, and a ten-cycle reasoning loop consumes fifty times the tokens of a single pass per the agentic complexity analysis. Unconstrained coding agents on real software tasks have been measured at $5 to $8 per task per the same analysis. A single file read is not the unit of cost; a file read repeated across every turn of a thirty-turn loop is.

The reference budget guidance is to multiply your raw estimate by 1.7 to 2.0 to account for retries, system prompts and context overhead per the token usage guide. That multiplier is doing real work: on a four-million-token codebase estimate, the difference between raw and budgeted is another four million tokens of headroom.

The structural fix is to treat the token budget as first-class agent state rather than an afterthought. Budgets need to be checkpointed and enforced during execution, not checked at the end per the same analysis. If you are routing work across several models or providers, centralizing that accounting saves a lot of duplicated bookkeeping, which is one reason a single API surface like Quanta Inference is worth having in the stack.

With a total token count in hand, the question becomes what to do with each file. The decision rule is a single ratio.

Let T be the total tokens the agent would need to read for a complete answer, and let W be the usable context window, meaning the window minus the room you reserve for the system prompt, tool schemas, the growing history and the output. Call the ratio R = T / W.

Two thresholds in that table need justification. The 0.3 figure is a hedge against degradation in the middle of the window; a context that is one third full leaves the model room to attend across its whole history. The 0.5 target inside the second band comes from the same logic: you want headroom for the tool outputs that will arrive during execution, which you did not count when you measured the codebase.

The summarize-versus-retrieve choice inside that rule is not arbitrary. Summarizing loses information permanently and costs a pass over the whole corpus to produce. Retrieving keeps information intact but pays a search cost per step and risks pulling the wrong files. Summarize when the corpus is large relative to the window and the same subset is needed repeatedly; retrieve when the corpus is large and each step needs a different, narrow slice. A codebase larger than five times the usable window almost never needs to be read in full for a single task, so the default should be retrieval with a good index, not a wholesale summary.

The rule also explains why an estimate that is 20 percent off does not change the decision. If R is 3.0, being wrong by 20 percent moves it to 3.6 or 2.4, and both are still in the retrieve band. Estimation precision matters at the boundaries, not in the middle.

Retrieval granularity matters as much as the decision to retrieve. Splitting a file at a fixed token count produces chunks that begin mid-function and reference variables that were defined in a previous chunk, which is the code equivalent of cutting a sentence in half.

Chunk on structural boundaries instead. In most languages those are functions, classes, methods and top-level blocks. A chunk that contains one complete function with its signature, body and docstring is intelligible on its own; a chunk that contains the second half of one function is not. Language-aware parsers exist for this, and the tooling discussed in this Stack Overflow thread on counting code tokens is a reminder that "token" means two different things in this context: lexical tokens in a programming language and model tokens. The structural parsers use the former; you are budgeting the latter. Do not confuse the two counts.

For each chunk, record the file path, the enclosing symbol name, the start and end line, and the token count. That metadata is what makes retrieval work: the agent can ask for "the definition of the function it just saw called" rather than searching by text similarity.

There is a parallel here worth noticing. Breaking work into small, independently meaningful pieces is the same principle that agile teams apply when they split large stories into vertical slices that each deliver something demonstrable, rather than horizontal slices across architectural layers as described in this guide to story splitting. The failure modes are similar too: a horizontal split produces pieces that cannot stand alone, and so does a chunk boundary that cuts through a function per this comparison of vertical and horizontal slicing. Keep chunks vertical: whole units of behavior, not layers of syntax.

The whole workflow collapses into a short pipeline. This is deliberately minimal; the point is the sequence, not the implementation.

import subprocess
from pathlib import Path
from token_count import TokenCount

SKIP = {".lock", ".min.js", ".map", ".png", ".jpg", ".svg", ".woff"}

files = subprocess.run(
    ["git", "ls-files"], capture_output=True, text=True
).stdout.splitlines()

tc = TokenCount(model_name="gpt-4")  # tokenizer matching your target model
inventory = []

for name in files:
    if Path(name).suffix in SKIP:
        continue
    p = Path(name)
    data = p.read_text(errors="ignore")
    lines = data.count("\n") + 1
    inventory.append({
        "path": name,
        "lines": lines,
        "bytes": p.stat().st_size,
        "tokens": tc.num_tokens_from_string(data),  # replace with sampling
    })

inventory.sort(key=lambda r: r["tokens"], reverse=True)
total = sum(r["tokens"] for r in inventory)
print(f"files={len(inventory)} tokens={total:,}")

Three edits make this production-grade. Replace the full tokenization with stratified sampling plus weighting, as described above. Add a per-extension breakdown so a single anomalous file type does not skew the total. And swap TokenCount for the tokenizer of the model you will actually call, since the same repository will produce different totals under different encodings per the OpenAI token guidance.

Once the total is printed, compute R and read the decision off the table in the previous section. That is the whole method: exact sizes, sampled token ratios, one ratio, one band, one decision.

If the agent work that follows is long-running, the same discipline applies inside the run. Agents that hold a stable goal structure while adapting the path, with explicit stop conditions and persisted progress, stay coherent far longer than agents that simply keep going per this guide to agent planning. Token budgets are the same idea applied to context: checkpoints, limits, and a clear rule for what happens when the limit is reached. Teams that would rather hand the whole pipeline to a specialist can do so through Quanta Experts, and the counting logic itself is worth packaging as a reusable skill so every agent in the fleet measures input the same way.