A Preflight Registry for File Hashes and Token Counts

The cache key needs a digest that changes whenever the bytes change and stays stable when they do not. Two candidates come up: SHA-256 and BLAKE3. Both are 256-bit cryptographic hashes targeting 128-bit security with no known practical attacks, but SHA-256 is a NIST standard used in TLS, JWT and code signing, while BLAKE3 is faster in software and parallelizes across cores (MojoAuth).

The performance gap is real but conditional. BLAKE3 only wins when it can use SIMD and threads. In environments that cannot exploit those, a pure software SHA-256 can actually be twice as fast, as the noble-hashes maintainer explains when comparing the two for file hashing (noble-hashes discussion). Benchmarks of these algorithms show the advantage depending on hardware and input size (arXiv).

For a preflight registry, prefer SHA-256. The reasons are practical rather than cryptographic:

• It is available in every standard library (Python's hashlib, Node's crypto, Go's crypto/sha256), so you do not add a dependency.
• Its output is a fixed 32 bytes, which makes the cache key compact.
• It is FIPS-approved where compliance applies, which matters if you ever ship this to a regulated environment (MojoAuth).

If your files are enormous and you control both ends, BLAKE3 is a defensible upgrade, but only after measuring on your actual hardware (MojoAuth). Do not switch mid-project without bumping a namespace in the cache, or you will mix digests from two algorithms in one table.

One trap to avoid: never hash the path. A file that is renamed or moved still has the same bytes, so it should still hit the cache. That is exactly the behavior you want in a repository where reorganizing directories is routine.

The invalidation rule is the whole design. The cache key is the tuple (file hash, tokenizer name, tokenizer version). Every part earns its place:

• File hash changes when the bytes change, so an edit forces a recount.
• Tokenizer name changes when you switch providers, since each tokenizer was trained on different data with a different vocabulary (OpenReplay).
• Tokenizer version changes when the same provider ships a new tokenizer, since Anthropic's docs describe a newer tokenizer that turns the same text into roughly 30 percent more tokens than earlier models did (OpenReplay).

That last point is the argument for treating version as part of the key rather than part of the value. A count taken with an old tokenizer does not transfer to the new one, so a version bump must produce a miss. If you stored the version only in the row, a stale count would look valid.

The registry itself is small. A single SQLite table is enough for most teams:

CREATE TABLE token_counts (
  file_hash       TEXT NOT NULL,
  tokenizer_name  TEXT NOT NULL,
  tokenizer_ver   TEXT NOT NULL,
  token_count     INTEGER NOT NULL,
  counted_at      TEXT NOT NULL,
  PRIMARY KEY (file_hash, tokenizer_name, tokenizer_ver)
);

The primary key is the cache key. The counted_at column is for auditing and for deciding when to prune, not for invalidation. You invalidate by changing the key, never by comparing timestamps. That is what makes the rule simple enough to reason about.

If you prefer no database, a JSON sidecar keyed by hash works for smaller projects, but it rewrites the whole file on every miss and loses the transactional guarantees SQLite gives you. A sidecar is fine up to a few thousand entries; past that, use SQLite.

The comparison below shows the trade-off:

Whichever you pick, keep the registry out of version control unless you want to share counts across machines. Committing it works, but it grows the repo and creates merge conflicts on the primary key.

The memoized function has a fixed shape. Read the bytes once, hash them, build the key, look it up, and only tokenize on a miss.

import hashlib, sqlite3, tiktoken

def count_cached(path, conn, enc, encoder_name="cl100k_base", encoder_ver="0.14.0"):
    data = open(path, "rb").read()
    file_hash = hashlib.sha256(data).hexdigest()
    key = (file_hash, encoder_name, encoder_ver)
    row = conn.execute(
        "SELECT token_count FROM token_counts WHERE file_hash=? AND tokenizer_name=? AND tokenizer_ver=?",
        key,
    ).fetchone()
    if row:
        return row[0], True
    text = data.decode("utf-8", errors="replace")
    n = len(enc.encode(text))
    conn.execute(
        "INSERT INTO token_counts VALUES (?,?,?,?,datetime('now'))",
        (file_hash, encoder_name, encoder_ver, n),
    )
    conn.commit()
    return n, False

A few details matter here. You hash the raw bytes, not the decoded text, so a file that is not valid UTF-8 still gets a stable key. You decode with errors="replace" only at tokenization time, and you accept that replacement characters affect the count for malformed files.

The tokenization itself should use the encoding for your target model. OpenAI documents tiktoken and encoding_for_model for this purpose (OpenAI Help Center), and the tiktoken library is the reference BPE implementation for OpenAI models (tiktoken). The count from a raw encode call does not include message structure, tools or images, so for a full request count you need the provider's input-counting route (OpenAI Help Center).

Tikkoken downloads encoding data on first use, which fails on air-gapped machines unless you pre-populate the cache directory with the correct blob (Stack Overflow). That is one more reason to have a registry: a warm cache means fewer lazy loads of the encoding file.

If you need a quick CLI for spot checks, small tools exist that wrap tiktoken and report counts for files and directories (count-tokens on PyPI, token-count). Use them for exploration and the registry for production runs.

The registry only pays off if you can prove it. Log hits and misses per run and compare the totals against a full recount. A worked example with numbers you can reproduce:

• Repository with 2,000 files, average 300 tokens per file, so a full recount produces 600,000 token operations per run.
• On a typical run, 40 files changed. The registry returns 1,960 hits and 40 misses.
• You save 98 percent of the tokenize work, and the total count still matches a full recount because the misses are exactly the changed files.

Over a month of 20 runs per day, that is 24 million token operations not spent. The cost shows up in wall-clock time for a preflight gate that runs before every agent call. It also shows up indirectly in agent budgets: teams that add caching and tiered models have reported large reductions in monthly spend, though specific figures depend on their setup (Dify discussion).

To compare fairly, record the full recount as a baseline at least once. Store the file hash and the count from a full pass, then verify that the cached run produces the same totals. If the two disagree, you have a cache bug, not a tokenizer bug.

Audit prompts are easy to underestimate. A browser agent's average test consumed roughly 6,500 input tokens and 500 output tokens in early data (GitHub discussion). If your registry is feeding an agent preflight that reads many files, the savings scale with how many files are unchanged between runs.

A useful comparison table for the run:

The upgrade-day row is the one teams forget. When you bump tokenizer version, expect one full pass. Schedule it, and do not treat the spike as a regression.

The registry is safe by construction because it never returns a stale count. A hit only happens when the exact bytes and the exact tokenizer were seen before. A stale count can only survive if you break one of the rules: reuse a version string across incompatible tokenizers, hash the decoded text instead of the bytes, or skip the hash and key on path plus modification time.

That last shortcut is common and tempting. A file's modification time changes when the bytes change, so mtime plus path usually works. It fails in one case: a script that rewrites a file with identical content, or restores it from backup, bumps mtime without changing bytes, causing a needless recount. It fails harder in the other direction: a tool that preserves mtime while editing the content, which some build systems do, produces a stale hit. Hashing costs far less than tokenizing a large file, so hash the bytes.

If you must approximate, do so in the audit layer. Store both the hash and a word count, and flag rows where the token-to-character ratio is outside the normal band. That catches encoding failures without a full recount. Heuristics are acceptable for capacity planning but never for billing or context-window decisions (OpenReplay).

The same reasoning applies to request-level counts. The billed prompt is the full request body, including system prompt, role framing, tool schemas and re-sent conversation history, not just the user's message (OpenReplay). A file-level registry tells you the cost of the document but not the cost of the envelope around it. Keep those two counts separate.

The registry fits naturally into a preflight gate. Before an agent reads files, it asks the registry for counts and assembles a context plan against a token budget. Gateways that enforce token-based rate limits count the same units, and reading them from a registry rather than recomputing them keeps the limit check cheap (Zuplo).

When you route across multiple model providers, you need one row per tokenizer for each file. A text that is 1,000 tokens under one encoding may be a different size under another, and a single requests-per-minute policy cannot account for that variation (Zuplo). The composite primary key handles this without schema changes: add rows, do not add columns.

If your work spans many model types and you would rather not maintain per-provider tokenizer plumbing, Quanta Inference offers text, image, video, audio and embedding models through one OpenAI-compatible API key, which reduces the number of tokenizer versions you have to track. Teams that want a coding agent to handle the plumbing can install a skill from https://quantaintelligence.ai/skills and let the agent write the registry code against your repo.

Keep the registry small and boring. It is infrastructure, and infrastructure should not need attention. The more you can delegate the tokenizer handling to one endpoint, the less surface there is for a version mismatch to hide in.