A Version Id That Survives a Prompt Hotfix

A content hash names a prompt by hashing its bytes. Git is the canonical example: it is a content-addressable filesystem that names files, trees and revisions by hash, which makes integrity checking trivial (a corrupted byte no longer matches its name) and lookup fast (Git's hash function transition doc). Prompt tooling vendors recognize the appeal and offer content-addressable ids specifically so identical prompts produce identical ids and avoid duplication and ambiguity (Braintrust on prompt versioning). Hash chains go further: a chain hash makes each version depend on all previous versions, so changing any version invalidates every subsequent chain hash and tampering becomes detectable (PromptOwl integrity docs).

The problem is exactly the property that makes it good: any difference, however small, produces an unrelated id. A support-triage prompt with a template containing Classify the ticket: {{ticket}} changes its greeting line from "You are a helpful assistant" to "You are a precise assistant." The old hash and the new hash have no characters in common. Every dashboard that groups by prompt id now sees two unrelated prompts, and a query like "how did the previous version perform" cannot find its children. A content hash is also unreadable, and whitespace kills it: if one service strips trailing spaces before hashing and another does not, the same semantic prompt gets two ids.

The monotonic counter gives you 1, 2, 3: stable, ordered, short, cheap to store, and opaque. A bare 42 tells a reader nothing. The killer problem is scope. A counter is only meaningful within a namespace (this database, this prompt family, this service). If two teams both have a prompt at version 3 and a log line records prompt_version=3, the join produces garbage. A counter assigned by the database is also unstable across environments: if migration scripts run in a different order in staging, version 3 there may be version 5 in production, and nobody notices until an incident.

The slug plus revision fixes the family problem. The slug is a stable human name (support-triage, invoice-extract, churn-reason). The revision is an integer that increments on every change. When the greeting line is reworded, support-triage.7 becomes support-triage.8, and every log row from both versions still groups under support-triage. A variable value change does the same. A whitespace normalization does the same. The scheme's weakness is that it is not exact: support-triage.8 says an edit happened but not exactly what text ran. That is why you store a content hash beside it.

Prompt versioning guidance converges on the same point in different language: every change, no matter how small, should produce a new version rather than mutating an existing one that may already be referenced in logs, traces or evaluations (Braintrust). Slug plus revision enforces that convention by construction.

Take a prompt family called ticket-triage that serves 100,000 calls a day, with 12 small edits landing across a month. Assume each edit ships on its own day, so roughly 12 of the 30 days run on a freshly changed prompt, and about 3,333 calls per day go out under the changed version.

With slug plus revision, you get 13 versions (ticket-triage.1 through ticket-triage.13) and one family. A dashboard filtering prompt_slug = "ticket-triage" covers all 3 million calls for the month and can pivot the revision to compare the 12 edit days against the 18 baseline days.

With content hashes, you get 13 unrelated ids and no family. To answer "did the month get worse after the edits" you must maintain an external mapping table from hash to family, and keep it correct as more edits land. If that table lives in a spreadsheet, it will be wrong within a week.

With a bare counter, you get 13 rows labeled 1 through 13 with no indication of which prompt family they belong to. If any other prompt also has a version 7 in the same store, a query grouping by prompt_version = 7 silently mixes two families, and the resulting report is worse than no report.

The numbers make the argument: 12 edits produce 13 ids under all three schemes, but only one scheme gives you 13 ids and one join key. Multiply by the number of prompt families in a real system (dozens) and the number of teams writing to the same log store (more than one), and the mapping-table approach collapses.

Instrument the prompt id as an attribute on the LLM span, not as a separate log line. Attributes are the cheapest and highest-correlation place to put data you will later ask questions of: a single event with feature flags, request properties and user ids lets you ask "what is different about the requests that failed" (Honeycomb on span versus attribute). Span events, by contrast, are being deprecated in favor of log-based events correlated with the current span (OpenTelemetry's deprecation announcement), so do not encode prompt versions as span events.

Attribute names should follow a consistent convention. The GenAI semantic conventions define a gen_ai.* namespace covering agents, request model, token usage and more, though the registry notes the attributes have moved to a dedicated repository (Gen AI attribute registry, semantic-conventions-genai repository). Prompt-specific attributes are your own extension, so pick names once and write them down: prompt.slug, prompt.revision, prompt.content_hash.

import hashlib

def derive_prompt_version(slug: str, revision: int, template: str) -> dict:
    normalized = "\n".join(line.rstrip() for line in template.strip().splitlines())
    content_hash = hashlib.sha256(normalized.encode("utf-8")).hexdigest()
    return {
        "prompt.slug": slug,
        "prompt.revision": revision,
        "prompt.version": f"{slug}.{revision}",
        "prompt.content_hash": content_hash,
    }

version = derive_prompt_version("ticket-triage", 8, TEMPLATE)
span.set_attribute("prompt.slug", version["prompt.slug"])
span.set_attribute("prompt.revision", version["prompt.revision"])
span.set_attribute("prompt.version", version["prompt.version"])
span.set_attribute("prompt.content_hash", version["prompt.content_hash"])

Note the normalization step: strip the template, strip trailing whitespace on every line, then hash, so a trailing space in one editor does not produce a different hash. Hash the template with placeholders unsubstituted, never the rendered prompt. The rendered prompt changes on every call because it contains the ticket text, so hashing it gives one hash per request and no version concept at all. If you need the rendered content for sampling, store a prompt hash and token counts on every call, but sample full prompts at a low rate because prompt bodies routinely contain PII (Techlusion's agent observability guide).

Choose the digest and stick to it. SHA-256 is the standard pick for content addressing: Git chose it over several contenders for its security properties and fast implementations (Git's transition document), and it is widely available in standard libraries (Encryption Consulting's SHA-256 overview). Never mix digests across services, because sha256(content) != sha1(content) and the join breaks silently.

Once the id is in place, the query pattern becomes straightforward. Group by slug for family-level trends, pivot by revision for version-level comparisons, and join on content hash only when you need to prove two records describe the identical bytes.

This matters most during an incident. Suppose the task success rate for ticket-triage drops. The first question is whether one revision is worse or the whole family regressed. With slug plus revision, the query is a GROUP BY prompt.revision on existing span data. With a content hash, you need the mapping table, and if it is stale you may conclude the family regressed when in fact only revision 11 is bad. That is a decision rule worth writing down: if you cannot group by family without an external lookup, your id scheme is wrong.

Evaluation records should carry the same attributes. A version tested in an eval run and the same version that ran in production should share the slug and revision, so the eval row and the trace row join without translation. Record which version a first attempt used and which the fallback used, because a version change mid-task is a common source of confusing outputs.

Percentiles matter for the comparison too. An average that lumps revision 11's slow tail with revision 10's normal traffic tells you nothing; P50 shows the typical case, P95 catches the early warning, and P99 exposes outliers, which is why latency is analyzed by percentile rather than mean (OneUptime's percentile guide). Pivot your latency and cost metrics by prompt.revision and the comparison becomes honest.

Most teams have some combination of hardcoded prompt strings, a config file and a few ad-hoc logs. The migration does not need a big-bang rewrite; it needs one rule enforced from a starting point.

1. Pick the starting revision. Set the current version of every prompt family to slug.1 on the day you migrate, and record the content hash of the normalized template at that moment. Do not reconstruct history; start clean.
2. Increment on every change. Generate the revision from the change itself (commit counter, migration number, or CI build number scoped per slug). Never edit a version in place. If a hotfix touches one line, the revision increments, and that is correct.
3. Emit all three attributes. prompt.slug, prompt.revision, prompt.content_hash. The hash is what lets you answer "did production run the artifact QA signed off" without trusting a mapping table.
4. Write the normalization rule into the repo. A short function with tests is enough, so every service strips and hashes the same way.
5. Keep a version map. One table or file per prompt family mapping revision to content hash to git commit to evaluation run ID. This is the only lookup table you need, and it is small because it has one row per edit.
6. Alert on family-level SLOs, drill into revision-level detail. Task success rate and end-to-end P95 belong to the family; token cost and per-call latency spikes belong to the revision. Mixing them produces alerts nobody trusts.

If the tooling around this is the sticking point rather than the naming, a specialist can build the registry, the normalization function and the dashboards with your team: describe what you need and it gets picked up and built with you (https://quantaintelligence.ai/).