A Repair Pass When a Checksum Mismatches: Find, Fix or Drop

Before choosing repair, rebuild or drop, gather three facts. Do not skip one because another looks obvious.

1. Does the source still exist at the recorded location, and is it readable?
2. What is the source's checksum right now, computed with the same algorithm and the same normalization as the stored one?
3. Does the stored checksum match the checksum recorded in the source's own provenance record (or the manifest that was published with it)?

You now have a comparison. Treat it as a small decision table:

The first row is the most common false alarm and the cheapest to fix. Checksums only compare meaningfully when both sides use the same algorithm and were computed over the same bytes, so a line-ending change, a re-serialized JSON document, or a different checksum tool will produce a mismatch with no content difference at all (FAIR Cookbook). This mirrors the classic Go modules mismatch, where developers hit checksum mismatch because the local cache or go.sum held a different hash than the freshly downloaded module (Stack Overflow: Go modules checksum mismatch); clearing the cache and regenerating the sum reconciled them because the bytes were never the problem.

If the source's current checksum differs from the stored one but matches the manifest, the source legitimately changed. Now the question is whether that change was authorized. This is an evaluation decision in the plain sense: a structured judgment against criteria tied to a standard, not a guess (Wikipedia: Evaluation).

Use this three-question rule. If all three answers are yes, repair by adopting the new checksum. If any answer is no, stop and escalate rather than silently rebuild.

• Was the source change committed through the normal path (a signed release, a reviewed commit, an approved upstream update)?
• Does the source manifest show the new checksum was published after the derived task was built?
• Does the downstream consumer of the derived task tolerate the change, or does its own contract pin the old version?

In practice, most rebuild systems pin by version, so "the source changed" usually means "a new version exists." The right move in that case is often to rebuild against the new version and keep the old derived task until the swap is verified, not to mutate the existing one. If you want most of this bookkeeping handled for you, a team plan at Quanta Intelligence (https://quantaintelligence.ai/) can host the rebuild and let specialists wire the checksum step into it with you.

The short answer compresses to one rule: repair when the content is right and only the checksum is stale; rebuild when the content genuinely changed and a new derived version is wanted; drop when you cannot establish what the content is.

Repair (recompute in place). The derived output is fine, the source is fine, and only the recorded checksum is wrong or was computed with different normalization. Recompute the checksum from the source, update the derived task's recorded input checksum, and re-run validation. This is the same remedy the Go community reaches for when the sum, not the module, is the stale part (Stack Overflow). Repair is cheap but should be rare; frequent repairs mean your checksum pipeline is inconsistent.

Rebuild (regenerate from the new source). The source changed and the change is authorized (see the three-question rule). Regenerate the derived task from scratch against the new version, record both old and new checksums, and gate the swap on a validation check so you never ship a rebuild that also fails.

Drop (remove and log). The source is missing, unreadable, or the manifest and the record disagree in a way you cannot resolve. Delete the derived task, mark it invalid, and keep a provenance row explaining why. Dropping is not failure; it is the correct outcome when trust cannot be established, and it is far safer than feeding an unvalidated task downstream. Storage and serialization choices affect whether bytes are stable across a round trip, which is one reason content can look identical yet hash differently (benchmark of JSON-compatible binary serialization).

The Fedora packaging case shows how subtle this gets: a maintainer hit a checksum mismatch only in the build system, not locally, and the culprit turned out to be how the cache served the file rather than the file itself (Fedora Discussion: checksum mismatch resolved). If the mismatch reproduces on one host but not another, suspect the transport or cache before you suspect the content, the same lesson seen in dependency-mismatch reports (k9s issue 2468).

Every repair pass needs an append-only record, one row per derived task touched. Provenance is the chain of custody, so the row must capture who decided, what evidence they used, and what changed (Monte Carlo on provenance vs lineage). A workable schema:

{
  "task_id": "derive_invoice_totals_0042",
  "run_id": "rebuild_2026_10_20T14:03Z",
  "source_uri": "s3://bucket/sources/invoices/v7.parquet",
  "algorithm": "sha256",
  "stored_checksum": "9f2c...a1",
  "source_checksum_now": "4b7e...c9",
  "manifest_checksum": "4b7e...c9",
  "decision": "rebuild",
  "reason_code": "source_changed_authorized",
  "evidence": ["manifest_published_after_task_build", "commit_signed"],
  "actor": "[email protected]",
  "decided_at": "2026-10-20T14:05:11Z",
  "old_task_retained_until": "2026-10-27T14:05:11Z"
}

Two details matter more than the field list. First, keep stored_checksum, source_checksum_now and manifest_checksum as three separate fields; collapsing them loses the exact evidence that justified the decision. Second, use an enum of reason_code values (no_real_mismatch, source_changed_authorized, source_changed_unexpected, checksum_corrupt, source_missing, transport_error) so you can count outcomes over time. When a rubric is used to judge outputs, separate, observable criteria make judgments consistent across reviewers (Twine on writing an evaluation rubric); the same holds for reason codes.

Make the decision auditable against a schema, not just free text. The JSON Schema validation vocabulary defines the structural rules you can assert (required fields, allowed enums, types) so a malformed log row fails loudly instead of drifting (JSON Schema validation). If you validate derived outputs against such a schema, watch for a subtle trap: a schema that is validation-equivalent can still change the behavior of an LLM that reads it, because member order can shift outputs even when the contract is identical (study on JSON Schema instruction artifacts).

Suppose you have 5,000 derived invoice-total tasks. A rebuild reports 200 with checksum mismatches, 4 percent of the set.

The three-way check splits them:

• 150 have a source that changed and a manifest that matches the new checksum, and the change is authorized. Decision: rebuild. These are cheap and mechanical.
• 30 have a source that changed but no matching manifest entry, or the change was not authorized. Decision: escalate, then rebuild only after approval. Do not auto-adopt.
• 15 have a source whose current checksum matches the manifest but not the stored value, and the stored value matches an earlier manifest. This is the stale-checksum case. Decision: repair by recomputing and updating the stored input checksum.
• 5 have a missing or unreadable source. Decision: drop and log source_missing.

That is 150 rebuilds, 15 repairs, 5 drops and 30 escalations. The math matters because it sets expectations: only 5 of 200 mismatches were genuine data loss, and nearly all the rest were the join being out of date rather than data being corrupt. Teams that treat every mismatch as corruption over-rebuild and waste compute; teams that treat every mismatch as stale checksums ship a bad rebuild.

One loop close is worth its effort. If your transport_error count is nonzero, fix the cache or transport before the next rebuild, because that class of mismatch will return. When you need to pull a fresh model or embedding to sanity-check a repaired output, Quanta Inference (https://quantaintelligence.ai/inference) gives you those calls behind one OpenAI-compatible key so you are not stitching providers into the validation step.