Provenance Fields That Survive a Regeneration

The contract has room for more, but these four are the minimum that survives a rebuild:

A few design notes on each.

generator_version should identify the whole generation recipe, not just a model name. If your contract includes a prompt template, a decoding configuration and a post-processing step, hash or version the combination. When someone later asks why the summaries for slice 7 got shorter, you want a single identifier that pins the recipe. This mirrors why golden datasets exist: a small, trusted reference set lets you compare model versions under identical conditions as DAC.digital describes. Provenance fields turn every task into a comparable record.

source_id is the document or record identity. It should be stable across edits, so a document that gets a new paragraph keeps its identity. That is the whole point of pairing it with a checksum.

slice_id is the region of the source used for the task, whether that is a paragraph, a page, a row range or a token window. It must be deterministic: given the same source version, the same slice must get the same identifier. If your slicer is chunking text with overlap, slice_id should encode the start offset and length so two runs agree.

source_checksum is the digest of the exact slice bytes. The FAIR Cookbook recipe on creating checksum files makes the underlying idea plain: a checksum is a short proxy for the content, and comparing checksums on both sides tells you whether the files are identical within the limits of the algorithm per the FAIR Cookbook. Here, "both sides" are the task record and the current source.

One field you might be tempted to add is a timestamp of generation. Keep it, but understand that a timestamp tells you when, not what. It will not survive a rebuild as evidence of which bytes were used.

The single most common failure is writing provenance after the output. It happens for a boring reason: the generation call is slow and the output row is written first so the UI can show progress. Then a second update sets the source fields. If the process dies between the two writes, or if a retry overwrites the row, you get a task with no provenance and no way to reconstruct it.

Write everything in one transaction. Here is a minimal shape in Python with a SQL insert:

import hashlib

def make_task(conn, source_id, slice_id, slice_bytes, generator_version, output):
    checksum = hashlib.sha256(slice_bytes).hexdigest()
    with conn.transaction():
        conn.execute(
            """
            INSERT INTO tasks
              (task_id, source_id, slice_id, source_checksum,
               generator_version, output, created_at)
            VALUES (%s, %s, %s, %s, %s, %s, now())
            ON CONFLICT (task_id) DO NOTHING
            """,
            (task_id, source_id, slice_id, checksum,
             generator_version, output),
        )
    return checksum

Three details make this reliable. First, the checksum is computed from the bytes that were actually sent to the generator, not from a re-read of the file after the fact. If your slicer reads from a database, hash the string you passed in. Second, the insert is idempotent on task_id so a retry cannot create a second row with different provenance. Third, the transaction wraps the output write and the provenance write together. Either both land or neither does.

The same discipline applies if the generator call itself is external and can fail. Structure the work as: resolve slice bytes, compute checksum, call the generator, then one transaction to insert. Never insert an output placeholder before the call and fill in provenance later.

A schema validator helps here. JSON Schema validation publications define how to describe required fields and constraints for JSON documents per the JSON Schema validation vocabulary. If your generation contract is expressed as JSON, make source_id, slice_id, source_checksum and generator_version required properties. That turns a missing field into a validation failure at write time rather than a missing answer at audit time.

One caveat on the checksum. Hash only the slice bytes. If you hash the whole document, then every edit anywhere in the file invalidates every task derived from it, and your audit query returns nothing useful. The FAIR Cookbook example computes a digest of one file at a time for the same reason: the unit of comparison should be the unit of identity.

The algorithm matters less than consistency. What matters is that the value stored in source_checksum is computed by the same function, over the same byte representation, every time.

Two decisions to pin down in the contract:

Encoding. If your slices are text, hash the UTF-8 bytes, not the Unicode string object. Python's str hashing depends on the interpreter and is randomized per process unless you set a seed, so use encode("utf-8") before hashing. If your slices are binary, hash the bytes directly.

Normalization. Decide whether trailing whitespace, line endings and Unicode normalization form count as part of the slice. They do if you hash the raw bytes, and that is usually the right choice because it is the most literal record of what the generator saw. If you normalize, document the normalization in the contract and apply it identically at write and at audit time. A mismatch in normalization settings between the writer and the auditor produces false "changed" verdicts, which are as harmful as missed ones.

For new work, SHA-256 is a reasonable default: it is available in every standard library, produces a hex digest that fits in a normal string column, and has no practical collision concerns for this use. Whichever algorithm you choose, store the algorithm name alongside the digest, either as a separate column or as a prefix like sha256:. That way a future migration from one algorithm to another does not require rewriting history before you can compare anything.

A small helper that both the writer and the auditor import is the cleanest way to guarantee agreement:

import hashlib

def slice_digest(raw_bytes: bytes, algo: str = "sha256") -> str:
    h = hashlib.new(algo)
    h.update(raw_bytes)
    return f"{algo}:{h.hexdigest()}"

If the writer calls this and the auditor calls this, a mismatch means the bytes changed, not that the tooling disagrees.

A rebuild is the moment provenance earns its keep. Suppose the source documents were re-sliced with a new chunker, or the generator version changed, or both. You now have an old set of tasks and a new set. The audit question is: for a given source, which tasks came from the current slice and which came from a superseded one?

The query is a join on the slice identity, filtered on the checksum:

SELECT
  t.task_id,
  t.generator_version,
  t.source_checksum       AS task_checksum,
  s.current_checksum,
  CASE
    WHEN t.source_checksum = s.current_checksum THEN 'current'
    ELSE 'stale'
  END                     AS status
FROM tasks t
JOIN source_slices s
  ON s.source_id = t.source_id
 AND s.slice_id  = t.slice_id
WHERE t.source_id = :source_id
ORDER BY t.slice_id, t.task_id;

This returns every task for one source, labeled by whether its checksum matches the current slice. The join key is (source_id, slice_id), not the checksum. That is deliberate: if you joined on the checksum, a task built from a stale slice would simply drop out of the result set and you would never see that it was stale. The checksum is a filter and a label, not a join key.

Watch for slices that no longer exist. If the new chunker produced different boundaries, an old slice_id may have no row in source_slices at all. Use a left join to surface those as orphans:

SELECT t.*, s.current_checksum
FROM tasks t
LEFT JOIN source_slices s
  ON s.source_id = t.source_id
 AND s.slice_id  = t.slice_id
WHERE t.source_id = :source_id
  AND s.slice_id IS NULL;

Every row this returns is a task whose slice identity is gone. In a rebuild, those are the tasks you must either regenerate or retire.

Treat the audit as an evaluation step, and give it the same care you give any measurement. Evaluation in the general sense is a systematic determination of merit against explicit criteria per the American Evaluation Association's framing. A stub evaluation of "does the provenance match" is worth only as much as its criteria are explicit. Define what counts as current, what counts as stale, and what counts as orphaned before you run the query.

The inverse question is just as common: a single slice turns out to be wrong, and you need every task that used it. Here the query is narrower:

SELECT task_id, generator_version, created_at
FROM tasks
WHERE source_id = :source_id
  AND slice_id  = :slice_id
ORDER BY created_at;

Run this before you regenerate anything. The result set is your blast radius: every downstream artifact that needs review. If your tasks feed further stages (summaries of summaries, extracted entities rolled up per document), you also need to know which of those stages consumed the affected tasks. That is classic lineage, and provenance gives you the starting point: from the slice, find the tasks; from the tasks, walk forward through your pipeline's dependency records.

A worked example makes the numbers concrete. Suppose one source document is split into 600 slices, and each slice produces three tasks (a summary, a question pair and a keyword list), for 1,800 tasks total. A source correction affects 12 slices. The slice query returns 12 slice rows and 36 tasks. Without the (source_id, slice_id) index, that lookup is a full scan of the task table; with it, the query touches 36 rows. If your task table holds a million rows, the difference between those two plans is the difference between an incident review and a coffee break.

Two query patterns worth adding to your tooling:

Query volume is a real cost. If you keep task-level rows for every generation attempt, the audit table grows quickly, and you may want sampling for the log stream. Log sampling keeps a representative share of events while keeping visibility into patterns that matter per groundcover's guide. Provenance rows are not logs, though, and should not be sampled: they are the record that answers "did this task use the current bytes," and a sampled record cannot answer that for the unsampled rows.

The four fields pay off across generator versions, which matters more as teams route work across several model families. If one task was produced by a generator version built on one model and another by a version built on a different one, generator_version is the handle that lets you compare them without guessing.

A sensible rule of thumb: if a task's provenance record is complete, you can attribute any quality change to the source, the slice boundaries or the generator recipe. If it is incomplete, you are reduced to regenerating and comparing outputs, which confounds all three at once. That is exactly the trap the golden dataset literature warns about: without a stable reference, you cannot tell whether the model improved or the input changed as DAC.digital notes on benchmark drift.

For teams running generation across more than one provider, Quanta Inference offers a single API key for text, image, video, audio and embedding models with an OpenAI-compatible interface. The provenance discipline does not change: record which generator version produced each task, regardless of which model sat behind it.

The audit itself is a good candidate for a small, opinionated script that runs after every rebuild and emits a one-line summary per source: how many tasks are current, stale or orphaned. That summary is the artifact you show when someone asks whether the index can be trusted.