How to Log a Rejected Comment So a Model Can Replay It

A record is replayable when a reader with no access to your production systems can determine exactly why the action fired. Every field below earns its place by answering a question an auditor asks.

Three fields get skipped in practice and cause most replay failures. model_params is often left out because settings live in code, but code changes and the record does not, so capture effective values per call. prompt_template_hash catches the quiet case where someone edits instruction text and scores drift with no version bump. normalizer_version matters more than it looks, since a change in how you strip zero-width characters can flip a block, and without the version you cannot explain the shift the unidata overview of moderation workflows.

A practical filter: if a field would not change the action, the reason code, or your ability to explain either, leave it out. Audit tables die from bloat as often as from gaps.

Ordering is a design decision. Audit logging guidance is consistent that logs should answer who, what, when and the outcome, be protected from tampering, and be reviewed rather than merely accumulated the Fortra audit log best practices guide. For moderation, the strict version is: persist durably, then return the decision to the caller.

That adds a write to the critical path of a user-facing action. Three patterns make it tolerable.

1. Synchronous, same transaction. Write the decision row and the outcome in one transaction if both live in one store. Simplest to reason about, and correct.
2. Synchronous append, asynchronous enrich. Write the minimum replayable fields inline, then attach reviewer notes and appeal outcomes later as rows keyed by the same decision_id.
3. Write-ahead to a queue. Append to a durable queue with an acknowledgment before returning, then materialize into the queryable table.

Design out the inverse order. Return block, then log, and a process death between the two leaves a user harmed by an action with no evidence, which is exactly what an audit trail exists to prevent.

# Pseudocode: persist before returning.
def moderate(comment, rubric, model):
    received_at = utcnow()
    normalized = normalize(comment.text)       # versioned normalizer
    output = model.score(normalized)           # raw response kept verbatim
    scores = parse(output)
    action, reason = apply_thresholds(scores, rubric)

    record = {
        "decision_id": uuid7(),
        "trace_id": current_trace_id(),
        "received_at": received_at,
        "decided_at": utcnow(),
        "input_text": comment.text,
        "input_hash": sha256(comment.text),
        "normalized_text": normalized,
        "normalizer_version": NORMALIZER_VERSION,
        "rubric_id": rubric.id,
        "rubric_version": rubric.version,
        "rubric_hash": rubric.hash,
        "model_name": model.name,
        "model_params": model.effective_params(),
        "prompt_template_hash": model.prompt_hash,
        "raw_output": output,
        "scores": scores,
        "threshold_applied": rubric.threshold_for(action),
        "final_action": action,
        "reason_code": reason.code,
        "decided_by": "model",
        "log_schema_version": "3",
    }
    decision_log.append_durable(record)        # must not fail silently
    return action, reason

Note the one subtle line: threshold_applied is read from the rubric object actually used, not from a live config service. If config is your source of truth, resolve and store the value anyway.

An unversioned rubric can be argued about but not audited. Guidance on rubric design stresses that criteria should be clear, tied to the standards being assessed, and paired with descriptors that separate levels rather than vague adjectives Appalachian State's guide to creating rubrics. Apply that discipline to moderation.

• Store each rubric as an immutable document with rubric_id, a monotonic rubric_version and a content hash. Publish new versions; never edit published rows.
• Make each criterion machine-addressable with a stable criterion_key, a scale, level descriptors and a threshold per action. Free-text criteria cannot be re-scored reliably.
• Record the hash, not only the label. Labels get reused by mistake during migrations.
• Keep a change log with author, rationale and effective date, so a reviewer knows which rubric governed an old comment.

The insight teams miss: rubric wording and ordering are not cosmetic. Experimental work found that the order in which performance levels are presented changes how readers allocate attention across levels and how they perform the Panadero et al. study on rubric level order. Reordering severity levels can shift human override rates, which is precisely the change your log must be able to isolate, and that is only possible if the rubric version is recorded per decision.

If an LLM judge also scores comments, version its evaluation prompt and criteria the same way. Rubric-based judging is a standard way to assess open-ended outputs, and practitioner guides are explicit that judge quality depends on prompt, model and task, with known biases such as verbosity and self-preference the Evidently AI guide to LLM-as-a-judge. Logging prompt_template_hash beside rubric_hash lets you separate "rubric changed" from "judge prompt changed" when block rates move.

Keep the write path narrow and shape the read path for the three queries that dominate: fetch one decision by ID, list decisions for a user, and compute action rates by rubric version and day.

A workable relational arrangement:

• moderation_decisions: the append-only table above, partitioned by decided_at month, indexed on (decision_id), (actor_id, decided_at), (rubric_id, rubric_version, decided_at) and (final_action, decided_at).
• moderation_overrides: one row per human action, linked by decision_id, carrying reviewer_id, new_action and override_reason_code.
• rubrics: immutable documents keyed by (rubric_id, rubric_version), with hash and effective dates.
• models: append-only registry of model names, parameter sets and prompt hashes.

Store raw input text in the decision row if it is small. If comments can be long or carry attachments, keep a sha256 hash plus a pointer to immutable object storage, and retain inline only the normalized text the model saw (with the truncation rule and character count, if you truncated). This keeps rows partition-friendly while still allowing exact replay by dereferencing the pointer.

-- Replay every block in a window, with the rubric and model behind it
SELECT d.decision_id, d.decided_at, d.final_action, d.reason_code,
       d.rubric_version, d.rubric_hash, d.model_name,
       d.threshold_applied, d.scores, o.reviewer_id, o.new_action
FROM moderation_decisions d
LEFT JOIN moderation_overrides o ON o.decision_id = d.decision_id
WHERE d.decided_at >= :window_start
  AND d.decided_at <  :window_end
  AND d.final_action = 'block'
ORDER BY d.decided_at;

Two rules follow from replay requirements. The decision table is append-only, so overrides are new rows and the original call stays visible. Delete and update rights on that table belong to a break-glass role, with write-once storage for the archive tier, since record integrity is a named best practice the Confluent Cloud audit logging best practices.

Most "we cannot replay this" incidents are not missing data, they are missing joins: the decision was logged but the rubric document was edited in place, or model settings lived in a deployed config that has moved on. Immutability plus hash references turns those incidents into one join.

Retention fails in two directions: purged too early to defend an old decision, or kept forever at full queryable cost. Use two clocks.

Clock one, rubric life. Keep records for as long as the rubric they reference is in force. Deleting them while the rubric is live makes a current rule unauditable for the oldest comments it still governs.

Clock two, a fixed audit window after retirement. Once a version is retired, keep its decisions for a fixed period measured from retirement, not from the decision date. A 13 month window is a defensible default because it covers a full annual review cycle plus the month findings are written up. Frameworks often demand more: retention guidance commonly cites seven years for SOX, six for HIPAA and one year for PCI DSS, and notes that internal investigation needs may exceed all of them the Confluent Cloud audit logging best practices. Treat 90 days as a floor for operational review, not a target the Fortra audit log best practices guide.

Two guardrails. A legal hold flag suspends purge for a specific actor_id or decision_id without duplicating rows. And override retention must never be shorter than decision retention, or replay shows a block with an invisible appeal.

That last row does the most work. A quarterly replay of a random sample catches schema drift, missing rubric documents and broken joins before an appellant or regulator finds them.

Assume a community platform receiving 400,000 comments a day, with 8 percent blocked and 12 percent warned, and a mean record of about 4 KB before compression.

• Records per day: 400,000
• Raw size per day: 400,000 x 4 KB = 1.6 GB
• Raw size per year: about 584 GB
• At 4x compression in columnar cold storage: roughly 146 GB per year

At that scale, storing every record for the full window is affordable. The rule flips for high-volume, low-risk channels such as reactions or chat. Log every warn and block in full, and log passes as a compact row with decision_id, actor_id, rubric_hash, scores, threshold_applied and input_hash, dropping input_text. You can still prove the pass was correctly classified without paying for text on the vast majority of rows.

Now a question the log makes answerable. Suppose the block threshold is a score of 0.80 and warn is 0.50. Review finds 6 percent of decisions land between 0.78 and 0.82, and humans override 40 percent of those blocks to warn. Logged as above, that is one query joining scores and threshold_applied to overrides. Without verbatim scores you would be guessing.

The fix is a policy choice, but the log tells you where to look: raise the block threshold, or add a criterion that separates the ambiguous band. Either way you publish a new rubric_version, old decisions stay attached to the old one, and you can measure the effect instead of debating it.

This is the same principle that makes rubric-based judging useful at all: explicit criteria at defined levels let you diagnose quality rather than stare at one number, and they need calibration against human judgment to stay honest the Medium analysis of rubric-based evaluations. Your decision log is that calibration dataset.

Two decisions remain: where scoring calls run, and who owns the reviewer tooling.

Moderation pipelines increasingly route text, image and audio scoring through several providers, each with its own SDK, keys and billing. One OpenAI-compatible endpoint for text, image, audio and embedding models removes per-provider plumbing, and prepaid US-dollar pricing keeps cost visible against per-decision volume. Quanta Inference is built for that pattern, and if you want the reviewer console and replay tooling built around your own schema rather than bolted on, a specialist can take it on with you at Quanta.

Resist building the reviewer UI before the schema. The schema is the contract: once decision_id, rubric_hash and threshold_applied are stable, dashboards and appeal tools can sit on top and each can prove its actions are traceable. Teams that build the UI first discover months later that their rejected state stores a boolean and no reason, and that the rubric justifying it was edited in place by someone who has left.

Write the replay runbook while the system is small: how to fetch a decision, how to find its rubric version, how to re-score the stored text by hand, and who signs off when a replay disagrees with the original. A log nobody knows how to read is just storage.