Generating the Fact List Automatically Before a Coverage Check

An atomic fact is the smallest independently checkable assertion a query requires. "The article explains how to handle shard allocation failures" is not atomic. "This symptom indicates this cause," "this setting must be applied before restart," and "this workaround does not apply to this deployment type" are atomic, and that decomposition is exactly what Elastic's verification pass does with each source article (Elastic). Research on atomic fact extraction and verification reaches the same conclusion from the other end: decomposing a complex claim into atomic facts enables fine-grained retrieval and reduces accumulated reasoning errors, outperforming static decomposition on five benchmark datasets (Fact in Fragments, arXiv).

Deciding what counts as atomic is a product decision, not a linguistic one, so write it down before prompting. Use this checklist:

Two consequences follow. First, a fact should never reference the query ("the answer mentions the first step"), only the world ("the first step is to stop the writer"). Second, bundle-free facts give you a clean per-fact score: coverage becomes the fraction of facts found in the answer, which is easy to average across queries and easy to eyeball in a review.

Model output must be parseable before it can be validated, so define the schema first. JSON with a fixed set of fields works because it removes the formatting variance that otherwise leaks into your metrics.

{
  "query_id": "q_1042",
  "facts": [
    {
      "fact_id": "q_1042_f1",
      "statement": "The retry limit applies to write requests only.",
      "type": "constraint",
      "required": true,
      "source_hint": "policy section"
    }
  ],
  "notes": "Empty if all facts are unambiguous."
}

The type field earns its place during validation: you can report precision separately for constraints and for steps, and you will usually find the two differ. Keep the allowed values short and closed (for example constraint, step, quantity, definition, exception). source_hint is optional and exists only to help a human reviewer find the relevant section quickly.

A prompt that works in practice has four parts: the role (extract required facts, do not answer the query), the schema with one filled example, the atomicity rules from the checklist, and an instruction to return an empty list rather than speculate when the query is under-specified. The last instruction matters more than it looks. Under-specified queries are exactly where a model invents plausible facts, and those inventions are the hardest errors to spot downstream.

Treat the extraction prompt like code. Version it, store the prompt version with each generated list, and re-run the validation set whenever the prompt changes. Without that version tag, a precision drop after an edit becomes an argument rather than a measurement.

Two models, or the same model twice, will emit "the timeout is 30 seconds," "the request times out after 30s," and "requests time out at the 30-second mark." Left alone, three duplicates turn one required fact into three coverage checks, and the answer that covers it scores as a third covered rather than fully covered. Normalize before you deduplicate.

A practical normalization pass, in order:

1. Strip hedging and meta-language ("the answer should mention," "note that," "it is important to").
2. Expand abbreviations that appear in the corpus, and keep a small domain glossary so the expansion is consistent.
3. Lowercase, collapse whitespace and remove trailing punctuation.
4. Convert numbers to a single form, then re-render them in the corpus's units.
5. Cluster near-duplicates with an embedding similarity threshold, then let a model confirm each cluster and pick one canonical statement.

Step 5 is the only one that needs judgment, and it is the one to log. Keep the cluster members alongside the canonical fact so a reviewer can see what was merged. If a merge is wrong, you can split it later without regenerating the whole list.

Deduplication also protects the coverage metric from a subtle bias. If verbose answers attract paraphrase-heavy fact lists, and paraphrases cluster into fewer unique facts, then verbose queries gain an easier bar. Collapsing duplicates to one canonical fact per requirement keeps the bar level across the set.

The validation set is small on purpose: 30 to 50 queries is enough to estimate whether the generator is fit to use, and small enough that two people can label it in an afternoon. Stratify the sample by query type (simple, multi-hop, under-specified) so the estimate does not hide a weak slice. Save the human list as ground truth in the same schema, with the same fact_id convention.

Score two numbers, and treat them differently:

• Precision: of the generated facts marked required, what share appears in the human list? A low precision means invented or over-specified facts, and those produce false "missing" judgments on correct answers.
• Recall: of the human facts, what share appears in the generated list? A low recall means the checker will pass answers that omit a required fact.

Deciding which to trade is the key judgment. A false positive (a hallucinated fact) makes good answers look bad and erodes trust in the checker, while a false negative (a dropped fact) makes bad answers look good, which is the failure that reaches users. If you must choose, protect recall on high-stakes queries and protect precision where reviewers will act on every flag. When you cannot choose, use the F1 score, which balances both. The information retrieval literature formalizes exactly this: precision@k measures whether the top results are relevant, recall@k measures whether the relevant set was captured, and F1 combines them (Towards Data Science on precision@k, recall@k and F1).

A worked example makes the decision concrete. Take 40 labeled queries with 6 human facts each, 240 facts in total. The generator produces 250 facts. Matching against the human lists finds 200 true facts and 50 invented ones, so precision is 200/250 = 0.80. Recall is 200/240 = 0.83, because 40 human facts never made it into the generated lists. The harmonic mean, F1, is about 0.815. Interpretation: roughly one generated fact in five is noise, and the checker will miss about one requirement in six. That is probably not good enough for a customer-facing gate. It may be good enough for a triage queue where a human reviews every flag. The numbers let you make that call instead of guessing.

A second worked number is worth computing. Suppose a mislabeled requirement flips the verdict on one answer in ten. At 0.80 precision and 0.83 recall on 6 facts per query, expect roughly one bad fact per query (5 invented and about 1 dropped), so the flip rate is driven by the facts, not by the answer. Fix the list before you lower the coverage threshold, or you will spend weeks tuning a checker that is measuring the wrong requirements. Evidently's guide makes the same point about test design: high-quality test cases come before metric choice (Evidently AI).

Then look at where the errors concentrate. If most false negatives come from exception facts, add an explicit instruction and one worked example for exceptions, re-run the prompt, and re-score the same sample. Iterate on the prompt, not the model, until the numbers move. Labeling part of the sample twice with two annotators also gives you an agreement figure, which tells you how much of the remaining error is the generator and how much is the rubric itself.

A validated pipeline still drifts, because queries, documents and models change. Two mechanisms keep it honest.

First, put a gate in continuous integration. Run the extraction on the validation set whenever the prompt, the schema or the model changes, and block the change if precision or recall drops below its floor. This is standard practice on the retrieval side: run evaluations in CI and block deploys that regress retrieval metrics, so quality drops are caught before users see them (DataAIHub). The same gate works for the fact list, and it costs a few dozen model calls per run.

Second, sample in production. Send a small random share of generated fact lists to a human reviewer each week and log every disagreement. That produces a rolling precision estimate and a growing set of hard cases you can promote into the validation set. Because generation and verification are separate passes, you can also check the checker: re-score a fixed answer set with the same fact list and confirm the scores are stable.

Two failure modes deserve their own alarms. Watch for queries that generate unusually long fact lists; they tend to signal a vague query where the model padded the list. And watch for facts that never match any answer, across every query of their type, because that pattern usually means the fact is uncheckable as written and needs rewriting.

Pick the gate thresholds by use case, not by ambition. A publishing workflow that blocks an article until every fact is covered needs recall near the top of your achievable range, because a dropped fact ships a gap. A dashboard that ranks answers by coverage needs precision, because noisy facts reorder the list and reviewers stop reading it. When the two diverge, split the list: gate on the hard requirement types and report the soft ones as advisory.

Teams that run several models in the pipeline (text for extraction, embeddings for deduplication, occasionally vision when queries reference screenshots) can route them through one OpenAI-compatible endpoint and keep key management and cost accounting in one place with Quanta Inference. If you would rather have the whole loop, schema, prompt, validator and gate, built with you as a finished deliverable, describe it to Quanta and a specialist picks it up.

Here is the whole loop on a support knowledge base, written as steps you can copy:

1. Freeze the schema. One JSON shape, five allowed fact types, required boolean, optional source hint. Store it next to the prompt in version control.
2. Label the validation set. Pick 40 real queries, stratified by type. Two people write human fact lists in the same schema, then reconcile disagreements and record the rubric change that settled each one.
3. Prompt for extraction. Role, schema with one filled example, atomicity checklist, instruction to return an empty list rather than speculate. Run it at temperature 0 so re-runs are comparable.
4. Normalize and deduplicate. Apply the five-step normalization, cluster with embeddings, confirm clusters with a model, and keep the cluster members in the log.
5. Score precision, recall and F1. Match generated facts to human facts, first automatically by normalized text, then by a model judge for paraphrases, with a human spot check on the judged matches.
6. Diagnose by type and query class. Report precision and recall per fact type. Fix the weakest type with a targeted instruction and example.
7. Gate and monitor. Put the score run in CI with floors, and sample production lists weekly.

The loop converges quickly in practice, usually in two or three prompt revisions, because the errors cluster by fact type rather than spreading evenly. The output is an input to the coverage check that you can defend: it has a known precision and recall on a sample you labeled, a version, and a gate that stops it degrading.

Treat the labeled sample as the lasting asset. Prompts and schemas will change, and models with more capacity will come along, but the human list stays the constant against which every change is measured. When the next generation of models arrives, you do not need a new methodology, only a re-run of step 5. If your team has never built an evaluation set before, the practical walkthrough in this evaluation course video covers the same ground for RAG systems and is a useful primer for whoever writes the human lists (Evaluating a RAG system: methods and metrics).