A Verification Prompt That Checks Enriched Fields

The verifier is only as good as its inputs. Three components, assembled by your orchestration code and not by the model:

1. The claim. The field name from your enrichment schema, the value the extractor produced, and the exact URL the extractor cited. If the extractor did not cite a URL, there is nothing to verify and the value goes to review.
2. The page text. The raw or lightly cleaned text your fetcher retrieved, plus the fetch timestamp and HTTP status.
3. The rubric. A short, field-specific definition of what counts as support. "An address is supported only if the page lists a street address at or near the company name" is a rubric. "Check the address" is not.

Do not paste the extracted value's surrounding JSON into the page slot. Keep the claim and the evidence in separate, labeled sections of the prompt so the model cannot confuse the thing being tested with the thing testing it. For structured inputs and outputs generally, the mechanics of enforcing a schema are well documented in OpenAI's structured outputs guide and the broader survey of approaches in Agenta's structured output guide.

A minimal request body:

{
  "claim": {
    "field": "company.phone",
    "value": "+1 512 555 0143",
    "cited_url": "https://example.com/contact"
  },
  "source": {
    "url": "https://example.com/contact",
    "http_status": 200,
    "fetched_at": "2025-03-04T10:12:00Z",
    "text": "Head office: 400 Congress Ave, Austin TX. Call us on (512) 555-0143."
  },
  "rubric": "The phone value is supported only if the page shows the same number, allowing differences in formatting and country code. A different number or no number means fail."
}

Three verdicts are the heart of the design. Pass means the page contains a sentence that supports the value. Fail means the page is readable and contradicts it, or plainly does not mention the field. Unclear means the verifier could not tell: the page is a login wall, the content changed, the text is truncated, or two plausible readings exist. Unclear is not a polite fail. It is a signal to a human.

Require an evidence quote for every verdict except unclear, and require the quote to be copied verbatim from the supplied text. That single constraint does most of the work: it turns the verifier from a rater into a witness. If the quote cannot be found in the page text by a substring check, treat the verdict as unclear regardless of what the model said. This is a cheap, deterministic post-check that catches fabricated quotes.

A strict enum is what makes routing safe. If the model can return free text, you end up writing regex over prose. Schema enforcement of enums and required keys is the specific problem structured outputs solve, as OpenAI's documentation and this practitioner write-up on replacing function calling with structured output both explain: without it you get valid-looking JSON with a renamed key or an invented verdict.

Fetch the page yourself and send the text. Sending the URL and asking the model to "look it up" puts the evidence outside your control and outside your logs.

Practical fetch rules:

• Follow redirects and record the final URL. A cited URL that redirects to a parked domain is itself a signal.
• Strip navigation, cookie banners and footers before sending, but keep headings and the body text. Aggressive cleaning that removes a contact block produces false fails.
• Set a character budget and note truncation. If the page exceeds your budget, send the head and the section around any keyword hit, and mark the source as truncated so the verifier knows to prefer unclear over fail when the field is absent.
• Store the fetched text hash and timestamp with the verdict. When a verdict is later questioned, you can show what the verifier read.

Cleaning matters more than people expect. The extraction benchmarking literature is blunt about this: errors introduced at the parsing layer, including in tables and multi-column layouts, propagate directly into the extraction result LlamaIndex's benchmarking guide. If your cleaner mangles a two-column contact table, the verifier will fail a value that was correct, and you will start ignoring the verifier.

Three page types break naive verification. Each has a specific answer.

Long pages. Chunk the text and verify per chunk, but make the verdict "pass if any chunk passes, unclear if no chunk passes and at least one was truncated, fail otherwise." Sending a truncated page and asking for a binary judgment guarantees false fails.

Paywalled or blocked pages. Do not let the verifier guess. Your fetcher should detect login pages and bot blocks and mark the source unavailable, which maps to unclear with reason "source unavailable." Never let a 403 become a fail; the value may be perfectly correct.

Changed pages. A page that no longer mentions the value is not necessarily proof the value was wrong when it was extracted. Compare the fetch timestamp to the extraction timestamp. If the page changed between extraction and verification, downgrade fail to unclear and send both timestamps to the reviewer. This is the same discipline behind keeping evaluator versions and evaluation data visible when comparing results, as Arize recommends.

A small routing table keeps the logic honest:

Before the verifier gates anything, check whether it agrees with a human. Sample 50 to 100 enriched fields, have someone read each cited page and label pass, fail or unclear, then run the verifier blind on the same cases. Do this before you or anyone else has tuned the prompt on those examples, or you will measure memorization.

Two numbers decide whether it is ready:

• Precision on pass. Of the fields the verifier passed, how many did the human also pass? This is your CRM contamination rate. A verifier that passes 98 percent of correct values but also passes 15 percent of wrong ones is not safe as a gate.
• Recall on fail. Of the fields the human failed, how many did the verifier catch? This is how much junk you actually block.

Tune toward precision on pass. A verifier that sends too much to review is annoying; one that passes bad values is worthless. Track disagreement cases in a spreadsheet with the evidence quotes side by side, and when the human and the model keep disagreeing on the same field type, fix the rubric rather than the prompt wording.

Benchmarking practice from the extraction world applies directly: use a task-specific, repeatable ground truth set and rerun it whenever the prompt, the model or the cleaner changes LlamaIndex's guide. The judge is a model too, and judges drift when their inputs change the LLM-as-judge survey covers the meta-evaluation problem at length.

A worked example with numbers. Suppose your last batch had 1,000 enriched records. The hand-labeled set says 80 percent of values are genuinely correct. Your verifier has 95 percent precision on pass and 70 percent recall on fail. Pass 810 records to the CRM, of which about 40 are wrong. Send 190 to review, which is roughly a day of work for one person. If you cut the review queue by loosening the rubric so that 950 pass, the wrong-value count rises unless precision rises with it, so check both numbers together before you change anything. And remember that repeated model calls are not free: the extraction survey notes that iterative prompting improved fidelity but added notable latency and variance the extraction survey.

The routing layer should be boring. Verdict pass plus a verifiable evidence quote equals a CRM write. Everything else lands in a review queue with the claim, the source URL, the fetched text hash, the quote and the confidence. When a reviewer overrides a verdict, log the override; that log becomes your next labeled set.

Two governance points that save pain later:

• Write the verdict and quote alongside the field in your CRM, in a custom property or a linked audit object. When someone asks why a record says what it says, the answer is one click away.
• Keep communication rules aligned. If verified contact data feeds outbound email, the rules that govern that email still apply regardless of how good the data is. Commercial email in the United States is covered by the CAN-SPAM Act, which applies to business-to-business messages and imposes requirements on commercial messages the compliance guide. Verified data and permission are different questions.

If you would rather not run the fetch, prompt and review plumbing yourself, a specialist can build the pipeline with you at Quanta.