Reading a Context File Aloud to a Cheap Model

Smaller models need explicit context and precise output formats, and you should expect to parse their output rather than assume it conforms (Practical prompt engineering for smaller LLMs, web.dev). Contradiction flagging is a short classification task, so it fits that constraint well.

It also has a shape small models handle badly. Most models degrade after a certain context size, and performance depends on where information sits in the window (Long Context RAG Performance of LLMs, Databricks). Self-consistency sampling, which helps on short inputs, fails and can hurt on long-context tasks (How Effective Is Self-Consistency for Long-Context Problems, arXiv). Never hand the model the prompt and the whole file together.

Split both documents into addressable units first. Give each line or short block a stable ID, pass the prompt once as fixed reference text, then send one context-file section per request and ask for a verdict on that section only. Each request stays short, output maps back to a location, and you dodge the failure where the middle of a long input gets ignored.

That table is the decision aid. If your file leans on the top four rows, automation pays. If it leans on the bottom four, automation gives you a false sense of coverage.

The prompt assumes you already split the file and pass a section ID with its text. Keep the system prompt verbatim and complete; it is the model's only source of constraints. Use strict verdict vocabulary so results count mechanically.

You are a contradiction checker. You compare one section of a context
file against a system prompt. You do not rewrite or judge style.

SYSTEM PROMPT (authoritative, do not modify):
<system_prompt>{{SYSTEM_PROMPT}}</system_prompt>

CONTEXT FILE SECTION:
<section_id>{{SECTION_ID}}</section_id>
<section_text>{{SECTION_TEXT}}</section_text>

A contradiction means the section cannot be true at the same time as a
statement or constraint in the system prompt. Different wording is not
a contradiction. Asking for more detail is not a contradiction. A
section silent about a prompt rule is not a contradiction.

Rules:
1. Judge only this section against the system prompt.
2. Use no knowledge outside these two texts.
3. If the section conflicts with a rule, quote the exact prompt sentence.
4. If unsure, return verdict "unclear" rather than "contradiction".
5. Do not flag a section for being long, vague, or off-topic.

Return JSON only:
{"section_id": "...", "verdict": "contradiction" | "none" | "unclear",
 "prompt_quote": "...", "section_quote": "...",
 "category": "role_permission" | "format" | "fact" | "date" | "numeric"
            | "cross_section" | "silence" | "other",
 "confidence": 0.0-1.0}

Three details carry the weight. The "unclear" verdict gives the model a legal way out, which cuts manufactured conflicts. Quoting both sides forces it to point at evidence, which speeds your human review. The category field lets you measure misses by type.

Small models vary in how well they honor a JSON contract, so parse defensively and treat malformed output as a non-answer, not a pass (Practical prompt engineering for smaller LLMs, web.dev). If you run this often, watch input tokens per call, since prompt growth is easy to miss as template edits pile up (How to track LLM token usage, Braintrust).

The bar needs two numbers and one process. On 40 hand-labeled planted contradictions, recall (flagged or marked unclear) decides whether the tool is worth running. On clean sections, false positives per 1,000 lines decide whether a reviewer can stand reading the output.

A reasonable starting bar for a small model is recall of at least 25 of 40, with no more than 15 flags per 1,000 clean lines. That allows misses because flags go to a person, so a miss becomes an ordinary manual discovery; a flood of false positives burns trust and gets the tool switched off.

Build the labeled set deliberately. Take a real file, plant contradictions in the categories above at roughly five per category, and label each location and the prompt sentence it conflicts with. Without labels you have anecdotes, and the point of an eval is evidence over assumption (What is prompt evaluation, Braintrust).

Track recall by category, not just overall. A file scoring 70 percent overall but 0 percent on numeric contradictions tells you exactly where the human must look, which beats a single number.

Take a 120-line context file and a 30-line prompt stating four constraints: reply in JSON only, never modify the billing table, the reporting API base is https://reports.internal.example, and the freeze runs March 1 to March 15. Plant eight contradictions.

1. Line 12 lets the agent remove stale billing rows. Role versus permission.
2. Line 27 asks for prose before the JSON block. Format.
3. Line 41 gives a different reporting host. Fact.
4. Line 55 says the freeze ends March 10. Date conflict.
5. Line 63 says write permissions follow an external release process. Silence.
6. Line 74 gives 3 retries per job, line 98 gives 5. Cross-section numeric.
7. Line 88 sets the credit cap at 40, line 104 at 60. Numeric.
8. Line 110 claims 80 GB total while listing 45 GB and 50 GB. Arithmetic.

Expect a small model to flag items 1 through 4, catch 5 only if you pass the process text alongside, and miss 6 through 8. Catching 5 of 8 with 2 clean-section flags is 62.5 percent recall at the bar, with about 17 false positives per 1,000 lines, just over budget. The fix is rarely a bigger model; split numeric checks into a script.

The arithmetic matters more than the guess: 40 plus 60 equals 100, not 80, and 45 plus 50 equals 95. No model call is needed for those, and a five-line assertion does it better.

import re

def numeric_conflicts(text):
    flags, seen = [], {}
    pat = re.compile(r"(?P<key>[a-z ]{3,30}) is (?P<n>\d+)")
    for line_no, line in enumerate(text.splitlines(), 1):
        for m in pat.finditer(line):
            key, n = m.group("key").strip(), int(m.group("n"))
            if key in seen and seen[key][0] != n:
                flags.append((seen[key][1], line_no, key, seen[key][0], n))
            seen.setdefault(key, (n, line_no))
    return flags

Run that before the model pass and you convert a category the model misses into one it never sees. Models for meaning, regular expressions for numbers, people for judgment.

Each miss has a mechanical cause, and the cause tells you whether to fix the prompt, add a script, or route to a person.

Arithmetic. The model reads three unremarkable phrases and does not simulate the sum. Small models are unreliable calculators. Fix with a script.

Cross-section inference. A permission granted in one line and revoked in another is consistent with the prompt line by line. Section-at-a-time input removes any chance to join them (Long Context RAG Performance of LLMs, Databricks). Fix by pairing sections that share a key and asking a second question about the pair.

Arguments from silence. There is no sentence to quote, and the prompt tells the model not to flag silence, so it complies. Fix with a second pass that asks which prompt constraint, if any, the section violates.

Buried references. A line pointing at a document the model has not seen cannot be checked. Resolve references first, or flag every unresolved one for a human.

Position effects. Where a conflicting sentence sits changes whether it is used, and self-consistency sampling does not repair this (How Effective Is Self-Consistency for Long-Context Problems, arXiv). Keep inputs short.

Judge biases. Position bias, verbosity bias and self-preference are catalogued failure modes of model-as-judge setups (Evaluation-Driven Iteration for LLM Applications, arXiv). Measure them by re-running your label set after every prompt edit. One study saw extraction pass rates fall from 100 to 90 percent when generic rules replaced task-specific prompts, so treat your checker prompt as versioned (Evaluation-Driven Iteration for LLM Applications, arXiv).

Output should be a table sorted by section ID, showing only contradiction and unclear rows with the exact prompt quote. The reviewer sees ten to twenty rows, each a conflicting pair, not the original file.

Two operational notes. Measure cost per run in tokens and check whether input or output dominates, since that tells you whether to trim sections or the prompt (How to track LLM token usage, Braintrust). Keep the run reportable, with recall on the forty-item set, false positives per 1,000 lines, unclear rate and parse failures, so a prompt change can be judged rather than assumed.

Run the check on every change to the prompt or file, the way a pull request triggers review. Where those files live in a repository, ownership conventions are the hook: owners get pulled in on changes to code they own, and the same logic applies to prompt artifacts you treat as owned (About code owners, GitHub).

If you would rather point one harness at many models instead of maintaining a client per provider, Quanta Inference offers text models through one OpenAI-compatible API (https://quantaintelligence.ai/inference). That is harness convenience, not a requirement for the technique.