Teaching an Agent to Predict Its Own Diff Size

Five required fields, fixed names, no prose inside the payload. Anything the orchestrator cannot parse is a failed preflight, not a best-effort guess.

lines_added plus lines_removed gives the changed-line total your budget compares against. files_touched must equal the length of files; treat a mismatch as malformed and retry once.

Confidence is where preflight earns more than a threshold. A confidence of 1 or 2 on a small task usually means the agent has not read enough code to know what it will touch, so the correct response is to make it investigate and estimate again, not to proceed on a guess. The Karpathy Guidelines skill encodes the same posture for coding: state assumptions explicitly, ask when uncertain, and stop when something is unclear (karpathy-guidelines). The estimate is "think before coding" made mechanical.

The prompt has three jobs: give the agent read access but no write access, describe the task precisely, and demand a schema-shaped answer. Enforce shape at the API layer rather than hoping. Structured output modes exist so a model returns a validated object instead of free text (OpenAI prompt engineering guide).

You are estimating a code change. Do not edit any files.
Read what you need. Return only this JSON object:

{
  "lines_added": <integer>,
  "lines_removed": <integer>,
  "files_touched": <integer>,
  "files": ["<path>", ...],
  "confidence": <1-5>,
  "reasoning": "<max 200 chars>"
}

Task: <task description>
Rules:
- Count every line the patch would add or remove, including tests.
- If unsure which files are involved, drop confidence to 2 or lower.
- List only files that exist or that you would create.

Two details matter more than wording. Allow the agent to count the tests it plans to write, because a feature change with tests can double the line count. And forbid edits at the tool level, not just in the prompt: if a write tool is registered during preflight, the estimate becomes a promise the agent can quietly break. The safety-gates pattern calls this a per-phase tool allowlist, default-deny (Agent Safety Gates).

Run preflight in its own context, not the write context. If the writing agent can see its own estimate it may pad or trim the diff to match, turning a forecast into a self-fulfilling target.

A budget is a pair of numbers per task type, not one global constant. A refactor and a config change carry different risk at the same line count.

The 200 to 400 line range and the under-100 guidance for bug fixes come from the PR-size analysis (Propel Code study). A caution: a study of pull request acceptance found static-analysis quality rules did not influence whether a pull request was accepted, so do not overfit the budget to lint rule counts (Lenarduzzi et al.). Size is a routing signal, not a quality verdict.

On rejection, three responses cover most cases: split cleanly decomposable tasks into sub-tasks and re-estimate each, narrow by trimming files outside the assigned directory, and escalate anything at confidence 1 or 2, or over budget with no clean split.

Worked example. Budget is 200 changed lines and 5 files. The estimate returns 340 lines across 9 files at confidence 2, failing both counts. Split into "add the data model and migration" (estimated 60 lines, 2 files) and "wire the API endpoint and tests" (estimated 210 lines, 4 files). The first passes; the second still fails on lines, so split again into "handler" and "handler tests." All pieces now pass preflight and land as reviewable diffs. Four small estimates replace one rejected 340-line patch.

The orchestrator owns the budget, the loop and the log. Four functions: estimate, compare, dispatch, record.

def preflight(task, budget, agent):
    est = agent.estimate(task)          # structured JSON, read-only tools
    if est["files_touched"] != len(est["files"]):
        est = agent.estimate(task)      # one retry on malformed estimate
    changed = est["lines_added"] + est["lines_removed"]
    verdict = "pass"
    if changed > budget["lines"]:
        verdict = "split_lines"
    elif est["files_touched"] > budget["files"]:
        verdict = "split_files"
    elif est["confidence"] <= 2:
        verdict = "escalate"
    return {"task": task, "estimate": est, "changed": changed, "verdict": verdict}

Run each task in its own worktree so an abandoned attempt leaves the main checkout untouched, and clean stale ones with git worktree prune (git-worktree docs). After the run, compute the actual diff with git diff --numstat, which prints added and removed counts per file and sums without hunk parsing. Capture it before the commit so the numbers match what the reviewer saw.

Each record should carry task id, task type, estimated changed lines and files, confidence, actual added and removed lines, actual files, verdict, and whether the run was split, escalated or shipped. Build two views from those rows: median drift by task type, and the share of runs where actual exceeded the estimate by more than 50%, grouped by confidence. If confident estimates (4 and 5) miss as often as unconfident ones, the score is noise and should not drive routing. Your own log is the only benchmark that speaks to your repository; review automation results vary substantially by task and evaluation setup, and strong benchmark behavior does not prove production readiness (Code Review Research overview).

The subtlest trap: a passing test is not proof the bug is fixed. Analysis of validation events in repair agents found 46.0% of positive comparable events carried no bug-discriminating information, and 23.8% of baseline rollouts closed with a patch whose entire positive evidence was of that kind (Validation Evidence in LLM Repair Agents). The estimate keeps the diff small; pair it with a test-quality gate that asks whether the test fails on the original code.

A second pattern: read the reasoning field for scope creep. If the agent's own note says it "also refactored the helper module while I was there," the surgical-changes rule has been violated and the estimate shows it. Every changed line should trace directly to the request (karpathy-guidelines). A 400-line estimate for a 50-line request is scope creep made visible before it costs a reviewer an afternoon.

For teams that would rather hand the loop to a maintained agent, Quanta builds estimate-and-review layers for coding work; Quanta Experts read your context, do the task and hand back finished deliverables, including preflight-and-review steps.

Once a structured estimate exists, the same three-line decision rule applies to other fields. Compare files against a module ownership map and block edits outside a team's area. Compare lines_removed against a threshold and require human sign-off on deletions, since deleting code whose callers you have not enumerated is a common way a small-looking diff breaks production. Route confidence 1 to a scoping conversation before anything runs.

If you want behavioral rules already written, a community skill such as the Karpathy Guidelines ships as plain Markdown any agent reading the skill format can load, so the same instructions work across models and providers (TypingMind skill listing). Skills that install into a coding agent, including behavioral guidelines like these, are collected at Quanta Skills, so the preflight rule and the surgical-changes rule can ship together as one installable instruction.

Start with one budget number and one field you trust: changed lines. Add files and confidence once you have a week of logs. Then let the numbers decide whether a task runs, splits or goes to a person, instead of finding out from the size of the diff.