When to Replace a Human Coder with a Model: A Decision Rule

Most failed calculations get the human number wrong in the same direction: salary divided by working hours, then stop.

Fully loaded cost per engineering hour is wage plus benefits plus the share of the manager, tooling, recruiting fee amortized over tenure, and the onboarding ramp. A defensible method is to take the team's fully loaded annual cost and divide by hours actually spent producing code, not by 2,080.

Then add coordination cost that never appears on a time sheet. A model that drafts a patch in two minutes and a reviewer who looks at it four hours later has a cycle time of four hours. If delivery depends on same-day merges, the bottleneck moved rather than disappeared.

Two corrections to apply:

• Count audit time as production time. A human writing a function and a human auditing a generated function are both doing the task. If the audit takes 40 percent of the human build time, model-plus-audit cost is model cost plus 0.4 of the human figure.
• Count rework from accepted bad output. Use your defect escape rate, the share of defects found in production versus all defects found across the lifecycle (Opsera on defect escape rate), multiplied by a rough cost per escaped defect. A metric like that is most useful interpreted in context alongside other quality signals rather than as a standalone verdict (Umbrex on defect rate per release).

Three components matter, and only one of them is the one teams measure.

Inference cost per task, not per token. A task that takes three attempts, a retry after a failed test and a long context window costs several times a single successful call. Log spend per task identifier.

The audit. A useful audit is not "read the diff." It is a checklist with a pass condition, and it costs time to run. Time twenty real audits and take the median.

The harness. Tests, fixtures, sandbox environments and CI minutes belong on the model side, because without them the audit is unreliable and the calculation collapses. If you must build a test suite to move the task, that build cost is a first-period cost, not an amortized footnote.

A worked example. Take a task a mid-level engineer does in 6 hours once fully loaded (reading context, writing, self-testing, opening a reviewable change). Measured audit time for a model's output is 1.5 hours. Inference and harness cost works out to 0.4 engineer-hours equivalent.

The ratio is 7.1 / 2.8, about 2.5x. Below the threshold, so the rule says keep a human or shrink the task. Now drop audit time to 1.0 hour once the reviewer has a checklist and a fast suite: the total becomes 2.3 hours and the ratio crosses 3x.

That is the insight the first search result will not give you. The threshold is usually crossed by making the audit cheaper and detection stronger, not by waiting for a better model.

You cannot run the calculation on intuition. Four measurements get you there in a week without a formal study.

• Baseline human time. Time ten real instances from pickup to mergeable change. Use the median.
• Audit time under a checklist. Write the checklist first, then time twenty audits. A checklist without a pass condition is not an audit.
• Detection rate. Of the outputs your audit rejected, how many would the tests have caught? This tells you whether the automated gate or the human is doing the work.
• Escape baseline. Your current defect escape rate in the task area, measured before the switch.

The audit checklist benefits from the same loop human evaluators use. The recommended method is to have two or three experts judge a first group together, writing down decision rules, then iterate: rate independently, compute agreement, discuss disagreements, update the rules until agreement reaches a target such as Krippendorff's alpha above 0.8 (LLM Guidelines for Software Engineering on human validation). A two-person team does not need the statistics, but the loop is exactly what makes an audit reliable.

Agreement metrics exist because raters agree by chance more often than intuition suggests; Cohen's kappa corrects for that (McHugh, the kappa statistic). If two auditors agree 95 percent of the time on a heavily skewed pass/fail split, chance agreement is doing most of the work (Wikipedia on inter-rater reliability). One caution from the same literature: high agreement between automated judges can reflect shared bias rather than correctness, so a sample must still be validated against humans before automated judgment substitutes for human work (LLM Guidelines). A second model is a fast pre-filter, not the audit.

Leads ask the question at the wrong granularity. "Should we use a model for backend work" has no answer. The decision applies per task type.

Two rows deserve comment. The failing-test fix stays with a human not because models cannot patch it but because audit time approaches human time: the reviewer must reconstruct the mental model the model lacked. The migration stays because detection and containment are both weak at once.

This is also the honest answer to the claim that agents make human review redundant for routine changes (Monperrus). The argument is strongest where detection is automated and weakest where the stakes are highest.

For teams generating patches faster than they can review, the fix is a pipeline that separates detection from judgment. Automated reviewers that post severity-ranked inline findings exist, and one vendor's internal testing reported the share of pull requests receiving substantive review comments rising from 16 percent to 54 percent, with fewer than 1 percent of findings marked incorrect (tessl.io on AI code review agents). Read that as a vendor's own testing, not an independent result, and use it for what it is: evidence that a cheap first pass raises coverage before a human spends minutes. Coverage is not correctness, and false assurance is the failure mode to design against (Codacy on the PR bottleneck).

A model update resets your evidence. Behavior changes are not always improvements for your task, and they are not always in the release notes. The model-side cost moves too, since a new version can shift both price and tokens per task.

Re-run this list before widening a task's scope or after any model or provider change.

• The audit sample. Re-run the same twenty audited outputs against the new version and compare rejection reasons. More style rejections and fewer logic rejections means the checklist needs updating.
• Inter-auditor agreement. Recompute agreement on the new sample. Falling agreement means the rules no longer describe the output.
• Detection rate. Recompute how many rejected outputs your tests would have caught. A drop means your automated gate is thinner than you assumed.
• Escape rate. Compare production defects in the task area before and after. A rise means the ratio was optimistic; revert until you know why.
• Cost per task. Re-measure inference spend, retries and harness runtime. A cheaper token price with more retries can cost more.
• Prompt assumptions. Any prompt encoding "the model tends to do X" is an unverified claim. Re-verify the ones the checklist leans on.
• Tool permissions. If the update touches tool use or file access, re-check what the agent can reach. More reach means more places to fail.

Keep the audit in place for at least as long as it takes to gather the new sample. Removing it because an update feels better is how teams find a regression in production.

If you switch between several models or providers, re-checking becomes the dominant cost. One endpoint and one key for text, image, video and embedding models makes the spend comparison mechanical instead of a per-vendor integration project; Quanta Inference is OpenAI-compatible and prepaid in US dollars, so the cost-per-task number in your spreadsheet maps to one invoice.

Four failure modes to know so you do not apply the rule mechanically.

Unmeasured rework. If you cannot estimate the cost of escaped defects in the task area, the ratio is fiction. Use a conservative multiplier over no multiplier.

Audit fatigue. Audit time is not linear. The tenth output in an hour gets less attention than the first. Rotate auditors, cap batch size, keep the checklist short enough to finish in one sitting.

Coordination tax on knowledge. Review also transfers knowledge and builds shared awareness of the codebase, and those goals do not vanish when the writer is a model (Monperrus). If your team learns the system by reading each other's code, an all-model pipeline starves that channel. Schedule design walkthroughs to replace it.

Scope creep after a good week. One clean batch is not a detection rate. The inputs are medians over a sample, so re-scope only when the sample says so. Counting how often a theme appears is useful, but frequency is not importance and the denominator must be reported (skimle on quantifying qualitative data).