A Review Checklist for Context File Changes

The first question has a mechanical answer, so make it mechanical. A named team owns the context file, and the repository enforces that ownership.

GitHub's CODEOWNERS file does this directly. Code owners are automatically requested for review when a pull request modifies files they own, and administrators can require approval from a code owner before a pull request can merge (GitHub code owners). The file can live in .github/, the repository root, or docs/, and must be on the base branch of the pull request for reviewers to be requested (GitHub code owners).

A minimal rule for a context file looks like this:

# .github/CODEOWNERS
AGENTS.md        @org/agent-platform
CLAUDE.md        @org/agent-platform
docs/context/    @org/agent-platform

Two details worth knowing. If a team is set as a code owner, enabling auto assignment can replace the team with a rotating individual, but that replacement does not happen when a branch protection rule requires review from code owners (GitHub code review settings). And a "request changes" review is informational unless a ruleset or branch protection rule makes it blocking (GitHub pull request reviews). If you want the owner's rejection to actually stop a merge, configure the protection, do not rely on convention.

The reviewer's job in this step is one sentence long: the approval in the pull request must come from someone in the owning group, and that person can explain why the file exists. If nobody can, the edit should not merge until ownership is assigned.

Reviewers cannot assess a diff they cannot name. Ask the author to write two short transcripts in the pull request description: what the agent did before the edit, and what it should do after.

A good pair is specific and falsifiable. "Before: when asked to add a test, the agent used the repository's old test runner and created a new config file. After: the agent uses the runner named in the context file and does not create a config file." A bad pair is "improves quality" or "makes the agent more reliable." Neither can be tested, so neither can be reviewed.

This step also forces the author to justify the edit against the file's purpose. The only content worth keeping is information the agent cannot discover from the repository itself: tooling gotchas, non-obvious conventions, landmines (Osmani on /init). If the proposed line restates what is already visible in the code, the correct review outcome is rejection, not a wording tweak.

Use the intended behavior to pick the right reference files, too. Effective setups keep separate documents for overview, architecture and a running decision log, and the agent reads the one relevant to the work at hand (Theresa Torres on context files). When an author insists that an architecture summary must go into the main context file, ask whether a scoped file read at the right moment would do the same job. A hierarchy of context files at directory level keeps each agent's context closer to the code it is working on than one monolithic file does (Osmani on /init).

This is the question most reviewers skip and the one that matters most. "I tried it and it looked better" is not a test. A reviewer needs three things: the input set, the scoring method, and the pass threshold.

Routine prompt evaluation runs a prompt across a set of representative inputs and scores the outputs on defined dimensions such as correctness, relevance, safety and format adherence (Braintrust on prompt evaluation). It exists because a prompt that works in manual testing can fail on production inputs, and small wording changes that help one case can hurt another (Braintrust on prompt evaluation). A context file edit is a prompt change with the same properties.

The evidence for skepticism is concrete. In reproducible local experiments, replacing task-specific prompts with a generic "improved" template improved instruction-following while dropping extraction pass rate from 100 percent to 90 percent and RAG compliance from 93.3 percent to 80 percent for one model (the evaluation-driven iteration paper). A change can be better on the axis the author cared about and worse on an axis nobody measured. Only a defined score catches that.

Ask for a table like this in the pull request:

That last row is not optional. Because context tokens are billed on every call, prompt growth shows up as rising input tokens long before anyone notices a quality change (Braintrust on token usage). A reviewer can ask for the token count per request in the baseline run and the candidate run and compare them directly.

If the team does not have an eval suite yet, the minimum viable path is to write one before approving behavior-changing context edits. Start with ten to twenty real inputs, a mix of automated checks and a small human rubric, and add an LLM judge only where the first two cannot score the output (the evaluation-driven iteration paper). The reviewer's decision rule is simple: no suite, no behavior change. Wording fixes and deletions that shrink the file can merge with a lighter check.

A passing eval before merge only proves the change worked on the day it was written. Context files go stale, and stale context is worse for an agent than no context at all, because the model reads it on every request and treats it as current (the AGENTS.md guide). The reviewer's fourth question is about the sensor that will fire later.

Two sensors are practical. The first is a scheduled eval run against the same suite, which turns a one-time gate into continuous monitoring and catches score drift after any model or file change (Braintrust on prompt evaluation). The second is a documentation drift check in CI: when a pull request merges, a workflow fires, reads the diff alongside the docs, and opens a follow-up pull request if anything drifted (Dosu on documentation drift). The same pattern works for context files, since a renamed path or deleted module makes any reference to it wrong.

The staleness risk is measurable. An analysis of more than 3,000 GitHub projects found that 28.9 percent of the most popular ones contained at least one outdated code element reference, and 82.3 percent were outdated at least once in their history, with references typically stale for years before anyone noticed (the outdated references study). Outdated docs also get worse quietly, because unlike code there is no crash to announce that the text is wrong (the outdated references study).

Ask the author to name the sensor and the trigger. "The nightly eval run fails if tool-choice accuracy drops below 90 percent." "The drift workflow opens a pull request when a referenced path disappears." If the answer is "someone will notice," the edit is unmonitored. For reference, the GitHub Actions route costs about an hour of setup and a day or two of prompt tuning to get working, per the walkthrough (Dosu on documentation drift).

The final question converts the review into an accountable decision. Name the approver, name the second check if the change is risky, and name the revert path.

A workable split for most teams:

Deletions deserve a higher bar than additions, because they are the one edit that reliably shrinks token cost and is therefore tempting, yet they can remove a guardrail nobody remembers adding. The reviewer should ask what breaks if the deleted line was load-bearing.

Rollback is a git revert plus a redeploy of the previous context file, and it should be stated in advance. The reviewer should also confirm the approval is not stale: if a code-changing commit lands on an approved pull request and stale review dismissal is enabled, the approval is dismissed and must be repeated (GitHub required reviews). That behavior is a feature, not friction. It ensures the person who approved the diff is the person who saw the final version.

One practical note on automation. An agent that opens or edits a context file should not also approve it. The sign-off comes from a human owner with write access, and the repository rules should make that structural rather than cultural.

Consider a pull request that adds three lines to AGENTS.md, instructing the agent to always run the full test suite before committing.

The owner check passes instantly: the file is in CODEOWNERS, and the approving review comes from the owning team. The behavior statement is clear: before, the agent ran only the tests for changed files; after, it runs everything. Good so far.

The test question is where the review slows down. The author ran the eval suite on ten tasks. Tool-choice accuracy held at 10 of 10, but input tokens per request rose from 1,800 to 1,950, a 8.3 percent increase on every call. At 50,000 agent requests a month, that is roughly 7.5 million extra input tokens, billed whether or not the rule matters for the task. Prompt bloat of this kind is easy to miss because it never causes a visible failure, only a cost curve (Braintrust on token usage).

The regression question produces a fix. The author discovers that only the release branch needs the full suite, so the rule moves into a scoped context file for that directory and the main file keeps a one-line pointer. The eval is rerun: tokens per request on the main file back at 1,810, release tasks up slightly and passing.

The sign-off question closes it: one owner approves the main file change, a second reviewer who owns the release pipeline approves the scoped file addition, and the rollback note says to revert both commits together.

That is the whole gate. Ownership that is enforced, a behavior claim you can check, a scored test with a threshold, a sensor that fires later, and a named approver with a revert path. None of it is exotic. All of it is cheaper than discovering, weeks later, that an agent has been following a rule someone added by accident. Teams that want to run this loop across several model providers can point one OpenAI-compatible key at the full model catalog through Quanta Inference, which keeps the eval harness stable when the model behind it changes.