A Guard Test That Fails When a Refactor Adds an Indirect Caller

The fixture should be boring: sorted, normalized, human-readable, and stable across line endings and platforms. Sort by caller identity, not by discovery order, or every refactor that reorders imports produces a noisy diff. Join the list with the target's canonical name so a single file can hold many targets.

{
  "schema_version": 1,
  "target": "payments.charge_card",
  "extractor": "ast-callers",
  "callers": [
    "api.checkout::submit",
    "api.webhooks::replay",
    "jobs.retry::run",
    "tests.payments::test_charge_retries"
  ]
}

Three rules make the schema useful.

First, exclude the target's own recursive calls and test files only if you decide to, but decide once and encode it in the extractor version. Including test callers is often what you want: it tells you which tests pin the function, and a deleted test that used to call it is itself a meaningful change.

Second, record a canonical identifier that survives file moves. In Python a stable option is module.path::qualified.name, built from the source path relative to the package root rather than an absolute path. Absolute paths make the fixture differ on every machine. If your project is organized as a monorepo with packages, anchoring identities to the package folder is the same principle that makes package workspaces more robust than long relative imports, since restructuring breaks path-based references [3].

Third, store an extractor or version field. When you upgrade the parser or change the exclusion list, the fixture is expected to churn, and reviewers need to know that the churn is caused by the tool rather than the code. Without that field you cannot tell the difference between "someone added a caller" and "the tool now sees a caller it used to miss."

You need two pieces: a function that walks the source and yields caller/callee pairs, and a wrapper that filters to your target and sorts. Keep extraction pure, with no file writes, so the same code path serves both the test and the update command.

For a minimal Python implementation, walk the AST and record which enclosing function each ast.Call node sits in. A compact example of this pattern exists in community code, though it has a known flaw: walking nested FunctionDef bodies from the outer function attributes a nested call to both functions [4]. The fix is to track a stack of enclosing scopes and attribute each call to the innermost one only.

import ast
from pathlib import Path

def iter_calls(root: Path):
    for path in sorted(root.rglob("*.py")):
        rel = path.relative_to(root).with_suffix("")
        module = ".".join(rel.parts)
        tree = ast.parse(path.read_text(encoding="utf-8"))

        def visit(node, scope):
            for child in ast.iter_child_nodes(node):
                if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                    visit(child, f"{module}::{child.name}")
                elif isinstance(child, ast.Call):
                    name = getattr(child.func, "id", None) or getattr(
                        child.func, "attr", None
                    )
                    if name and scope:
                        yield scope, name
                    visit(child, scope)
                else:
                    visit(child, scope)

        yield from visit(tree, None)

def callers_of(root: Path, target_name: str):
    return sorted({caller for caller, callee in iter_calls(root)
                   if callee == target_name})

This handles direct calls by name and method name but misses attribute chains, aliases and getattr string usage. Production extractors in this space use tree-sitter for structured parsing and report exactly that split: a fast regex mode for speed and a precise AST mode for edge cases, with per-language quality ratings that rate Python as good rather than excellent [5]. Choose precision level deliberately and write it into the fixture.

The limits are real. JavaScript call graphs miss edges from eval, bind and apply, and static graphs are conservative by construction, including edges that never execute for any input [6]. If you need cross-language edges, a system that normalizes into a universal AST can represent calls across languages, which a per-file parser cannot [7]. A guard test against a single-language, name-based extractor is still useful; it just protects a narrower claim.

Two scripts, one policy. The test reads the fixture, recomputes, and asserts. The update command rewrites only on an explicit flag, and the diff lands in the pull request.

# tests/test_caller_guard.py
import json
from pathlib import Path
from callers import callers_of

ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "tests" / "fixtures" / "pinned.json"

def test_caller_list_is_frozen():
    pinned = json.loads(FIXTURE.read_text(encoding="utf-8"))
    current = callers_of(ROOT / "src", pinned["target"].split(".")[-1])
    # New or removed callers show up as readable set subtraction.
    added = sorted(set(current) - set(pinned["callers"]))
    removed = sorted(set(pinned["callers"]) - set(current))
    assert not added and not removed, f"added={added} removed={removed}"

# tools/update_callers.py
import json, sys
from pathlib import Path
from callers import callers_of

if "--approve" not in sys.argv:
    sys.exit("Refusing to rewrite. Re-run with --approve and review the diff.")

root = Path(".").resolve()
fixture = root / "tests" / "fixtures" / "pinned.json"
data = json.loads(fixture.read_text(encoding="utf-8"))
data["callers"] = callers_of(root / "src", data["target"].split(".")[-1])
fixture.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")

The --approve flag is not ceremony. Approval testing culture is explicit that an approval file is a reviewed example, not a magic truth, and that without review the technique degenerates into accepting whatever the program printed last [8]. Practical safeguards that follow from that: never let the update run in CI, never let it run as part of the normal test command, and fail if the rewrite produces no change so the flag cannot be copy-pasted out of habit.

Here is a worked example. Suppose the fixture has 4 callers, and a refactor adds a job wrapper that calls charge_card through a helper named charge: one new entry appears, the test fails, the developer checks whether the indirect path was intended, and either reverts the wrapper or runs the update, which shows a one-line addition in the diff. Cost of the check is the parse of the source tree, which for a small package is well under a second; the debug time avoided is the hour spent hunting why a payment suddenly runs twice from a background job.

A guard test that fails on unrelated churn gets disabled within a month. Before committing to it, measure both error directions on your own repository.

False positive rate: pairs the fixture rejects that are not real caller changes. Sources include reordered files, formatting, new decorators, and parser upgrades.

False negative rate: new callers the extractor never sees. Measure by introducing a known new call by hand in a scratch branch and confirming the test fails, then repeat for the tricky forms you actually use: higher-order functions, defaults, registries, and test doubles.

The published comparisons justify optimism on the first and caution on the second. In a study of static JavaScript call graph builders, ACG had the highest precision, and combining ACG with TAJS covered 99% of true edges found by all tools while keeping precision at 98%, though each tool also found valid edges the others missed [6]. That is the shape you should expect: good precision, no single tool complete. If your design requires complete coverage of one tricky path, a dynamic capture is complementary, since dynamic analysis sees real executions and misses everything else [1].

The decision rule is simple. Pin a function when its behavior is expensive to change or its callers are security- or money-sensitive. Use a precise tree-sitter extractor for those targets and accept that the fixture may periodically need a tool-version bump. Prefer a fast extractor for wide but shallow inventories where churn tolerance is higher. Do not pin everything; an unreviewed fixture covering a thousand functions is a checklist nobody reads.

If you are running this over a large codebase and want the inventory kept fresh automatically, an agent can do the mechanical part. Quanta experts read your repository context and return finished deliverables such as an updated fixture with the diff called out, which you then review before approving it.

Four maintenance habits keep the guard useful.

Rewrite on one command, review on every pull request. If the diff has no reviewer, the fixture has no authority. In repositories using approval-style tests elsewhere, keep the same vocabulary so people already know that an approved artifact is a reviewed one [8].

Pin what you call, not just the function. When a call goes through a stable public function like a module entry point, the caller identity is the module, not the internal helper. That keeps identity stable when code is reorganized.

Treat tool upgrades as separate commits. Bump the extractor, regenerate, and label the pull request as tool-only so nobody mistakes a broad diff for a semantic change. This is the same reason the extractor field exists in the schema.

Fail the build on update-in-CI. A CI job that can rewrite the fixture is a guard that guards nothing. The check must be read-only in CI, and the write must require a human command.

One more habit worth copying from structured-query workflows: give developers a way to ask questions about the fixture. A small trace callers-style command that prints the current list with file and line context makes the fixture legible, which is the same value a graph query tool gives when it answers "what is the maximum call chain length" over the code graph [2]. The fixture is the contract; the query command is how people inspect it before changing it.