A Call Graph Diff an Agent Runs Before It Edits

The first step is a static graph over the repository. A static call graph is a directed graph where nodes are functions and edges exist when one function calls another, as described in the introduction to a JavaScript call graph study. For Python, you can build one from the standard library's ast module plus NetworkX, which is the approach in this worked example. For a maintained tool, pyan is a static call graph generator for Python, and grepai trace offers a callers query with JSON output suited to agents.

A minimal query layer looks like this:

import ast, networkx as nx

def build_graph(paths):
    graph = nx.DiGraph()
    for path in paths:
        tree = ast.parse(open(path).read())
        funcs = {n.name: n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)}
        for fname, node in funcs.items():
            graph.add_node(fname)
            for call in ast.walk(node):
                if isinstance(call, ast.Call) and isinstance(call.func, ast.Name):
                    graph.add_edge(fname, call.func.id)
    return graph

def direct_callers(graph, symbol):
    return {src for src, dst in graph.in_edges(symbol)}

def transitive_callers(graph, symbol, depth=3):
    seen, frontier = set(), {symbol}
    for _ in range(depth):
        frontier = {src for node in frontier for src, _ in graph.in_edges(node)}
        new = frontier - seen
        if not new:
            break
        seen |= new
    return seen

The transitive_callers function is the important one. A direct caller set tells you who calls the symbol today. A caller set with depth two or three tells you who is reachable through another function, which is exactly the kind of indirect breakage a careless signature change introduces. Choose a depth and keep it constant between the before and after runs, because comparing a depth-three before set against a depth-two after set produces phantom removals.

Two properties are worth registering before you trust the numbers. First, static call graphs over-approximate, because of higher-order functions and polymorphism: the pyan-based write-up says as much. Second, they also under-approximate. A study of Android static analysis ran 13 tools and a dynamic analysis over 1000 apps and found the tools failed to capture, on average, 61% of the dynamically executed methods, with high precision a synonym for high unsoundness: the Android call graph paper. Plan for both directions. A caller set is evidence, not a proof.

The agent must not edit the working tree before the check passes. Copy the relevant tree to a scratch directory, apply the patch there, and rebuild.

before.json is a plain map from symbol name to a sorted list of callers. Write the same map after the patch, using the same depth, the same file globs, and the same parser version. If the agent uses a vendored graph tool in one run and a different one in the next, you are diffing tools, not code. Pin the tool.

The scratch copy also protects against a subtler failure. An agent that edits files in place and then needs to discard its work has to reconstruct the original, either from a stash or from version control. Neither is guaranteed to be available in the sandbox. Copying first makes the discard cost one rm -rf.

The diff is a set comparison per symbol. Callers that are present before and absent after are usually fine, because they indicate the symbol shrank. Callers that are absent before and present after are the ones that matter, because they indicate a new dependency on something the agent changed.

before, after = load("before.json"), load("after.json")
regressions = {}
for symbol in before:
    added = set(after[symbol]) - set(before[symbol])
    if added:
        regressions[symbol] = sorted(added)

if regressions:
    print(json.dumps({"new_callers": regressions}, indent=2))
    discard_patch()
    raise SystemExit(3)

Exit code 3 is arbitrary but should be stable and documented, so the orchestrator can distinguish "new callers appeared" from "the build failed" and "the tool crashed." Keep the diff document as evidence. The reviewer reads it, not the agent's summary of it, which is the same principle approval testing relies on when it makes a reviewer read a rendered diff rather than a pile of assertions: the approval testing guide. A patch that changes behavior silently is even harder to catch from a text review than a rendered output, so the machine-generated caller diff earns its place.

Be explicit with the team about the scope. The caller diff catches new direct calls to a touched symbol and, at depth, new paths that reach it. It does not catch a caller whose call reaches the symbol through a function pointer, a dictionary lookup, a decorator, or a framework callback. Function pointer resolution needs dataflow analysis that a lightweight AST walker does not have: the Stack Overflow thread on function pointer call graphs makes the distinction plain.

Reflection is the same story in dynamic languages. Python's own inspect module is at the edge here, since a call made by string name leaves no AST edge at all: the inspect docs. If a repository leans on reflection or dependency injection, lower the trust in the diff and increase the depth so the approximation widens in the safe direction.

WebAssembly, and by extension any build target with indirect call tables, adds a third bucket: the paper on WebAssembly call graph challenges identifies indirect calls via a mutable function table as a distinct difficulty and shows that all four analyses it studied were unsound: the WebAssembly call graph paper. If your repository ships a compiled target, run the diff on the source and treat the compiled layer as unverified.

Soundness and precision trade off in Python tooling too. PyCG is described as the state-of-the-art for Python, and a later tool, Jarvis, reports being at least 67% faster with 84% higher precision and at least 20% higher recall on its benchmark: the Jarvis paper. Those numbers come from that paper's own evaluation, not from an independent test. A separate paper proposes PyPt and reports beating PyCG on soundness and completeness over 99 programs and six real projects: the PyPt abstract. Read both with the same posture: the graph you build is as good as its approximations, and the diff only tells you what the approximations disagree about.

Some edits deserve the full before-and-after run and some do not. A decision rule that works well:

• Run the full diff when the patch changes a function signature, removes a parameter, changes a return type, or moves a symbol between modules.
• Run the direct-caller check only when the body changes but the signature does not.
• Skip the check when the patch touches only tests, fixtures, or documentation.

A worked example with numbers. Suppose the agent plans to rename parse_config to load_config. The before snapshot at depth two returns 4 direct callers and 11 transitive callers. After the patch, the snapshot returns 4 direct callers under the new name and 12 transitive callers, because a module the agent never opened imports the old name through a re-export. The diff names that module. The agent exits 3, discards the patch, and rewrites with a compatibility alias. That is one wasted patch instead of one broken deploy. Without the check, nothing in the patch would have failed a unit test, because the new module is only exercised at import time in a path the tests do not load.

The pattern generalizes to TypeScript monorepos, where a symbol can be re-exported through a path alias and re-imported several layers away. Packing your packages into isolated modules reduces the chance of an accidental cross-package import, but it does not remove the need to know who will call the changed symbol: a monorepo guide on package boundaries covers path aliases, project references and workspaces as the structural answers.

The general principle is well established in program analysis. A static call graph lets you identify which functions may be affected by a change and evaluate reachability, which is what a caller list is: the minimal call graph post. The agent's contribution is not a new analysis idea. It is placing the analysis before the write instead of after the compile.

Put the check in the agent's tool list, not in its prompt. A tool the model can call and read is more reliable than an instruction it has to remember, and a non-zero exit from a tool is harder to explain away than a sentence in a prompt.

The plumbing is short. The tool exposes two functions, snapshot and check, each returning JSON. The agent is told to call snapshot with the planned symbol list before invoking any write tool, and check after applying the patch to a scratch copy. If check exits non-zero, the agent has to amend the plan rather than retry the same patch. Retrying a patch that widened the caller set produces the same diff, and each retry costs another graph build.

Skills help here. A skill that bundles the graph builder, the depth constant, the exit codes and the scratch-copy routine into one installable unit means every agent in the team runs the same check, with the same depth, against the same parser. Put the skill in a coding agent and the check travels with it. See Quanta Skills for the skill library pattern and how to install one into a coding agent.