Characterization Tests for Untested Code

Choosing the unit is a scoping decision, not a technical one. Pick the function or module your change touches, and pick the narrowest boundary that still produces an observable result. The guidance from a worked modernization tutorial is to choose one capability before writing any tests, then find the smallest test boundary that exercises it (freeCodeCamp tutorial).

The obstacle is almost always a dependency: a database connection, an HTTP client, a clock, a global. Feathers calls the escape hatch a seam, "a place to alter program behavior, without changing the code" (understandlegacycode.com). In object-oriented code the usual seam is a class you can subclass in the test to neutralize the problematic call. For example:

class FakePaymentGateway(PaymentGateway):
    def charge(self, amount, currency):
        return {"status": "ok", "id": "test-1"}

That fake is not a characterization of anything. It is the scaffolding that lets your characterization test run offline. Keep scaffolding and recorded behavior in clearly separate files so nobody later mistakes the fake's numbers for real ones.

Two practical tests for whether you picked a good unit:

Feathers' working definition of a unit test is blunt: it runs fast (under 100 ms) and it does not talk to infrastructure such as a database, the network, the file system or environment variables (understandlegacycode.com). If your characterization test cannot meet that bar, you are writing an integration characterization test, which is fine, but mark it and expect it to be slower.

The mechanical procedure is identical across languages and frameworks. Daniele Scillia's version is six steps: determine the piece of code, write a test with an assertion you know is wrong, run it and read the error to learn the real behavior, change the assertion to match, repeat until coverage is reasonable, then give each test a name that expresses the behavior it verifies (Medium, Dan the Dev).

The Honeybadger walkthrough shows this concretely on a Ruby mask method with no tests. The author first writes expect(mask('simple text')).to eq(nil), runs it, and reads the failure: expected: nil, got: "*******text". That string becomes the assertion. The same loop on '[email protected]' produces "***************.com", and on the digit string '635914526' produces "*****4526" (Honeybadger). Three inputs, three recorded facts, zero guessing.

In Python with pytest the workflow looks like this:

def test_masks_number_as_string():
    # Step A: deliberately wrong assertion
    assert mask("635914526") == "PLACEHOLDER"

Run it, read the assertion diff, and replace the placeholder with the observed value:

def test_masks_number_as_string():
    assert mask("635914526") == "*****4526"

Two rules keep this honest. First, never type an expected value from reading the code; always copy it from a real run, because the code path you think is taken and the one actually taken often differ. Second, rename the test immediately after it passes. x and test_1 are the most common failure mode of this technique: six months later nobody knows what the recorded string means.

The happy path is the least interesting part of a legacy function, because it is the part the agent will probably reproduce correctly. The edge paths are where behavior diverges. Production failures cluster at boundaries: the value exactly at a limit, the value just past it, empty collections, unusual encodings, and the interaction of two conditions such as a maximum-length name submitted during a session timeout (Virtuoso's edge case guide).

The systematic method is boundary value analysis. Split the input domain into equivalence classes where the function should behave uniformly, then test at, just below and just above each boundary, plus degenerate inputs such as empty collections and missing values (Varsity Tutors on edge case testing). Applied to a single function, that gives a checklist you can work through in twenty minutes:

The last row matters more than it looks. A legacy function that returns results in database order will pass today and silently reorder after an agent adds a GROUP BY. Recording the actual order turns that into a test failure instead of a support ticket.

Error paths deserve their own discipline: assert the exact exception type and, where practical, the message or error code. A test that only asserts "raises" will pass when the agent introduces a KeyError where a ValidationError used to be, and callers that catch the specific type will break in production.

Not everything deserves a recorded test, and freezing too much is its own failure mode. The freeCodeCamp guide draws the line at behavior versus implementation detail: test what a caller can observe, and avoid locking in private helpers, internal call sequences or log formatting that nobody consumes (freeCodeCamp). A test that asserts a private method was called three times will fail on every legitimate refactor and teach the team to ignore red builds.

Use this decision rule for each candidate behavior:

• Characterize it if a caller, a downstream service, a report, or a support process depends on the exact result.
• Characterize it if the behavior looks like a bug but changing it would alter stored data or output someone reads.
• Do not characterize it if it is a private implementation detail, a log line format, or a timing artifact that varies run to run.
• Do not characterize it if the output is genuinely non-deterministic (timestamps, random IDs). Inject a fixed clock or seed, or record only the deterministic parts.

The bug question needs a policy, because characterization tests will surface real defects. The rule that works: record the buggy behavior, mark the test with a comment and a ticket reference, and fix it in a separate, deliberate change where the test is updated on purpose. That way the fix shows up in review as an intentional edit to a named test rather than an invisible side effect. Feathers' point is that once you have the recorded suite, you can see when your changes make the system behave differently, which is exactly the notification system the suite exists to provide (DaedTech). A deliberate diff to a test named test_truncates_negative_amounts_to_zero is a conversation. A silent one is an incident.

This is the payoff step. A documented local setup is the prerequisite for letting an agent self-correct: if the agent knows how to run the compiler, the linter and the tests, it can try dozens of changes, catch failures and fix them without you umpiring each one (Google Cloud Tech walkthrough). The characterization suite is the part of that feedback loop that knows what "correct" means.

A workable order of operations for one function:

1. Write and record the characterization tests. Confirm they pass on untouched code.
2. Commit the tests on their own, before any source change. This commit is your baseline.
3. Give the agent the task plus the test command. Ask it to keep all current tests green and to state explicitly if it believes a recorded behavior is wrong.
4. If the agent wants to change an expected value, treat that as a decision for a human, not a fix.
5. After the change, run the suite and read every diff. A green suite with a modified expected value is not a pass.

The wider case for context before change is well documented: a 10-million-line enterprise project adopted agent tooling without a context layer, the agents were productive but created additional debt, and the engagement stalled for over nine months before pre-computed context changed the trajectory (Driver.ai). Characterization tests are a cheap, high-signal slice of that context, tied to behavior rather than prose.

If you are running agents across several models to compare implementations of the same function, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid dollar pricing, which makes A/B runs on the same characterization suite straightforward (https://quantaintelligence.ai/inference).

Take a 40-line function that computes a customer's invoice adjustment, called from three places, with no tests. A reasonable budget is one focused block of work rather than an open-ended exercise.

Eighteen tests covering the function is roughly the scale most teams find sufficient for a unit of this size. The coverage number itself is secondary; what matters is that the error paths and boundaries are represented, because those are the branches an agent rewriting the function is most likely to reinterpret. Once the suite is green on untouched code, the refactor or migration proceeds against a recorded baseline, and the strangler-style approach of replacing one piece at a time keeps each change small enough to verify (Martin Fowler on strangler fig, Microsoft's Azure guidance).

One further caution: do not let the agent write the assertions from a description of the function. The failure mode is subtle and expensive. A team that had four developers independently generate documentation with an AI, cross-check against each other and validate with the client still found errors (Driver.ai). Independent agreement is not the same as ground truth. Assertions copied from a real execution are.