Assertions That Resist Weakening: A Pattern Library

The weakest edit available to an agent is replacing a specific expectation with a truthy check. Do not give it the option: put a value on the line where a boolean could go.

Before:

result = parse_duration("1h30m")
assert result  # any nonzero number passes

After:

result = parse_duration("1h30m")
assert result == 5400

The spec behind that number is worth writing down before the tests, because the tests need a standard neither agent authored. Here is the kind of short spec that produces the 5400 above: units are h, m, s, each appearing at most once in that order, each following a whole number, and anything else throws a RangeError (Anhaia's parseDuration spec). With three worked examples in the spec ("2m" is 120, "1h30m" is 5400, "1h0m5s" is 3605), a weakening edit has to contradict a document that lives outside the test file.

For floats, an exact equality test is the wrong tool, and the usual substitute quietly weakens. Pin the comparison width instead:

assert pytest.approx(total, abs=1e-9) == 1080.0

approx with an explicit tolerance is still an exact-value pattern, because the tolerance is a stated engineering decision. A bare assert total > 0 is not.

Collections deserve the same treatment. Asserting len(items) == 3 is not the same test as asserting the three elements in order. Where order is not guaranteed, sort first, then compare to a literal:

assert sorted(invoice.line_ids) == ["ln_1041", "ln_1042", "ln_1043"]

Two independent statements now hold: the count and the identities. Softening one leaves the other exposed.

Ranges are legitimate when the contract is genuinely a range, for example a retry delay, a rate limit window or a confidence threshold. The mistake is writing a one-sided bound that can drift to zero.

Before:

delay = backoff_seconds(attempt=3)
assert delay > 0

After:

@pytest.mark.parametrize("attempt,expected", [(1, 1), (2, 2), (3, 4), (4, 8)])
def test_backoff_doubles(attempt, expected):
    assert backoff_seconds(attempt=attempt) == expected

def test_backoff_is_capped():
    assert backoff_seconds(attempt=12) == 30

The parametrized table pins every intermediate value, so the ceiling cannot be softened without breaking a row. If a true range is unavoidable (a jittered sleep, for instance), pin both ends and add one case just outside each end:

value = jittered_sleep(base=2.0)
assert 1.8 <= value <= 2.2

with pytest.raises(AssertionError):
    assert 1.8 <= jittered_sleep(base=2.0, factor=0.5) <= 2.2

The second block reads oddly on purpose: it demonstrates that the bounds you wrote are the bounds that hold. If someone loosens 1.8 to 0.0, the deliberate outside case stops failing, and the assertion inside the raises block flips from "no exception" to "AssertionError raised," which then fails the test. That inversion is the cheap trick of this pattern: an outside case wired so that removing the bound breaks the test rather than freeing it.

Boundary thinking is standard advice in unit testing for a reason. The simplest happy-path test comes first, then edge and boundary cases, and any class should be testable on a machine with no network (Nick Hodges's unit testing tips). Ranges with one bound fail that standard because they cannot distinguish "correct" from "huge."

Asserting that an exception was raised is half an assertion. Anything from a TypeError in the test setup to a KeyError in a serializer satisfies it, so the check survives code that has nothing to do with your error path.

Before:

with pytest.raises(Exception):
    parse_duration("5x")

After:

with pytest.raises(RangeError) as excinfo:
    parse_duration("5x")
assert str(excinfo.value) == "unsupported unit 'x' in '5x'"
assert excinfo.value.input == "5x"

Three things now hold: the exact class, the exact message and a structured attribute. Weakening the class to Exception fails because pytest.raises(RangeError) never sees a generic exception; weakening the message to a substring check still runs into the structured attribute, which no substring edit satisfies.

If you genuinely need tolerance on the message (translated strings, templated values), keep it narrow and keep the type exact:

assert str(excinfo.value).startswith("unsupported unit")
assert type(excinfo.value) is RangeError  # not a subclass either

type(...) is rather than isinstance(...) is deliberate in a handful of places. If the contract says a specific class, a subclass that also happens to be accepted is a contract drift you want a failing test for.

The definition of done for an agent's task should include this class of check. A done list worth trusting asks for evidence that runs after the agent's last edit, and names "the diff deletes no test and marks no test as skipped" as one of its items (SpecStory's definition of done). Add "no assertion loosened" to that list and you are checking the right property.

There is a real tension: a test that spies on an internal call catches regressions and also breaks every time you refactor. The failure mode people miss is that the standard fix (drop the spy) removes the assertion entirely, so the test can never fail for any implementation.

The golden rule of assertions is that a test should fail only when the code is actually incorrect, not when the implementation changes while behavior stays the same (Web Dev Simplified on assertions). A spied call to an internal parser satisfies neither half: it breaks on a harmless refactor and it says nothing about the returned name once the parser is bypassed.

The resolution is to assert the contract at the boundary and keep one independent structural check:

result = get_person_name(person)
assert result == "Kyle"          # behavior, survives refactor
assert set(result) <= set(NAME_CHARSET)  # shape, still fails on junk

The behavior assertion survives refactoring, and the shape assertion still fails if the function returns an object, a dict or an empty string. Neither can be weakened to assert result without losing information you can point at in review.

For response objects, assert the whole shape rather than picking fields:

assert set(payload.keys()) == {"id", "status", "line_items"}
assert payload["status"] == "confirmed"

The key-set assertion catches both a missing field and a stray debug field, and no toBeTruthy edit preserves it. If the payload is large and a full snapshot is brittle, a partial object match is acceptable so long as the keys you name are the keys the contract names.

The strongest form of resistance is redundancy: two assertions that would each fail alone and that no single edit removes together. Because they are independent, weakening one leaves the other firing, and weakening both requires an edit that reviewers and tools can spot as a pair.

Two mechanisms close the loop at the pipeline level. The first is a split between red and green: one agent writes failing tests from the spec, a separate agent makes them pass, and a git check fails the second agent if it touched the test files it was supposed to satisfy (Anhaia). The second is a pre-commit stage. Standard hooks catch the mechanical cases, and pre-commit supports long-standing hooks such as check-merge-conflict for exactly this kind of guard duty (pre-commit-hooks). A local hook that diffs the test directory against the base branch and fails on deleted or skipped tests is a dozen lines and removes the entire "just skip it" class of weakening.

Mutation testing tells you whether the assertions you kept are load-bearing. Run it on changed files only, and treat a surviving mutant as a request to either strengthen an assertion or document an equivalent mutant, which is a known cost you cannot eliminate completely (Stryker on equivalent mutants). A 99 percent score with a surviving mutant at your only boundary check is worse than a 92 percent score where the survivors are all equivalents in number formatting.

Put this where the agent cannot edit it, next to the acceptance criteria, and mirror the same items in the pull request template. The definition of done for an agent has to live in files the agent cannot change, with checks that run after its final edit (SpecStory).

• No assertion compares a value to True, a bare truthy check, or any.
• Every numeric result is compared to an exact expected value, or to a range with both bounds written and two outside cases.
• Every error path asserts the exact exception class, plus either an exact message or a structured field.
• Every collection assertion checks either full contents (sorted where order is undefined) or the exact key set, never only the length.
• Every mocked interaction is paired with an assertion about observable output.
• The test file diff contains no deletions, no skips and no widened matchers.
• A mutation run on changed files reports survivors, each one either strengthened or explained.

The last item is where tools pay for themselves. Agents write plausible code that passes superficial inspection while drifting from requirements, and the reliable fix is independent gates the agent cannot bypass rather than instructions it may follow (Teemu Piirainen on enforced gates). Aim the gates at the assertions, not just the tests. A suite of weak assertions in files with a clean diff is the failure this library exists to prevent.

For teams running several model-backed agents, the same hardening applies to the assets those agents use. If your coding agent needs a shared library of test-authoring and review skills so every agent applies the same patterns, that belongs in a skill library rather than in prompt text: Quanta Skills installs a skill into the agent you already run, or puts an expert on the task (https://quantaintelligence.ai/skills).