How to Test a Billing Agent's Idempotency and Rollback

Your fake processor is the most important test infrastructure you will write. Give it the smallest possible surface: create a charge, retrieve a charge, and emit webhook events. Everything else is a hook for making the test fail in a specific way.

Three controls cover almost everything.

• Behavior before commit: return a 500, return a validation error, or hang until timeout. Nothing reaches the fake's charge table.
• Behavior after commit: write the charge, then return a timeout or a 502. This is the dangerous case, because the charge exists.
• Delivery control: queue a webhook event for manual dispatch, dispatch it twice, or dispatch it out of order.

A compact implementation of the charge endpoint:

import uuid
from dataclasses import dataclass, field

@dataclass
class FakeProcessor:
    charges: dict = field(default_factory=dict)
    key_index: dict = field(default_factory=dict)
    fail_after_commit: bool = False
    fail_before_commit: bool = False
    pending_events: list = field(default_factory=list)

    def create_charge(self, idempotency_key, amount, currency="usd"):
        if self.fail_before_commit:
            raise ProcessorError(status=500)
        if idempotency_key in self.key_index:
            # Repeated key returns the original charge, no new money moves.
            return self.charges[self.key_index[idempotency_key]]
        charge = {"id": "ch_" + uuid.uuid4().hex, "amount": amount, "currency": currency}
        self.charges[charge["id"]] = charge
        self.key_index[idempotency_key] = charge["id"]
        self.pending_events.append({"type": "charge.succeeded", "charge": charge})
        if self.fail_after_commit:
            raise ProcessorError(status=504)
        return charge

The property to note: key_index makes the fake behave the way a real processor does with idempotency keys. A repeated key returns the original charge, and charges does not grow. Your test asserts on len(processor.charges), never on how many times create_charge was called.

Build the same discipline into the webhook side. Store every inbound event id behind a unique constraint, and write that row before you apply any side effect. On a unique violation, return success and do nothing, because you have already processed it webhook idempotency guide. If a handler inserts a second payment row while returning 200, it is worse than one that returns 500 and forces a retry.

CREATE TABLE processed_events (
    event_id text PRIMARY KEY,
    received_at timestamptz NOT NULL DEFAULT now(),
    status text NOT NULL DEFAULT 'STARTED'
);

Delete the row only when processing fails, so the next delivery can retry. Never delete it on success.

This test catches the most expensive bug, so walk through the arithmetic.

Take an invoice of 200 dollars. The agent is asked to charge it once. The fake is configured to commit and then return a 504. The agent sees an error and, because of its retry policy, sends the same request again with the same key.

If the agent generates a fresh key on retry, the second row reads a charge count of 2 and a ledger balance of 400. The test fails at the level of money, not at the level of an internal counter, which is what you want.

def test_timeout_then_retry_charges_once():
    proc = FakeProcessor(fail_after_commit=True)
    ledger = FakeLedger()
    agent = BillingAgent(processor=proc, ledger=ledger)

    agent.run_once()          # commits, then times out internally
    agent.retry_pending()     # same logical operation, same key

    assert len(proc.charges) == 1
    assert ledger.balance_for("cust_1") == 200

One subtlety decides the test. The retry must carry the same key, so store the key on the logical operation record before the first attempt and have every retry read the persisted value. Domain objects that derive a key from a random value per call will pass trivially against a fake that ignores keys and fail against a real processor that honors them.

This is also where the ratchet for statistical confidence belongs. Run the timeout scenario across every retry policy you ship, not just the default, because a policy that gives up after one attempt hides the duplicate-charge path entirely.

Example tests cover the failures you thought of. Property-based tests cover the sequences you did not. The property is total: for any sequence of attempts on the same logical operation, the number of committed charges is 0 or 1.

from hypothesis import given, strategies as st

outcomes = st.sampled_from(
    ["ok", "timeout_after_commit", "500_before_commit", "dup_webhook"]
)

@given(st.lists(outcomes, min_size=1, max_size=8))
def test_never_more_than_one_charge(sequence):
    proc = FakeProcessor()
    ledger = FakeLedger()
    agent = BillingAgent(processor=proc, ledger=ledger)
    op = agent.create_logical_operation(amount=200)

    for outcome in sequence:
        proc.configure(outcome)
        agent.attempt(op, idempotency_key=op.key)

    assert len(proc.charges) <= 1
    assert ledger.balance_for("cust_1") in (0, 200)

Two assertions, both on durable state. The first says the processor holds at most one charge. The second says the ledger agrees with the processor: the customer is either charged once or not at all, never charged zero times while a charge exists at the processor. The root cause of most failures here is that a POST creates a new record on every call and idempotency has to be built deliberately payment webhook reliability.

Run the same property on the inbound side. Generate random delivery orders, including duplicates, and assert that the total invoice line count is unchanged by every replay after the first. Out-of-order sequences belong in the matrix too: a refund event before a success event should either queue or fail loudly, never silently skip a transition out-of-order webhook matrix.

Two practical notes. Seed the generator so failures are reproducible, and print the sequence on failure so the shrinking output becomes a named regression test. Property tests that cannot be reproduced get deleted.

When an invoice change fails partway, you cannot simply restore the old row. Other writes may have touched the same ledger since, and restoring the prior state would discard them. What you need is a compensating transaction: a new entry that reverses the effect of a completed step compensating transaction pattern.

The pattern requires that the compensation account for concurrent work and may depend on business rules, for example whether a cancelled booking earns a full refund compensating transaction design. In billing terms: reversing a charge is a refund entry, not a deletion.

Write down the inverse of each forward step before you write code. If the invoice amount rose from 200 to 260 and the charge for the extra 60 fails, the inverse removes 60 from the ledger and asserts no charge of 60 exists. If the charge succeeded but the webhook never arrived, the inverse queries the processor and refunds if the charge exists.

A decision rule keeps this small: every forward step has a named inverse, and both are idempotent, keyed by the logical operation.

The fourth row is easy to miss and worth testing. A compensation that reverses the money but leaves the customer holding an email saying the invoice changed is still a support ticket. This is why reconciliation, the act of matching what the ledger says to what the processor actually holds, has to be an assertion rather than a human task at month end payment reconciliation.

Now build the test that proves the rollback works. Snapshot the ledger, force a failure at step two of three, run the agent, and compare.

def test_failed_invoice_change_restores_ledger():
    proc = FakeProcessor()
    ledger = FakeLedger(balance=200)
    agent = BillingAgent(processor=proc, ledger=ledger)

    before = ledger.snapshot()

    proc.fail_before_commit = True        # step 2 will fail
    result = agent.change_invoice_amount("cust_1", new_amount=260)

    assert result.compensated is True
    assert ledger.snapshot() == before
    assert len(proc.charges) == 0         # nothing committed, nothing to refund

Then invert the injection point. Put the failure after the commit and assert that the ledger equal to before coexists with exactly one refund entry, so the customer's net position is unchanged even though money briefly moved.

Two checks make the suite trustworthy. First, assert snapshot equality rather than checking individual fields, because rollback bugs hide in fields nobody thought to assert. Second, run the rollback test against three fakes: one that never commits, one that always commits, and one that commits then times out. A compensation that runs the inverse of a step that never happened creates a phantom credit, and a compensation that skips the inverse of a step that did happen leaves the customer short.

Track rollback test results as part of your release gate along with the cost side of the bill. If your agent's own model calls are also metered, the same discipline applies: the amount the provider reports after a retry should not grow when the logical operation is the same usage-based billing operations.

Before a billing agent goes anywhere near a real card, it should pass everything below. Tie each box to a named test so the checklist is not a vibe check.

• Signature verification happens before any business logic, and a forged signature returns 401 with no side effects webhook signature tests
• Every logical operation has a persisted idempotency key, created before the first attempt
• A replayed key returns the original charge and does not grow the charge table
• Duplicate webhook delivery leaves the invoice line count unchanged
• Out-of-order events do not produce illegal state transitions
• Every forward step has a named inverse, including queued notifications
• The rollback test compares ledger snapshots and passes with failures injected before and after commit
• Metered events define whether retries and failures are billable, and a test proves it usage event definitions

Teams running several money-touching agents tend to find these gaps late. Where a billing agent is one of several, Quanta Experts can carry the design and test harness work for that line of business, reading your ledger and processor contracts and returning finished deliverables Quanta Experts.