Contract Tests Between Your Smoke Suite and Its Vendors

A contract test for a third-party payment API is narrower than a consumer-driven pact between two teams you both employ. You cannot ask the vendor to verify a pact file. What you can do is pin the parts of the response your probe depends on, and assert them against a recorded baseline.

The method matters less than the discipline. Pactflow's walkthrough of schema-based contract testing shows the shape of the workflow: author the contract, generate a schema from the consumer's types, then diff the provider's actual output against that schema. You can do the same with the vendor as the "provider," using a JSON schema you keep in your repo.

Two choices are worth calling out because they decide whether the test is useful or noisy:

• Assert on presence and type, not exact values. Production traffic and sandbox differ in values constantly. You want to know that amount is an integer, not that the fee is 30 cents.
• Allow additive change by default, block on removal and retype. A vendor adding a field is usually harmless. A vendor removing status or turning amount into a decimal string is not. If your schema validator rejects unknown properties, every additive vendor release becomes a false alarm.

DigitalAPI's contract testing writeup uses the example of a weather API that promised a number and started returning text: the mismatch is caught by shape, not by any functional expectation. Your amount field is the same story with money attached.

The single most important design decision is where the contract comes from. Record it from the vendor's documented sandbox with test credentials, and never from the live billing endpoint. QASkills' production smoke suite skill is blunt about this: production smoke tests must never create permanent data, modify accounts, trigger billing events, or send real notifications, and every test should be read-only or use dedicated accounts with sandboxed permissions. A smoke test that accidentally charges a customer is worse than no test.

A workable recording procedure:

1. Pick three or four representative interactions, not the whole API. A successful intent creation, a declined card, an invalid request (400), and an auth failure (401) cover most of what a billing probe touches.
2. Capture only the shape. Strip values that change on every call (ids, timestamps, idempotency keys) and keep the type and presence. A small script that maps a response to a shape, or a library that infers a schema, both work.
3. Store the result in version control next to the contract test, so a change to the expected shape is a reviewable diff.
4. Record against the vendor's documentation as well. If the sandbox and the docs disagree, that is itself a useful finding, and Pactflow notes that comparing against a published schema helps stop documentation drifting from implementation.
5. Re-record deliberately, not automatically. If your test rewrites the baseline every time it runs, it can never fail. Recording must be a separate command a human runs and commits.

The schema below is the smallest useful version. It encodes required fields and types, tolerates unknown extra fields, and pins the status enum:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id", "object", "status", "amount", "currency"],
  "properties": {
    "id": { "type": "string" },
    "object": { "const": "payment_intent" },
    "status": { "enum": ["requires_payment_method", "requires_confirmation", "succeeded", "canceled"] },
    "amount": { "type": "integer" },
    "currency": { "type": "string", "pattern": "^[a-z]{3}$" },
    "payment_method": { "type": ["string", "null"] }
  },
  "additionalProperties": true
}

additionalProperties: true is the decision that keeps the test quiet through additive vendor releases. The const on object and the enum on status are the decisions that make it loud when the vendor renames a discriminator or a state.

GitHub Actions is the convenient default. Workflows are YAML files stored in .github/workflows and can be triggered by events or by a defined schedule, per GitHub's workflow syntax reference, and the Actions documentation covers the runners and concepts if you are new to it. freeCodeCamp's step-by-step guide is useful for the vocabulary of events, jobs and runners.

The contract job has two triggers, and the split matters:

• On pull request: run the contract test against the recorded baseline using a mock or replayed response. This catches the case where you change your parsing code but not the schema, and it runs in seconds with no vendor call.
• On a schedule: call the vendor sandbox, validate the live response against the baseline, and alert on mismatch. This catches the case where the vendor changed overnight and no commit of yours was involved.

name: billing-vendor-contract
on:
  pull_request:
    paths: ["src/billing/**", "contracts/**"]
  schedule:
    - cron: "17 */4 * * *"
jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm ci
      - name: Validate recorded baseline against fixtures
        if: github.event_name == 'pull_request'
        run: npm run contract:fixtures
      - name: Validate live sandbox shape
        if: github.event_name == 'schedule'
        env:
          PAYMENTS_SANDBOX_KEY: ${{ secrets.PAYMENTS_SANDBOX_KEY }}
        run: npm run contract:sandbox

Two practical notes from the CI side. First, preserve the exit code. Momentic's CI/CD testing guide warns that shell wrappers and constructs like || true can turn a failed run into a passing job, so capture the status and return it after any artifact upload. Second, give the scheduled job a distinctive name, because branch protection and alert routing both key on it, and renaming it can silently detach the rule.

If your pipeline already gates on test results via exit codes, the contract check fits the same pattern. TestCollab's quality gate documentation lays out the exit code convention clearly: 0 meets every criterion, 1 evaluated and failed, 2 could not evaluate. Treat every nonzero status the same way and block.

The second half of the job is cleaning up the smoke probe itself. A contract test upstream is worthless if the probe then asserts on vendor internals anyway. The rule is simple: the probe should assert on your outcome, the contract test should assert on the vendor's shape.

MakerKit's SaaS smoke testing guide puts payment integration in the smoke suite as a plan selection plus a test card and a visible success state, which is the right level: does the flow complete, not what every field looks like. Momentic's overview lists "can critical integrations be reached" as a smoke concern, and that is precisely where the boundary sits. Reachability and outcome belong to the probe. Shape belongs to the contract.

When a contract test fails, the alert should name the path. A message like "amount expected integer, got string" tells the on-call engineer in one glance that the vendor changed something and that the probe failure, if any, is downstream. A message like "contract test failed" tells them nothing and costs another twenty minutes, which is the same cost the test was supposed to remove. Momentic's CI guide makes the point for Slack alerts generally: send the failed check and run details, not a bare pass or fail. For contract checks the diff is the detail that matters.

Store the last passing schema and the current failure side by side in the alert or artifact. That turns the alert into a reviewable change, and the person who investigates can decide in seconds whether the vendor's change is additive (update the baseline, note it in the changelog) or breaking (open a support ticket and patch your parsing). Redocly notes that contract tests give early alerts on discrepancies so teams can address problems before they reach production; the alert content is what makes "early" real.

One more discipline: version the schema and treat a baseline change as a code change. Pactflow's workflow publishes contracts and runs a can-i-deploy check before promoting, so a schema change is reviewed and deliberate. You do not need the full broker for a single vendor, but the review habit is free: require one approval on diffs under contracts/.

You will not get this right on the first pass, because the first pass usually encodes too much. Use this test to decide whether a field earns a schema assertion:

• Does your probe or app code read it? If no, do not assert it. It is the vendor's business.
• Would a type change break arithmetic or a branch? Yes means assert the type (integers stay integers, currency stays a string).
• Would a rename break a branch? Yes means assert presence and, where it is a discriminator, the value.
• Is it a free-text message or a timestamp? Assert type only, never format, because vendors reformat those without warning.

Run the first pass, then wait. Every time your contract test fails, note whether it caught a real breaking change. If two consecutive failures were additive changes you had to update, loosen the assertion or accept the maintenance as the cost of the warning. The shape of a mature suite is a handful of assertions that almost never fire, each of which means a genuine integration problem.

Contract testing is a discipline first. Where the test touches a vendor with a complex auth flow or several response shapes, the recording and the scheduled runs are repetitive to build and maintain. Teams that do a lot of this often centralize the plumbing, and the same is true of the probe: if a smoke job calls models or third-party APIs in several steps, routing them through one gateway keeps the credentials and the shape checks in one place. Quanta Inference is worth a look for that pattern, since it puts text, image and embedding models behind one OpenAI-compatible key and pricing in dollars, which reduces the credential sprawl a billing probe has to manage.

The general discipline holds regardless. Record the shape, assert it away from the live endpoint, schedule the check, and alert on the field that moved. When the vendor changes something at 2 a.m., the contract test should tell you what changed before your billing probe tells you something broke.