How to Test a Policy Agent on Edge Cases

Synthetic scenarios tend to be polite and unambiguous. Real disputes are not. The best source of edge cases is the places where humans already argue: complaint records, escalated tickets, legal review notes, and the interpretive material your counsel has produced.

Contract ambiguity has a well-developed vocabulary you can borrow for test design. Courts distinguish patent ambiguity, where the conflict is visible on the face of the document, from latent ambiguity, where a facially clear term becomes uncertain only when applied to a real situation (Plunkett Cooney). A contract is ambiguous when two provisions conflict or when a term is reasonably susceptible to more than one meaning (Plunkett Cooney). Contract management vendors describe the same failure modes in operational terms: undefined terms, vague language, conflicting clauses and inconsistent terminology (Sirion).

That vocabulary maps directly onto test categories. "Reasonable notice" is a latent ambiguity. "Cancel within 30 days of purchase, or within 14 days of delivery" is a patent conflict if both sentences appear in different documents. "Industry-standard retention" is an undefined term that an agent will happily fill in with a guess. Each of these is a scenario with a correct answer, and the correct answer is often "escalate to a human," which is exactly the kind of behavior that is never tested by default.

California's statutory rules give you a ready-made checklist for what a well-behaved agent should reason about: give effect to mutual intent, read the whole contract together, give words their ordinary meaning, and construe remaining uncertainty against the drafter (Stimmel Law). You do not need your agent to act like a judge. You do need your test expectations to be consistent with how a judge would read the same text, because that is the standard your escalation threshold should approximate.

A scenario that lives in a spreadsheet will rot. A scenario that runs in CI will not. The practical format is a small record with five fields: the case inputs, the retrieved documents, the expected decision, the expected rationale anchor (the clause that justifies it), and the escalation flag.

id: cancel-co-0031
inputs:
  jurisdiction: OH
  contract_type: prepaid_entertainment
  signed_at: "2025-03-01T14:00:00-05:00"   # Saturday
  cancel_requested_at: "2025-03-05T21:00:00-05:00"
  notice_channel: email
  cancellation_form_provided: true
retrieved:
  - ohio_home_solicitation_act
  - ohio_prepaid_entertainment_contracts_act
  - company_refund_policy_v4
expect:
  decision: accept
  deadline: "2025-03-05T23:59:59-05:00"      # midnight of third business day
  rationale_clause: "ORC 1345.41"
  escalate: false

The deadline field is where most suites earn their keep. Notice periods that cross weekends, federal holidays and month boundaries are where date arithmetic quietly breaks. Ohio's own worked example is a Saturday signing with a Wednesday deadline, because Sunday is not a business day (Ohio Attorney General). Florida adds a second dimension: a sale of future services can be cancelled within three business days by notifying the seller, with no written notice required, and the refund must arrive within 20 days of the seller receiving notice, while a home solicitation sale requires written notice postmarked by midnight of the third business day and a refund mailed within 10 days of cancellation (Florida Attorney General). Two similar-sounding contracts, two different notice and refund regimes. If your agent collapses them, the test catches it.

The mailbox rule is another trap worth a dedicated scenario: a cancellation is effective on the postmark date, even if the seller does not receive or process the letter until after the deadline (Ohio Attorney General). An agent that reasons from receipt date will deny valid cancellations, and that error is invisible in a happy-path test.

The hard part is not writing scenarios. It is deciding what the right answer is when the policy is genuinely unclear. Without that decision, the test suite becomes a mirror of the agent's own behavior.

Use a three-way outcome model. accept and deny are self-explanatory. The third bucket, escalate, is the one teams forget, and it is the one that keeps you out of trouble. A contested case is not a case where the agent should pick the more generous reading; it is a case where the agent should stop and hand off with a specific question.

That last row matters for partial refunds. A customer who cancels a bundle where one component is clearly refundable and another is contested should get the undisputed money and an escalation on the rest. Agents tend to either refund everything or nothing, and both are wrong.

The human-in-the-loop literature gives you the design rationale for this bucket. Approval gates should be required where the agent's next action is irreversible, costly, regulated or high blast radius, and refunds are called out explicitly as a write action that should be supervised by default until proven safe (StackAI). Notice also the point that the delay in human review usually comes from poor routing and incomplete context rather than the existence of a human step, and that an evidence pack should make the decision obvious in ten to thirty seconds (StackAI). That reframes your test expectation: for escalated cases, the agent's job is not to decide, it is to produce a clean evidence pack.

Policy text changes. Retrieval changes. A new document gets ingested and starts outranking the one that used to win. Any of these can silently change decisions, which is why the suite belongs in the deploy path rather than in a quarterly review.

Build the gate in three tiers so it runs fast enough to be mandatory:

The reasoning behind tiering is the same one that drives agentic regression testing in software: large suites become too slow to run, engineers start hand-picking tests by gut feel, and coverage degrades unevenly (Tricentis). The same source notes that teams commonly spend 40 to 60 percent of automation effort fixing existing tests rather than expanding coverage (Tricentis). For a policy suite, the equivalent maintenance tax is scenario drift: the policy changes, the expected outcome is stale, and someone has to decide whether the new behavior is a bug or a policy update. Assign an owner per scenario group so this decision has a name attached.

One discipline makes the gate trustworthy: never let the suite pass by loosening an expectation. If the agent's behavior changed and the new behavior is correct, update the expected outcome in the same commit as the policy change that caused it, with a note pointing at the clause. If you cannot point at a clause, the policy did not change; the agent did, and you have a bug.

A pass or fail tells you almost nothing. What you need after a failure is the retrieved passages, the clause the agent relied on, and the point where its reasoning diverged from the expectation.

Log four things per decision: the exact retrieved chunks with document identifiers and versions, the clause reference the agent cited, the extracted facts it used (signing date, notice date, jurisdiction, contract type), and the outcome bucket. Then store the run alongside the policy version so you can diff two runs and see which decisions flipped.

Contract extraction work shows why this is harder than it looks. Contracts rarely follow a standard format, legal language is dense with cross-references and conditional clauses, and important details may sit in exhibits rather than the main body (Icertis). Keeping human validation and oversight in the loop is what keeps extraction accuracy high over time (Icertis). Translated to testing: your retrieved chunk identifiers must point into the actual document, not into a summary of it, or you cannot tell whether the agent read the right sentence or a paraphrase of it.

Approaches that turn legal text into explicit rules give a useful structural model here. A rule-extraction pipeline that produces explainable decision support makes each step inspectable rather than collapsing everything into one verdict (Bertl, Price and Draheim). You do not need to formalize your whole policy into logic, but you do want each scenario to be traceable through retrieval, extraction, rule application and outcome. When a scenario fails, the log should tell you which of those four stages broke.

Suppose cancel-co-0031 above fails: the agent denies the request because it compared the cancellation email's received timestamp of March 6 against the March 5 deadline. Three possible causes, three different fixes.

1. Retrieval failure. The Ohio statute chunk was not retrieved, and the agent fell back on the generic company policy. Fix: check the retrieval ranking and add the statute to a jurisdiction-filtered index.
2. Extraction failure. The statute was retrieved, but the agent read "third business day" as three calendar days. Fix: add explicit guidance or a deterministic date tool, and add sibling scenarios for the holiday cases.
3. Policy gap. The statute was retrieved and read correctly, but the company policy document contradicts it by defining the deadline as receipt-based. Fix: the policy is wrong, not the agent. This is the failure mode that justifies the whole exercise, because the same gap would have produced complaints from customers you cannot see.

Only the third case requires a lawyer. Distinguishing it from the first two is what the log is for.