A Yes, No, Not Applicable Answer Schema for Agent Tool Results

The pattern is simple to state. For each field a tool reports, you define exactly three outcomes and give each one a distinct representation.

The key rule: not applicable is never false and never an empty string. It is a separate, named state with a reason attached. The reason matters because it feeds the next decision. If a model knows the field was unavailable because of permission, it can ask for access; if it knows the field was out of scope, it can drop the question.

Two encodings work well. The sibling status field is the easiest to read:

{
  "user_id": "u_8812",
  "email_verified": { "value": true, "status": "present" },
  "phone_verified": { "value": false, "status": "present" },
  "billing_country": { "value": null, "status": "not_applicable", "reason": "out_of_scope" }
}

The sentinel object is useful when you want the value and the status in one place with a fixed shape, which keeps every property the same type across the response and makes schema constraints simpler. Pick one shape per tool and never mix.

The reason is that a model parsing a tool result should never have to infer which kind of "no" it is seeing. As the buyer-focused evaluation checklist from Viktor puts it, a client wants a concrete description of a failure surfaced in the channel where the request was made, with enough detail to debug (Viktor evaluation checklist). A reason code on a not applicable state is exactly that detail for a missing field.

The hard part is not encoding; it is choosing the state. Three conditions put a field into not applicable rather than no.

Data is missing because the tool could not fetch it. The record exists but the upstream service timed out. Returning false would lie, so return not applicable with reason "fetch_failed". A separate retry or fallback can then act on that reason.

The field is disallowed by authorization. The caller authenticated but the token lacks the scope to read the field. Return not applicable with reason "no_permission". Do not return false; false implies a real negative answer you were allowed to observe.

The field is out of scope for the request. The user asked about account status only, so the tool did not query billing. Return not applicable with reason "not_requested". This is different from fetch_failed because there is nothing to retry.

Everything else is a real value. If the user exists, is active, and is not an admin, the admin flag is present with value false, status present.

A useful decision rule: a field is not applicable when any of the following is true, and no otherwise.

1. The answer would require data the tool did not and should not have called.
2. The caller is not permitted to see the answer.
3. The field does not exist for this kind of entity (an individual user has no parent team).

If none of those apply and the value is genuinely false or empty, it is a no. This is the same reasoning a reviewer applies when a code review checklist item does not apply to that change: the Turing Way guidance explicitly says some items may not apply to your project or language, in which case they should be disregarded (Turing Way code review checklist). You would not mark a language-specific checkbox as "failed" just because it was irrelevant. Tool output deserves the same honesty.

The next step is to make the three states enforceable. Start with a JSON Schema that gives the status an enum, so a model cannot return a fourth invented value. Then add a small validator that also checks the value and status agree.

STATUS = {"present", "not_applicable"}
REASONS = {"fetch_failed", "no_permission", "not_requested", "out_of_scope"}

def validate_field(name, field):
    errors = []
    if field["status"] not in STATUS:
        errors.append(f"{name}: bad status {field['status']!r}")
    if field["status"] == "not_applicable":
        if field.get("value") is not None:
            errors.append(f"{name}: n/a must have null value")
        if field.get("reason") not in REASONS:
            errors.append(f"{name}: n/a needs a known reason")
    else:
        if "reason" in field and field["reason"] is not None:
            errors.append(f"{name}: present field should not carry a reason")
    return errors

Run this over every field after generation. It catches the specific bug the schema cannot: a field that says not applicable but still carries a value, or a present field that carries a permission reason. The schema guarantees shape; the validator guarantees the three states mean what they say.

Schema-constrained generation is the right default, with prompts and regex as a fallback only, because the runtime validator is what catches value errors that schema enforcement lets through (MLflow on structured outputs). Keep the schema minimal, with required fields, strict types, and enums, because that is what makes validation at scale workable (Agenta on structured outputs and function calling).

A tool result is only as good as the test that can prove it. Write one test per state per field, and at least one test that distinguishes the three reasons a field can be not applicable. The shape of the assertions:

def test_phone_verified_false_is_a_no(client):
    out = client.user("u_8812")
    assert out["phone_verified"]["status"] == "present"
    assert out["phone_verified"]["value"] is False

def test_billing_country_missing_permission(client):
    out = client.user("u_8812", scopes=["basic"])
    field = out["billing_country"]
    assert field["status"] == "not_applicable"
    assert field["reason"] == "no_permission"
    assert field["value"] is None

def test_no_field_ever_defaults_to_false(client):
    out = client.user("u_8812", scopes=["basic"])
    for name, field in out.items():
        if name == "user_id":
            continue
        assert field["status"] in {"present", "not_applicable"}
        if field["status"] == "not_applicable":
            assert field["value"] is None

The third test is the one that prevents regressions over time. It is the machine version of the checklist habit Roussy describes: make sure the important thing has been addressed because the list is non-trivial (Marc Roussy code review checklist). Here the important thing is that no field ever silently becomes false.

Run tests at the component level and end to end. Viktor distinguishes deterministic graders (what a script can verify) from LLM-as-a-judge graders (open-ended scoring) and notes that builders need both kinds of evaluation, because a right answer reached through a wrong step breaks on the next run (Viktor evaluation checklist). Your validator is the deterministic grader for state correctness; the judge is for prose quality and should never be asked whether a field is "really" not applicable.

For evaluation of model output more broadly, note the caution that LLM judges drift between runs and can be manipulated by instructions embedded in the text they grade (SoK on grading open-text legal answers). A field's three-state status should never depend on a judge; it should depend on the tool's own logic and the validator.

Suppose a support tool returns a single response for one ticket, with four fields: resolved, refund_issued, customer_vip, and account_manager. Four fields times three possible states is 12 test cases, but the three not applicable reasons collapse some of them. Write the table first.

Now you can assert the count. In the test suite, require that every field in the response is either present or not applicable with a reason from the closed set, and that the response contains at least one present field. Then add a routing test: given the field statuses, the agent's next step is decided.

• If account_manager is out of scope, the agent does not ask a human manager to approve anything.
• If refund_issued has reason no_permission, the agent asks the caller to grant scope and does not retry.
• If customer_vip has reason fetch_failed, the agent retries once and then drops VIP routing.

This is where the pattern earns its keep. A plain {"account_manager": false} would make the agent treat the account as having no manager and skip escalation, which is wrong for a business account. The three-state encoding makes the next action depend on the reason, not the boolean.

Expected pass rate matters too. If your schema validation rejects 3 percent of tool calls, that is a budget for retries and fallbacks. Set a metric for validation failure rate and alert on it, because structured output regressions are otherwise silent (MLflow on structured outputs). A spike in not_applicable with reason fetch_failed is an upstream outage; a spike with reason no_permission is a scope misconfiguration.

The pattern above is portable, but the pipeline has a lot of moving parts: tool calls, schema-constrained generation, validation, retries, and evals across models. Quanta Inference lets a team call text, image, video, audio, and embedding models through one OpenAI-compatible API key, which simplifies running the same validator test against several models without separate SDKs (Quanta Inference). The other lever most teams want is skills a coding agent can install to generate the schema, validator, and tests together (Quanta Skills).