Running One Eval on Two Providers Without Rewriting It

Most of the time lost on a second provider comes from three surfaces. None of them is the model call itself.

Message formats. They look similar and are not. OpenAI-style chat uses an array of messages with a role and content the endpoint comparison. Anthropic-style messages also use an array, but the system prompt is a top-level field rather than a message with role: system, and tool results arrive as user-role content blocks the API structure comparison. A newer OpenAI Responses style uses an input field and returns typed output items rather than a single message the migration gist. Three families, three shapes.

Tool-call schemas. JSON Schema for parameters is close to universal, but the envelope around it is not: the key that holds the schema, how the model's chosen call is represented in the response, how tool results are fed back, and whether calls carry an ID you must echo. Code written against one envelope will silently produce zero tool calls against another.

System prompt handling. This is the most common silent failure. A canonical "system" string that gets placed in the message array for one adapter and dropped into a top-level field for another will behave differently on multi-turn and tool-use turns, and you will attribute the difference to the model.

The canonical shape is deliberately boring. It should hold only what your assertions and your prompt builders actually need, because every extra field is another thing two adapters must agree on.

from dataclasses import dataclass, field

@dataclass
class Turn:
    role: str            # "system" | "user" | "assistant" | "tool"
    text: str = ""
    tool_call_id: str = ""

@dataclass
class ToolSpec:
    name: str
    description: str
    parameters: dict     # JSON Schema object

@dataclass
class CanonicalRequest:
    system: str
    turns: list[Turn]
    tools: list[ToolSpec] = field(default_factory=list)
    max_tokens: int = 1024
    temperature: float = 0.0

@dataclass
class CanonicalResponse:
    text: str
    tool_calls: list[dict]   # [{"name": ..., "arguments": {...}, "id": ...}]
    stop_reason: str         # "end_turn" | "tool_use" | "max_tokens"
    raw: dict                # kept for debugging, never asserted on

Two design choices carry most of the value. First, temperature defaults to zero for evals, and the adapter must say so explicitly if a provider treats it differently, because sampling differences will otherwise show up as quality differences. Second, raw exists so you can debug a surprising failure without letting any assertion read provider-specific fields. If an assertion touches raw, portability is already gone.

The system prompt lives in its own field, not in turns, because placement is an adapter decision. Keep a system_placement config per adapter (top_level, first_message, or prepend) so the choice is visible in one line of configuration rather than buried in translation code.

An adapter is two functions. Keep it that small.

class ProviderA:
    name = "a"
    system_placement = "top_level"

    def to_wire(self, req: CanonicalRequest) -> dict:
        messages = []
        for t in req.turns:
            if t.role == "tool":
                messages.append({"role": "user", "content": [
                    {"type": "tool_result", "tool_use_id": t.tool_call_id, "content": t.text}
                ]})
            else:
                messages.append({"role": t.role, "content": t.text})
        body = {
            "system": req.system,
            "messages": messages,
            "max_tokens": req.max_tokens,
            "temperature": req.temperature,
        }
        if req.tools:
            body["tools"] = [{"name": t.name, "description": t.description,
                              "input_schema": t.parameters} for t in req.tools]
        return body

    def from_wire(self, resp: dict) -> CanonicalResponse:
        text, calls = [], []
        for block in resp.get("content", []):
            if block.get("type") == "text":
                text.append(block["text"])
            elif block.get("type") == "tool_use":
                calls.append({"name": block["name"], "arguments": block["input"],
                              "id": block["id"]})
        return CanonicalResponse(
            text="".join(text), tool_calls=calls,
            stop_reason=resp.get("stop_reason", "end_turn"), raw=resp)

The second adapter differs in exactly the places the table predicted: tools are wrapped under a function key with a parameters field, tool results go back as a message with role: tool, and the system string is pushed into the message array. If system_placement is first_message, insert it at index zero and assert in a unit test that it is still present after three turns.

Two habits keep adapters honest. First, round-trip tests: take ten saved canonical requests, run to_wire, run from_wire on a saved response fixture from that provider, and assert the canonical response matches a checked-in expectation. Second, an assertion in every adapter that the tool count in the outgoing body equals len(req.tools) when tools were supplied, which catches the silent zero-tool-call failure in one line.

One subtlety worth writing down in your own code: when a model returns a tool call, your runner executes it and appends a tool result turn. That loop must use the provider's call ID, and a provider that omits IDs forces you to synthesize one and keep the mapping in the adapter, never in the runner.

Assertions are where portability pays off. Because they read CanonicalResponse only, the same suite scores both providers, and a difference in the score is a difference in behavior, not in plumbing. That is the property you want when comparing options, and it is the same reason a weighted scoring model works: scores are computed against shared criteria with fixed weights, so ranking is comparable across options the ProductPlan glossary and the multiplication of score by weight before summing is what makes the result comparable the Savio guide.

Practical checks for an assertion layer:

• Assert on structure first (stop_reason, tool name, required argument keys), then on content. Structural failures are portability bugs; content failures are model behavior.
• Score each dimension separately (format compliance, factual accuracy, instruction adherence) rather than one blended number, so regressions stay diagnosable.
• Use rubric-based judging for open-ended output but calibrate it against human labels, since judges carry known biases such as verbosity preference the rubric-based eval analysis.
• Never let a judge see the provider name in its prompt.

A worked example makes the payoff concrete. Suppose a support-triage eval has 200 cases. On provider A, 186 pass format, 171 pass content. On provider B, 184 pass format, 158 pass content. The format gap of 2 cases is small enough to be sampling noise at temperature zero with occasional non-determinism. The content gap of 13 cases is the finding. Without a shared assertion layer you would have spent the week debugging two harnesses instead of one behavior.

If a provider's outputs are good enough that you want it in production alongside the first, running both behind one interface is the operational version of this pattern, and Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible API key, which removes the second adapter entirely for serving Quanta Inference.

Run this before comparing any numbers.

Two of these deserve emphasis. The determinism check catches adapters that quietly pass a default temperature through when the canonical request says zero. The assertion-neutrality check is a grep, not a judgment call: search the assertion module for the provider names and for raw.

Teams that lean on evaluation daily often find the harness work is the bottleneck, which is where Quanta Experts can take a described eval design and return finished deliverables such as the rubric, the fixture set and the adapter skeleton for review Quanta Experts.

There is a decision to make about how far the canonical shape should stretch. The rule I use: if two providers disagree on a capability your assertions depend on, put it in the canonical shape; if they disagree on something only one provider can do (a provider-specific reasoning control, a cache hint), hide it behind an optional per-adapter settings dict and never assert on it.

Apply some weight to the trade-off. Score each candidate extension (streaming, structured output enforcement, tool-choice forcing, image input) on how many of your eval cases need it, out of one. An extension needed by under a tenth of cases should wait. An extension needed by most cases belongs in the canonical request on day one, because retrofitting it means editing every adapter and every fixture.

The broader migration literature makes the same point from the other direction: API abstraction frameworks normalize the mechanical differences competently, but they do not make your prompts or your test data portable the portability write-up. The runner and its assertions are yours to keep. That is the piece worth investing in, and it is the piece that makes a second provider a configuration change rather than a project.