How to Catch an Interviewer Agent That Drifts Off the Guide

Instrument the loop where the agent produces its next utterance. At each turn, emit one record with a stable schema. The fields below are the minimum that make drift computable rather than arguable.

Two decisions matter more than the rest. First, log guide_state_before, not the state after the turn. If you only store post-turn state you can never tell whether a topic was skipped because the orchestrator never advanced or because the agent ignored the instruction. Second, log guide_step_id from the orchestrator's perspective, separate from whatever topic the response turns out to be about. Those two values diverging is the definition of drift you are trying to measure.

If your runtime does not expose reasoning traces, log a short structured rationale field that the agent fills in the same response, for example "probing on onboarding friction because respondent mentioned a delay". That self-report is not ground truth, but it is far better than zero context, and it gives the post-call classifier something to compare against the guide step. Monitoring practice for agents treats behavior signals like step counts and policy-trigger rates as first-class metrics, alongside the usual performance and output families, per Collibra's guide to agent monitoring metrics.

Store turns in an append-only table, one row per turn, and never update a row after the call ends. Corrections go in a separate annotation table keyed by call_id and turn_index. This keeps the audit defensible: the record you audit is the record that was written in production.

A workable write path looks like this:

def log_turn(call_id, turn_index, guide, prompt, response, reasoning, latency_ms):
    record = {
        "call_id": call_id,
        "turn_index": turn_index,
        "timestamp_utc": utcnow_iso(),
        "guide_id": guide.id,
        "guide_version": guide.version,
        "guide_step_id": guide.current_step_id,
        "guide_state_before": guide.snapshot_before(),
        "prompt_sent": prompt,
        "response_text": response,
        "model_reasoning": reasoning,
        "latency_ms": latency_ms,
    }
    turn_store.append(record)  # append-only, no updates

Replay requires that the record contains everything needed to re-render the prompt. If the prompt is assembled from retrieved context (participant profile, prior answers, a knowledge base), store the retrieved chunk IDs and their hashes alongside prompt_sent. Without them a replay months later may render a different prompt against a changed knowledge base, and your comparison silently compares two different experiments. Static event correlation on individual rows cannot recover a behavior change that only appears in the sequence of turns, which is the argument runtime analytics tools make for capturing full execution paths, per Zenity's writeup on runtime AI analytics. The same logic applies here at smaller scale: keep the sequence, not just the rows.

Two practical rules for the store:

• Partition by date and index on (guide_id, guide_version, call_id, turn_index). Your audit queries are almost always "all calls under guide version X" or "all turns for one call in order."
• Keep raw and derived data separate. The classifier output belongs in a derived table so you can re-run a better classifier over old calls without touching the original record.

For privacy, store audio and verbatim transcript references behind the same retention policy as the raw logs, and redact direct identifiers in prompt_sent and response_text at write time. An audit trail that cannot be shared with a compliance reviewer because it is full of unreviewed personal data does not get used.

You do not need a sophisticated model to start. Three metrics, each computable in SQL plus one small classifier call, cover the common failure modes.

Topic match rate. For each turn, classify the response into one of the guide's topics and compare it to guide_step_id. topic_match = matches / total_turns. A healthy semi-structured interview will not score 100 percent, because probing inside the current topic is expected. Use it as a trend line, not a pass/fail gate, and set your threshold from your own baseline rather than from a published number.

Required-probe coverage. Some questions are mandatory; others are optional follow-ups. Tag every guide step required: true/false and compute coverage per call as required steps asked divided by required steps defined. This is the metric that matters for research integrity, because a missing mandatory probe is a data gap.

Guide-position monotonicity. Track the sequence of guide_step_id values across a call. Legitimate interviews move forward and sometimes step back one or two positions to revisit a topic. A jump of three or more positions with no intervening probe is a skip. Count skips per call as skips = count(i where position[i+1] - position[i] > 2 and no_bridge_between). A worked example: a 12-turn call with step positions 1,1,2,3,3,3,7,8,8,9,12,12 has two suspicious jumps (3 to 7 and 9 to 12), so skips = 2 even though the topic match rate may look fine if the classifier is lenient about the questions at positions 7 and 12.

Set alert thresholds from your own data. Pull the last 200 calls under a guide version, compute all three metrics, and take the 10th percentile of coverage and the 90th percentile of skips as your working bounds. Those bounds are yours, are defensible, and will not drift when someone else's benchmark changes.

Reasoning traces are the most illuminating field and the most misleading. They tell you why the agent says it asked a question, which is genuinely useful for triage: a rationale like "respondent mentioned a competitor, so I asked about switching costs" explains a one-step detour that the monotonicity metric would otherwise flag. Keep them.

But do not treat the trace as a description of what happened. Self-reported reasoning can diverge from the actual computation, and it cannot be audited at the same confidence level as the prompt and the response. The practical compromise is a tiered audit:

• Tier 1, deterministic: topic match, probe coverage, monotonicity. Computed on every call, no model required, cheapest.
• Tier 2, semantic: embed the response and the expected step description, compare similarity, and review the lowest-scoring turns. Run this only where Tier 1 flags a turn or on a random sample of clean calls to check for false negatives.
• Tier 3, human: reviewers read a small stratified sample plus every call that breached a Tier 1 threshold twice.

Runtime auditing research on multi-agent systems found that auditing every step is expensive, which is the motivation for budget-aware schedules that place audit points where unchecked exposure is highest, per the BRA-Audit paper on budgeted runtime auditing. The same cost logic applies to an interviewer: full semantic audit on every turn is not affordable at volume, so spend it where the cheap metrics point.

The audit becomes useful when it gates changes to the guide or the prompt, not when it produces a dashboard nobody opens.

Wire it as a step in your evaluation pipeline:

1. Run batch calls against the current guide version with scripted respondent personas, including at least one persona designed to pull the agent off topic.
2. Compute the three metrics per call plus the pooled averages.
3. Compare against the previous guide version and the previous prompt version. A drop in required-probe coverage larger than your own noise band blocks the release.
4. Store the audit snapshot with the commit hash so the number is reproducible.

The scripted personas do double duty: they are the regression test for drift, and they let you verify the classifier itself. Label 50 turns by hand, compare the classifier's topic labels to yours, and report agreement. When agreement is poor, fix the label set before you trust any drift number. This is the same discipline that keeps any human oversight meaningful rather than symbolic: reviewers need context and authority before the decision, not a rubber stamp afterward, per Sweet Security's overview of human oversight for AI agents.

If your interviewers run on several models, whether that is one hosted model per use case or a mixture of text and audio models for transcription and dialogue, keeping the same turn schema across providers is what makes the metrics comparable. Calling each family of model through one OpenAI-compatible endpoint keeps the logging wrapper in one place instead of five, which is where Quanta Inference fits if you are consolidating model access.

Suppose a 10,000-call study with a guide of 14 steps, 9 of them required. You turn on turn logging and run the audit.

• Total turns logged: 148,000, an average of 14.8 per call.
• Calls where required-probe coverage fell below 0.9: 1,340 (13.4 percent).
• Calls with at least one skip of three or more positions: 410 (4.1 percent).
• Turns flagged for Tier 2 semantic review: 9,800, roughly 6.6 percent of all turns.

You then read 100 of the 410 skip calls. In 70 of them the skip traces to one guide step: the agent advanced past the required pricing probe whenever the respondent gave a long answer to the preceding value question, because the orchestrator advanced on any response longer than a token threshold. That is a testable hypothesis. Change the advance rule to require an explicit completion signal from the agent, rerun the scripted battery, and confirm coverage on the skipping persona rises before you touch the live guide.

That loop, from log to metric to hypothesis to a single controlled change, is what the manual audit was never able to do reliably. If you want help designing the logging schema and the audit thresholds for your own interview or evaluation agent, a short AI discovery call is a reasonable place to scope it.