What to Log for Every AI Support Turn

Use an ID that is sortable and unique. A ULID or UUIDv7 sorts by creation time, so you can list a customer's sessions in order without a secondary sort on a timestamp column that may carry clock skew. Set the ID when the session opens, not when the first model call returns, so that a turn that times out still gets logged under the right session.

The session ID does three jobs. It groups turns for replay, it joins the turn log to the human handoff and the eventual ticket, and it lets you measure repeat contact: a customer who chatted at 10:00 and called at 10:40 can be linked if your channel layer carries the session ID into the ticket. That last job matters more than it looks. Contact centers that use a loose definition of containment, counting any session that ended without an escalation, overstate performance because frustrated abandonments look identical to resolutions; reclassifying sessions followed by a new contact on the same issue within 24 hours as unresolved can lower reported containment by 5 to 15 percentage points the Decagon containment glossary.

Two rules for the schema:

• Never key on customer PII. Use an internal customer or account ID, and keep email and phone out of the event body. Storing less in the log reduces the blast radius of any leak.
• Log the turn when the decision is made, not when the reply is sent. If you write the record before the network call, a failed send still leaves evidence of what the agent intended.

The retrieval step is where many support failures originate. A team that switched from pure generative responses to a retrieval-first design that referenced real resolution patterns from past tickets reduced incorrect responses by 60% the Bland.ai challenges post. You cannot get that benefit, or protect it, if you do not log what was retrieved.

Log three things per passage: the document or chunk ID, the relevance score your retriever returned, and the rank. Do not log the passage text in the event. Store text once in the knowledge base with a version or content hash, and reference it from the event. That keeps events small, avoids duplicating sensitive content into your analytics store, and lets you reconstruct what the model saw as long as the knowledge base version is pinned.

{
  "session_id": "sess_01J8ZK4M2Q",
  "turn_index": 3,
  "timestamp": "2025-11-04T14:22:07Z",
  "retrieved": [
    {"doc_id": "kb_4471", "version": "v12", "rank": 1, "score": 0.81},
    {"doc_id": "kb_1180", "version": "v12", "rank": 2, "score": 0.64},
    {"doc_id": "kb_2205", "version": "v12", "rank": 3, "score": 0.31}
  ],
  "confidence": {"value": 0.42, "source": "posterior"},
  "action": "clarify",
  "handoff_reason": null
}

With this shape you can answer questions that a prompt log cannot. Which document is most often retrieved for turns that end in escalation? When the top score is below 0.5, what share of turns end in a clarification? Did a knowledge base edit on a given day coincide with a rise in escalations for one product line? The last question catches stale documentation, which is a data failure rather than a model failure. Most AI support failures trace back to the agent acting on stale customer history or an out-of-date knowledge base, which makes support automation a data observability problem as much as a conversational one the Thinklytics deflection analysis.

Your knowledge base needs the same care. Teams that audit their top 20 to 30 most common questions and write a dedicated article for each, in the customer's language rather than internal jargon, give the retriever something worth returning the Pylon deflection guide. A good retrieval log tells you whether you did that.

Confidence is the field teams get wrong most often. There are at least four different things people call confidence, and they are not interchangeable:

• Retrieval confidence: the top relevance score, or the gap between the first and second passage.
• Model token probability: the probability the model assigned to the tokens it generated.
• Posterior or ensemble score: a probability derived from several samples or several models agreeing.
• Judge score: a second model or classifier rates whether the answer is supported by the retrieved passages.

Each one means something different, so the schema needs a source field next to the value. Raw token probabilities are not calibrated probabilities of correctness. ING's wholesale banking analytics team contributed functionality to the transformers package to extract the raw probabilities of generated text so they could calculate a confidence score for answers in a retrieval-augmented question-answering pipeline, and they still required expert validation of every extracted answer the ING confidence score write-up. Treat the logged number as a ranking signal to improve, not a guarantee.

The practical use is threshold tuning. Suppose you log 10,000 turns in a month and bucket them by confidence source and value. You find that turns with a retrieval top score under 0.45 that were answered anyway have a repeat-contact rate three times the average. That gives you a decision rule with numbers behind it:

Decision rule: if action == "answer" and the top retrieval score is below your measured floor, require a supported-answer judge score above your threshold, or switch the action to clarify. Re-measure the floor after every knowledge base refresh and every model change.

Escalation design in general rests on the same signals: ambiguous intent, missing required parameters such as an order number, contradictory statements, and a clarification loop that produces no progress. When the agent cannot confidently classify a request or keeps asking without moving forward, continuing wastes the customer's time. If your clarify turns never convert to answer within two or three tries, the log will show it.

action is the enum that makes the log aggregatable. Three values cover almost everything:

• answer: the agent gave a substantive reply grounded in retrieved content.
• clarify: the agent asked a follow-up question because something was missing or ambiguous.
• escalate: the agent moved the conversation to a human or created a ticket.

Add decline only if you have a policy path where the agent refuses a request for reasons other than ambiguity, such as a request outside the product scope.

handoff_reason is null for every turn that is not an escalation, and a value from a short fixed enum for every turn that is. Start with a list of about eight and resist adding to it casually:

The reason enum does triple duty: it drives routing, it drives reporting, and it gives the human agent a one-line context header. A handoff should carry customer identity signals, relevant timestamps, the exact request, prior attempts and extracted constraints, so the human does not start from zero. Zendesk's workflow for collecting customer information before escalating does the same thing in a platform setting, with the AI agent gathering details into a form so human agents have what they need before the request reaches them the Zendesk workflow recipe.

A caution on counting. Industry writing on deflection makes the point that a contact counts as deflected only when the underlying issue is actually resolved, not when the customer abandons the interaction before reaching a human the Decagon deflection glossary. A high automation rate next to a high repeat-contact rate is a warning, not a win the Thinklytics analysis. The handoff reason enum lets you separate the two: escalations with explicit_request after a clarify loop are a different story from escalations with policy_authority on the first turn.

Logging is only worth the storage if you do something with it. Four routines cover most of the value.

Replay a bad turn. Pull the session ID, list turns in order, and reconstruct each one: retrieved doc IDs and versions, scores, confidence, action. For a single confusing run this is the fastest path to the truth, because agents are non-deterministic and replaying the same input may not follow the same path the groundcover guide. The log is the only record of what actually happened on that request.

Attribute the failure to a layer. Retrieval returned nothing relevant, so fix content. Retrieval was fine but the answer invented a figure, so fix grounding and add a supported-answer check. A tool timed out, so fix integration reliability. Routing sent the request down the wrong branch. When routing, retrieval, model calls and tool calls show up as separate steps, you can tell whether the fix belongs in a prompt, a data source, a tool API or the workflow the groundcover guide. This is the difference between saying the bot is bad and assigning a ticket to the right team.

Weekly review of escalations by reason. Count turns by handoff_reason and week. If repeated_loop is rising for one intent, that intent's knowledge needs work. If tool_failure spikes, talk to the integration owner. If policy_authority dominates, the agent may be attempting decisions it should route cleanly from the first turn, which is fine as long as the handoff carries context.

Quarterly calibration check. Join turns where action == "answer" to repeat contact within 72 hours and to any later satisfaction signal. If your confidence buckets do not separate good from bad outcomes, your confidence source is not earning its place in the schema. Recalibrate or replace it.

Two implementation notes save pain later. First, write events through an async queue so a slow logging backend never delays the customer's reply, and sample nothing at the decision level: log every turn's action and handoff reason, and sample only verbose payloads if volume forces it. Second, keep the schema versioned in the same repository as the agent code, so a change to confidence computation ships with a change to the schema version.

Use this list in order. Each item is verifiable in staging before production.

The replay tool is the piece teams skip and then regret. It is a day of work, and it turns every future customer complaint into a five-minute investigation instead of an argument about what the bot said.

If you are running this across several support intents and want the schema to survive a model change without a rewrite, it helps to route all model calls through one interface so the logging layer does not change when the backend does. Quanta Inference puts text, image and embedding models behind one OpenAI-compatible API key with prepaid US dollar pricing, which keeps the logging layer stable when you swap models: Quanta Inference.