Adding Human Handoff to an Agent: Design and Metrics

The agent should not decide alone when the stakes are high. Put each trigger where it can be evaluated deterministically, and let the model call the tool only for the triggers that genuinely need judgment.

Explicit user request. Detect phrases like "human," "agent," "representative," or "supervisor" in the input, and treat them as a hard trigger. This is the one condition where you should not optimize for containment: every extra turn spent talking the user out of a person raises the chance they leave. If a request is detected, escalate on the same turn.

Low confidence. Confidence needs a source. Three practical options: a calibrated probability from the model's own logprobs on a classification step, a small scorer model that grades whether the draft answer is supported by retrieved facts using an evaluation pattern, or a deterministic signal like "retrieval returned zero documents above the similarity cutoff." Whatever you use, set the threshold from data, not intuition. Pull a sample of past conversations, label each as resolved or not resolved, and pick the cutoff that maximizes the number of real failures caught while keeping the escalation volume your staff can absorb. If you want to run both a classifier and a fallback model behind one key to keep the scorer and the agent on the same platform, Quanta Inference exposes text and embedding models through one OpenAI-compatible API.

Repeated failure. Count tool calls by name and arguments within a conversation. If the same call fails twice with the same error class, escalate rather than let the model retry a third time. This is a deterministic rule the runtime can enforce without asking the model, so implement it in code and tell the agent about it in the system prompt so it also cooperates.

Policy risk. Maintain a list of topics that always go to a person: billing disputes above a value cutoff, safety incidents, account deletion, legal threats, and anything touching a regulated process. Match on a classifier plus a keyword backstop. Risk-based routing, where review is reserved for high-impact or uncertain decisions rather than every decision, is the standard argument in the human-in-the-loop literature (Databricks): it is what makes oversight scale.

The record is the spine of the whole system. If it is a flat, well named row, routing, metrics, and audit all read from the same place. A workable schema:

Two decisions matter here. First, store a pointer to the transcript rather than duplicating it, or the row becomes a privacy liability. Second, keep failed_tool and confidence nullable and separate: a user-requested handoff has no failing tool, and conflating "no value" with "zero" will corrupt your aggregates.

This is the same discipline that makes general structured logging useful: key-value fields with consistent names across services let you filter, group, and aggregate instead of guessing where a value sits in a free-text line (SigNoz). The most commonly skipped step is field-name consistency, where one service logs userId and another logs user_id and no single query works across both (OpenObserve). Decide the names once, put them in a schema file, and enforce them at write time.

Instrument with OpenTelemetry so the handoff appears inside the same trace as the model call and tool calls that led to it, whichever backend you use (OpenTelemetry). A trace is a tree of spans that share a trace ID, each carrying attributes and events, which is exactly the shape you need to answer "what happened in the two turns before this person got involved" (traces).

Use the GenAI conventions where they already fit: gen_ai.conversation.id binds every span across every trace in one conversation, gen_ai.agent.name identifies the agent, and gen_ai.operation.name classifies the span as a chat, tool, or agent invocation (Honeycomb). The full attribute registry lives with the GenAI semantic conventions (semconv).

For the handoff itself, create a dedicated span named escalate (or handoff) as a child of the agent invocation span, and add these attributes:

with tracer.start_as_current_span("handoff.escalate") as span:
    span.set_attribute("gen_ai.conversation.id", conversation_id)
    span.set_attribute("handoff.id", handoff_id)
    span.set_attribute("handoff.reason", reason)          # the enum
    span.set_attribute("handoff.priority", priority)
    span.set_attribute("handoff.topic", topic)
    span.set_attribute("handoff.confidence", confidence)  # omit if None
    span.set_attribute("handoff.queue_depth_at_assign", queue_depth)
    span.set_attribute("handoff.skills_required", ",".join(skills))
    span.add_event("handoff.recorded", {"handoff.id": handoff_id})

When the person picks up, close the loop with a second span (handoff.pickup) carrying assignee_id, handoff.wait_seconds, and whether the routing function fell through to a fallback. Attributes are cheap and set per event; use a new span only for units of work that need a duration or a boundary (Honeycomb). Time-to-pickup is a duration you want to measure, so it earns a span; the reason code is a fact about one event, so it earns an attribute.

One caution the OpenTelemetry GenAI docs make explicit: prompt content and tool arguments are not captured by default because they can hold sensitive data, and enabling content capture populates spans with full messages and tool results (GenAI observability). Handoff summaries are user content. Set the summary as an attribute only if your backend and retention policy allow it, and otherwise keep it in the record store behind transcript_ref.

Traces and logs do different jobs here. Logs record discrete events and are fine for audit; traces preserve the relationship between steps, which is what you need because an agent can take a different path on every run and a failing turn cannot be reconstructed from an interleaved log stream (Braintrust). Keep both, joined by handoff_id.

Routing is a ranking problem, and you should solve it as a scored function rather than a chain of if-statements. For each candidate assignee, compute:

• Skill match: 1.0 if the assignee holds every required skill for the topic, 0.6 if they hold the primary skill, 0 otherwise. Derive required skills from topic and the classifier, not from free text.
• Availability: 1.0 if online and under their concurrent-chat cap, 0.5 if online but at cap, 0 if offline. Tracking a per-assignee cap is what stops your best agent from being handed six chats at once.
• Load: normalize current open handoffs against the team median, so a person sitting at twice the median scores lower.
• Recency and continuity: small bonus if the assignee handled this conversation or this account before, which cuts re-explanation time.

A simple weighted sum (skills 0.45, availability 0.30, load 0.15, continuity 0.10) is enough to start. Sort candidates, then apply priority as a queue-ordering rule rather than a scorer weight: p1 handoffs skip the line, p3 handoffs wait for capacity. If you want the weights to respond to outcomes later, log the chosen assignee and the resolution on every handoff so you can fit the weights from your own data instead of arguing about them.

The fallback path is not optional. If no candidate scores above zero on availability, do not leave the user hanging or silently keep them with the agent. Pick one of three explicit behaviors and tell the user which one applies:

1. Callback: record the handoff, tell the user when a person will reach out, and stop the agent from making commitments.
2. Asynchronous ticket: convert the handoff into a queued item with the summary and blocking facts attached, and give the user a reference number.
3. Deferred with status: hold the conversation open, tell the user the expected wait, and let them leave a message.

Whichever you choose, set handoff.routing_fallback on the span so the fallback rate is visible. A rising fallback rate is your first signal that staffing and escalation volume have diverged.

Instrument against service level indicators built from what users experience, not from infrastructure health, and set objectives against those indicators with an explicit error budget (OpenObserve). The vocabulary is worth keeping straight: the SLI is the measurement, the SLO is the target over a window, and the error budget is one minus the SLO (SRE golden rules). Latency figures are the clearest example of why the distinction matters: an average latency can look healthy while one request in twenty is slow, which is why you set targets on a high percentile rather than the mean (Nova AIOps).

Track these as your handoff dashboard:

Two of these are the ones most teams forget. Agent turns before handoff exposes the failure mode where your low-confidence threshold is set so conservatively that users exhaust themselves before the agent escalates. If the median is above three turns for user-requested handoffs, your detection is late. Bounce rate is the quality check on the summary field: a high bounce rate on policy-risk handoffs usually means the summary omits a blocking fact the human had to rediscover.

Set the objective where it belongs. A defensible starting SLO is "95 percent of p2 handoffs are picked up within five minutes," measured on the trace data, with alerting on the burn rate rather than a single threshold breach. Review escalation rate by reason monthly. If one reason dominates and the resolved rate after handoff is high, that reason is a candidate for the agent to learn to handle, which is the point of the loop: every human correction is training data for the next version of the agent, not a dead end in a review queue (Databricks).

Take a support agent that handles 10,000 conversations a day. You label a 1,000-conversation sample and find 180 unresolved. You score each with your confidence signal and get this table:

Setting the threshold at 0.7 escalates 550 conversations in the sample, or 55 percent, and catches 159 of the 180 failures, about 88 percent. At 10,000 daily conversations that is 5,500 handoffs a day, which almost certainly exceeds your staffing. Setting it at 0.4 escalates 20 percent and catches 96 failures, or 53 percent, leaving 84 bad conversations a day with no human. Neither number is right in the abstract; the decision rule is to choose the highest threshold whose daily escalation volume your team can absorb at the pickup SLO, then close the gap by adding the deterministic triggers (repeated failure, policy risk) that catch failures confidence misses at low volume.

That is the split most teams get wrong: they tune the confidence threshold to fix quality, when the threshold is really a staffing dial. Repeated-failure and policy triggers are quality controls, because they cost almost nothing in volume and catch the failures that a confident model gets wrong.

Run the scoring model and the escalation classifier side by side while you tune, then move the winning configuration into production. When you are ready to put an expert, not just a threshold, on a business line, Quanta Skills is the library behind every Quanta expert, installable into a coding agent or staffed by a person.