How to Turn Routing Logs into a Product Feedback Loop

Log at the message level, not the conversation level. A single conversation can contain a clean routing decision followed by a miss, and averaging them destroys the signal. The table below is a practical minimum schema; add fields rather than remove them.

Two fields do more work than the rest. fallback_reason should be a small closed set: no_intent_above_threshold, below_confidence_threshold, no_knowledge_match, tool_error, policy_block. That last one matters: if your assistant refused to answer because of a guardrail, that is not a content gap and should never be routed to a writer.

candidate_intents is the field most teams skip and later regret. When the router picks billing.refund at 0.38 confidence and the true intent was billing.duplicate_charge, the fix is usually to sharpen the two intent descriptions, not to write a new article. Without the candidate list you cannot see that.

If you run an assistant across several model providers, the schema stays stable regardless of which model answered; one OpenAI-compatible endpoint such as Quanta Inference keeps the logging contract identical no matter which model is behind it.

{
  "message_id": "msg_8f31a",
  "conversation_id": "conv_2c9",
  "timestamp": "2026-03-04T14:22:07Z",
  "channel": "in_app_chat",
  "raw_query": "why did my card get hit twice",
  "normalized_query": "duplicate charge",
  "chosen_intent": null,
  "confidence": 0.41,
  "fallback_reason": "below_confidence_threshold",
  "candidate_intents": [
    {"intent": "billing.duplicate_charge", "score": 0.41},
    {"intent": "billing.refund", "score": 0.38}
  ],
  "retrieved_articles": [],
  "response_outcome": "escalated",
  "user_feedback": "thumbs_down"
}

A worked sizing example. Suppose you process 60,000 assistant messages a month. A 12% message-level fallback rate yields 7,200 fallbacks. If 40% of those turn out to be duplicates of each other once you group by normalized_query, you are looking at roughly 4,300 distinct fallback queries, which is far too many to fix one by one and entirely manageable if you rank them by frequency and sort them into the four cause buckets below.

A flat list of fallbacks sorted by frequency is misleading, because the top of the list is usually dominated by a few channels or a few broken intents. Segment first, then classify inside each segment.

Three segmentations earn their keep:

By intent, using candidate_intents when chosen_intent is null. Group fallbacks by the top candidate intent even if it failed the threshold. This tells you which intents are chronically under-confident, which is a threshold or training-example problem, not a content problem. If you have no candidate list, fall back to clustering the raw queries, but you lose the direct link to your taxonomy.

By channel. An in-app chat sees short, context-carrying messages ("it still won't work") that a web widget will never see. Pooling them inflates the fallback rate of the well-performing channel and hides the bad one.

By conversation position. First-message fallbacks are usually genuine coverage gaps. Fallbacks on turn three or later are usually a recovery-flow problem: the assistant failed to ask a clarifying question and the user rephrased. A recovery flow that offers clarifying questions before handing off is a known fix for this pattern, according to the same fallback-rate guide.

A quick way to run the first two segments without a BI project:

SELECT
  channel,
  COALESCE(chosen_intent, candidate_intents[1].intent, 'unrouted') AS likely_intent,
  fallback_reason,
  COUNT(*) AS messages,
  COUNT(DISTINCT normalized_query) AS distinct_queries
FROM routing_log
WHERE fallback_reason IS NOT NULL
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 28 DAY)
GROUP BY 1, 2, 3
HAVING COUNT(*) >= 20
ORDER BY messages DESC;

The HAVING clause is a deliberate choice. Ignore segments under about 20 messages for the first pass; they are noise at the individual level and occasional gold at the aggregate level. Run the full dataset through the cause classifier later so nothing gets permanently lost.

This is the heart of the loop. Every fallback or miss gets exactly one cause label, and each label has a default owner. The four buckets are not equally common or equally expensive, and conflating them is why feedback loops stall.

Missing help article. The signals are a confident intent and an empty retrieval list. The fix is content, and the quality bar is set by how retrieval works: content is chunked into smaller pieces, embedded, and matched semantically, so an article that buries the answer under three headings will be retrieved badly or not at all, as Zendesk's guidance on optimizing content for generative AI explains. That guidance is specific and worth applying at the point of writing: focus each article on one topic, answer the question directly, make the article self-contained, repeat the title's topic in the body, and avoid vague pronouns. Chunking choices also matter: a NVIDIA study of chunking strategies found page-level chunking gave the highest average end-to-end RAG accuracy at 0.648 with the lowest standard deviation, while very small and very large chunks generally underperformed. Translation: keep article sections coherent and self-contained rather than chopping them mid-thought.

Broken feature. The tell is a tight cluster of queries about one action, usually with error language, and often a spike that starts on a specific day. This is a gift: your assistant is surfacing a product defect before it reaches your bug tracker. The action is to file it with the transcript attached, then watch the same cluster disappear. If it does not disappear after the fix, either the fix did not stick or the users are describing a different problem, which means reclassify.

Ambiguous phrasing. Confidence is low, two or three candidates are close, and the top intent is not obviously right. The cause is usually the taxonomy, not the user. The standard remedy is a clarifying question, and intent-design practice supports keeping intent sets tight and distinct: one intent-recognition guide suggests starting with 30 to 40 intents and expanding to 60 to 80, with 80 to 100 diverse training examples per intent, distinct categories and a fallback intent for unmatched messages, per IrisAgent's intent detection guide. It also notes that putting fewer than about 20 intents in a single prompt helps, which is relevant if your router is one big prompt.

Product should answer itself. These are the highest-leverage misses. The user asks "what's the status of my order" or "why was I charged this amount," and the assistant, lacking a lookup, either falls back or writes an article-shaped answer that is half right. Cheap models with poor confidence can also trigger this: the model cascade research discussion points out that low thresholds mean lower cost but also raise the question of whether a small model's confidence score can be trusted at all. If the question is about live account state, no article will ever be enough. The fix is a tool call, and the classification guide above is right that routing a request into two simpler decisions, first what type it is and then how to handle it, beats asking one model to do everything.

Classification itself can be done cheaply. A first pass with a small model and a fixed label set works, and the labels are stable enough that a fine-tuned classifier would eventually be cheaper; the cost-aware model selection paper found fine-tuned encoder models from the BERT family were competitive with or better than prompted LLMs on fixed-label classification while running at one to two orders of magnitude lower cost and latency. You do not need that on day one. You need labeled examples, and your first few hundred manual labels become the training set.

The loop that actually closes looks like this, and it works on a sprint cadence:

1. Pull the last four weeks of fallback and miss events, segmented by channel and likely intent.
2. Cluster the distinct normalized_query values and rank by message volume, not by distinct query count. A query used 900 times matters more than a query used once.
3. Label the top 30 clusters with one of the four causes. Thirty is a real number here: it fits in one sitting and usually covers the majority of fallback volume.
4. Assign each cluster an owner and a definition of done. For a content fix, done means "the exact logged queries now resolve." For a feature fix, done means "the cluster volume drops below baseline."
5. Ship the top three or four fixes.
6. Re-run the same queries through the assistant and compare. Keep the queries as a permanent regression set.

A worked example makes the math concrete. Say the four-week sample contains 7,200 fallbacks. Clustering gives 1,450 distinct queries. The top 30 clusters account for 3,900 messages, about 54% of the volume. Labeling them yields: 14 missing articles (2,600 messages), 6 ambiguous phrasing cases (700), 5 broken features (400), and 5 product-should-answer cases (200). The missing articles are the biggest volume but the cheapest to fix; the broken features are the smallest volume but the highest severity, because each one is likely generating escalations. That mismatch between volume and severity is the insight most frequency-sorted dashboards miss, and it is why the owner column belongs next to the count.

Two guardrails on measurement. First, watch ticket deflection alongside resolution quality, not instead of it. Deflection measures how often people resolve without a ticket, and a high deflection rate has to be paired with strong satisfaction and first-contact resolution to mean anything real, as Crisp's guide to ticket deflection notes. If your fixes push the fallback rate down and CSAT down with it, you have taught the assistant to answer confidently and wrongly.

Second, watch model drift on the routing layer itself. Products change, policy changes, and customer language changes, and the AI can lose its edge as a result; catching that early is described as the core job of the person who owns assistant performance in Ada's guide to building an AI customer service team. The routing log is how you catch it: a sudden jump in one intent's fallback rate is drift, not a content gap.

If your team does not have someone whose job is reading these logs, that is the first fix. The role does not need to be new headcount; it needs to be named. A practical pattern is one owner for the log-to-label step and one owner per cause bucket, with the log's top 30 clusters reviewed on the same cadence as your support queue. When the volume of interpretation work outgrows the team, a specialist assistant that reads the transcripts and drafts the cause labels and the rewritten articles is the natural next step, and that is roughly the shape of the work Quanta Experts is built for.

Logging everything feels safer and is almost always a mistake, because storage is cheap and attention is not. Three fields routinely earn their removal:

Full model responses. Keep a response ID and a token count, not the whole generated text, unless you have a specific evaluation that needs it. The transcript system almost certainly has it already.

Per-token debug traces. Useful during a single debugging session, noise in a product analytics table. Put them behind a flag.

Free-text agent notes with no schema. A column called notes that says "weird one" 400 times has negative value: it looks like data and cannot be grouped. If reviewers need to add context, give them a controlled vocabulary plus one short free-text field.

What should stay, even if it looks redundant: user_feedback, because it is the cheapest way to catch a confident wrong answer, and response_outcome, because it is the only field that distinguishes a fallback that got rescued by a clarifying question from a fallback that became a ticket.