How to Build a Retrieval Set for Customer Support Triage

A working triage index usually mixes three document types, and they need different treatment.

Policy documents (refund rules, escalation matrices, acceptable-use terms, regional consumer rights) answer the question "what are we allowed to do." They are the highest-stakes corpus because a wrong citation creates a legal or financial exposure.

Help articles answer "how does the product work." They are the most volatile corpus: they change with every release and every UI change.

Resolved tickets answer "what did we actually do last time." They are the only corpus that captures judgment calls the written policy does not cover, such as how much goodwill credit a particular issue type usually attracts.

A useful rule of thumb: if a corpus item has no owner and no change trigger, it will rot. Policy chunks need a named owner in legal or compliance, help chunks need a trigger on the release process, and ticket chunks need a rule that excludes reopened or reversed resolutions.

Retrieval hands the model a passage with no surrounding document. If the passage says "the 30-day window above does not apply to digital goods," the model has no idea what window, which product, or which jurisdiction. The fix is structural chunking: split on the document's own hierarchy so the heading travels with the body.

SurrealDB's survey of chunking strategies makes the tradeoff explicit. Fixed-size chunking is simple and fast but "completely ignores sentence and paragraph boundaries" and can split mid-sentence or mid-idea. Document-structure chunking, which splits on headers and sections, is slower to prepare but produces chunks that match how the document is actually organized SurrealDB. For policy documents, where a section is already the unit a human would cite, structure chunking wins.

A practical policy-chunk recipe:

1. Split at heading level, using the deepest heading that still covers one rule.
2. Prepend the heading path to the chunk text, for example "Refunds > Digital goods > EU consumers:" so the passage restates its own scope.
3. Keep the effective date in the text or metadata, and keep superseded versions retrievable but filtered out by default.
4. Cap the chunk at the point where a single section covers more than one distinct rule. Two rules in one chunk means the model can cite the passage for the wrong one.

Here is what a policy chunk looks like once prepared:

{
  "source_id": "policy-refund-0042-v3",
  "doc_title": "Refund Policy",
  "heading_path": "Refunds > Digital goods",
  "effective_from": "2025-03-01",
  "product": ["streaming", "ebooks"],
  "region": ["US", "CA"],
  "plan": ["consumer"],
  "text": "Refunds > Digital goods. Digital goods are refundable within 14 days of purchase only if the download has not been started. If the download was started, no refund applies unless the file is defective."
}

The AWS Bedrock documentation notes that when documents are ingested, content is split into manageable chunks, converted to embeddings, written to a vector index, and kept mapped back to the original document AWS documentation. That mapping is your citation trail, so never let a chunk lose its pointer to the parent document and heading.

Help articles need a different rule. Split at step boundaries so a procedure step does not arrive without the step before it. For FAQ-style articles, split per question and answer pair. A tutorial with eight numbered steps should produce eight chunks plus one overview chunk, not nine arbitrary token windows. The general guidance for prose knowledge bases favors sentence-aware or recursive splitting that respects natural language boundaries, which is close to what help articles need at the sentence level, while section boundaries should be respected at the article level Medium.

Tickets are the corpus most teams get wrong, because the obvious approach (dump the whole thread) produces long, noisy passages full of greetings, apologies, agent names and internal notes. The model then retrieves the noise and cites the wrong thing.

Store each ticket as a pair: the customer's problem statement as written (cleaned of PII) plus the resolution that actually closed it. Keep the subject line attached, because it is often the densest summary of the issue available.

Chipper Cash's fraud email triage pipeline is a useful model. Before automation, an analyst read each alert, cross-referenced accounts across three dashboards and copy-pasted data by hand, at roughly 30 minutes per email. The rebuilt pipeline cleans HTML, strips disclaimers and links, and sends the cleaned text to a model with a strict JSON schema; it processes each email in under 20 seconds, with every action writing an audit trail Chipper Cash. Two lessons carry over directly to ticket ingestion: clean the text before it ever reaches the index, and make the model's output schema strict so downstream routing does not have to guess.

For a ticket chunk, keep:

• the subject line
• the customer's original problem description, trimmed to the relevant part
• the resolution text, and whether it was a refund, replacement, workaround, escalation or no-fault answer
• the outcome flag: resolved, resolved with reversal, reopened, or closed without resolution

Exclude internal notes by default. If an internal note explains the resolution, summarize it into the resolution field rather than appending the raw note, so the chunk reads as a clean precedent. Exclude tickets closed without resolution from the precedent corpus entirely; a no-resolution ticket teaches the model nothing useful and can teach it to give up.

The multi-evidence clinical triage study published in JMIR Medical Informatics is worth knowing about here. It built a retrieval-augmented system over a 3,000-case database for emergency triage and compared it against a prompt-only baseline, evaluating five-level triage classification with confusion matrices and quadratic weighted kappa ScienceDirect. The transferable insight is the evaluation shape: build a fixed test set of past cases with known correct triage levels, and compare the retrieved verdict against the human verdict using a graded agreement metric, not just a right or wrong count. Support triage benefits from the same treatment because the severity of a misroute varies.

Metadata is where support retrieval diverges most from general document search. A generic RAG pipeline embeds everything and hopes the query matches the right passage. A support pipeline can do better, because it already knows facts about the ticket before it searches: which product, which plan, which region, which channel, which customer tier.

Add filters on at least these dimensions:

Two rules keep filters honest. First, filter before you rank, not after. If you retrieve the top 20 passages and then filter to region, you can end up with two results, and one of them may be irrelevant. Filter first so the ranking has a clean candidate pool. Second, make the absence of a match a valid outcome. If a US consumer asks about a policy that only exists for EU consumers, the model should return "no applicable policy found, escalate" rather than borrowing the EU text because it scored highest.

Chipper Cash's pipeline shows the conservative version of this principle in practice: its action flags for restricting a user or placing a lien are set to true only when the email explicitly requests it, and ambiguous language routes the case to manual review, because delaying action is preferable to locking the wrong account Chipper Cash. Apply the same asymmetry to refunds and account actions: when the retrieved evidence is thin or contradictory, triage to a human instead of guessing.

A support verdict without a citation is a rumor. The citation is what lets a supervisor audit the answer, lets a customer see the rule, and lets the next agent trust the handoff.

Build citation into the data model rather than the prompt. Every chunk carries a source_id that resolves to a human-readable location: document title, heading, and if useful, a URL and anchor. Then require the model's output schema to include a citation field that must reference one of the retrieved source_ids. If the model returns a citation that is not in the retrieved set, treat the response as a failed generation and retry or escalate.

MLflow's guidance on structured outputs maps directly here. It recommends schema-constrained generation or function calling as the default over prompted JSON, paired with runtime validation and a fallback path, and notes that schema enforcement guarantees syntactic validity but not value correctness MLflow. For citations, that distinction matters: a schema can force the model to emit a source_id, but only your validator can confirm the source_id was actually retrieved and actually supports the claim.

A minimal verdict schema looks like this:

{
  "issue_type": "refund_not_received",
  "severity": "low",
  "route_to": "billing_ops",
  "recommended_action": "issue_refund",
  "citation": ["policy-refund-0042-v3", "ticket-2023-1187"],
  "confidence": 0.82,
  "escalate": false
}

Two validation checks after generation:

1. Every value in citation exists in the retrieved candidate set for this query.
2. The recommended_action is consistent with at least one cited policy chunk, checked by a second small model or a rules table that maps policy clauses to allowed actions.

The second check is cheap and catches the most damaging error class: a correct-looking answer citing a real document that says something else. If you want a single, consistent endpoint for the extraction model, the router model and the validation model, Quanta Inference exposes text, image, audio and embedding models through one OpenAI-compatible API key, which keeps a multi-model pipeline to one integration.

You cannot improve what you do not measure, and a support retrieval system fails in two different places. Either the right passage was never retrieved, or it was retrieved and the model wrote something the passage does not support. Evaluate those separately.

Evidently's RAG evaluation guide splits the work exactly this way: retrieval quality is measured with ranking metrics such as recall@k against ground truth, or with manual and LLM-judged relevance scoring, while generation quality is measured with reference-based comparison to correct answers and reference-free checks for faithfulness and completeness Evidently AI. For support triage, recall@k is the metric that matters most, because a routing decision built on the wrong precedent is worse than an escalation built on no precedent.

A practical test set for a support desk:

1. Pull a few hundred real tickets that were resolved by experienced agents, and label the correct issue type, route and action for each.
2. For each, mark which policy chunk and which ticket precedent should have been retrieved.
3. Measure recall@5: how often the correct policy chunk appears in the top five results after filtering.
4. Measure citation validity: how often the model's citation is in the retrieved set and consistent with the action.
5. Track escalation rate on the test set. A model that escalates everything scores well on safety and badly on usefulness; track both.

Synthetic question-answer pairs generated from the knowledge base can bootstrap the test set, and stress and adversarial testing should cover the cases where customers argue, quote a policy out of context, or ask for something outside the plan. A prompt that says "always cite policy" is not a control; a validator plus a test set is.

For teams that want the retrieval and evaluation harness built and maintained by specialists rather than staffed internally, a free discovery call with Quanta Consulting is a low-cost way to pressure-test the design before committing engineering time.

The index is a living object, and most failures show up after week one. Build the operating loop at the same time as the first index.

• Change triggers. A policy edit should invalidate and re-chunk the affected section the same day. Tag chunks with effective dates so a stale version can be filtered out before a fresh one is live.
• Precedent decay. Ticket precedents lose value as policies change. Review precedents older than your policy cycle or, better, filter them by effective date alongside policy chunks.
• Citation review. Sample a fixed number of automated verdicts per week and check the citation against the action. Log failures by source_id so you can see which documents cause the most bad citations.
• Schema drift. Log raw and parsed outputs and set validation failure metrics, per the MLflow recommendation, so a model upgrade or prompt edit that breaks the schema shows up as a metric rather than as a customer complaint MLflow.
• Human in the loop by design. Keep the escalation route visible in every verdict. The goal is not to remove the agent; it is to give the agent the passage and the precedent, so the decision takes seconds and the evidence is already attached.

The pattern generalizes beyond refunds and logins. The same model plus retrieved context plus verdict structure works for abuse review, chargeback intake and fraud first response, which is why the payments research on grounded classification, RAG evaluation and structured output all points in the same direction. Build the retrieval set once, keep it clean, and the model's job becomes small and auditable.