Turning Support Ticket Edits Into a Fine-Tuning Set

The comparison itself is mechanical. Sentence-level or line-level diffs are enough. What you need from each diff is a structured record: which spans were deleted, which were inserted, and what the replacement changed in meaning.

A simple extraction pipeline looks like this:

from difflib import SequenceMatcher

def diff_record(draft, sent):
    sm = SequenceMatcher(a=draft, b=sent, autojunk=False)
    changes = []
    for tag, a0, a1, b0, b1 in sm.get_opcodes():
        if tag == "equal":
            continue
        changes.append({
            "op": tag,                     # replace, delete, insert
            "before": draft[a0:a1],
            "after": sent[b0:b1],
            "position": a0,
        })
    edit_ratio = 1 - sm.ratio()
    return {"changes": changes, "edit_ratio": round(edit_ratio, 3)}

Run this over your captured pairs and store one row per ticket. The edit_ratio gives you a cheap first-pass score: a ratio near zero means the agent sent the draft essentially unchanged, and a high ratio means substantial rework. That number alone is useful as a baseline before you label anything.

The pipeline has two failure modes worth handling up front. Agents who paste the draft into a different editor and then rewrite it produce noisy diffs that mix real corrections with reformatting. And agents who delete the draft entirely and start over give you a diff that says "everything changed," which is a signal about prompt quality but a poor training example. Tag both cases and exclude them from fine-tuning data unless a reviewer salvages them.

Privacy belongs in this step too, not later. Customer data flowing through AI workflows raises exposure, consent and retention concerns that are much easier to satisfy at ingestion than after a dataset has been replicated across a training pipeline (Glean on customer data privacy in AI). Redact personal identifiers at capture, keep the original pair in a restricted store, and run training only on the redacted copy. Permissions-aware architecture from the start is cheaper than retrofitting it.

Not every edit teaches the same lesson. A four-way label does most of the work.

The distinction between tone and preference is the one teams get wrong. Style guidance in support writing is real and documented: customer service language guidance routinely flags dismissive phrasing such as "you're mistaken," "calm down," and "that's our policy" as trust-destroying, and recommends specificity, ownership and clear next steps instead (Gorgias on customer service phrases). An edit that removes "unfortunately, you are mistaken" is a tone correction with a defensible rationale. An edit that swaps a synonym is a preference. Train on the first and you get a more respectful model. Train indiscriminately on the second and you teach the model to imitate one agent's typing habits.

Labeling mechanics are simple enough to run inside your helpdesk review step or a small internal tool. Present the diff, ask for one primary label plus an optional secondary, and require a one-line rationale for factual and policy labels. The rationale is what lets you turn labels into prompt changes without re-reading the diff months later.

Labeling is also human work, and the humans doing it deserve care. Public litigation has alleged psychological harm to workers repeatedly exposed to distressing content while labeling AI models (Epstein Becker Green on AI model training and workplace implications). Support diffs are usually benign, but abusive customer messages are not, so give reviewers the option to skip tickets flagged as hostile and keep the review queue short.

Edits are not ground truth. Agents make mistakes, apply outdated policy, or respond to one customer's mood in a way that should never be standardized. A senior reviewer needs to approve, reject or reclassify each sample before it reaches a dataset.

Run the queue on a fixed weekly cadence with a small decision rule per sample:

• Approve when the correction is defensible against the knowledge base or policy document, regardless of who made it.
• Reject when the "correction" is wrong, contradicted by policy, or purely stylistic.
• Reclassify when the diff contains more than one type of change; split it into separate samples.

Keep the reviewer pool small and rotating so one person's preferences do not define the target behavior. Two reviewers on a sample of 20 diffs per week is enough to catch drift. If they disagree on more than about one diff in five, the labeling guide needs rewriting before the dataset grows.

Volume targets should be honest. Prompt edits can be validated with a few dozen reviewed examples, because you can read them and see the pattern. Fine-tuning needs more: a few hundred approved samples as a starting floor, with coverage across your main ticket categories. Balance matters more than raw count. A set of 400 samples where 350 are tone edits will produce a model that is politer and no more accurate. When you audit the set, check the label distribution and deliberately pull in factual and policy corrections until no category is starved.

This is also the point where a domain expert earns their keep. Reviewing diffs against policy requires someone who knows what the company is allowed to promise, not just someone who writes well. If your team does not have that reviewer in-house, a structured conversation about which corrections actually matter is the fastest way to scope the work (Quanta Consulting).

The decision rule is straightforward: fix prompts and retrieval first, fine-tune only when the same correction type keeps recurring in reviewed samples.

Prompts and retrieval handle factual and policy corrections better than fine-tuning in most cases. If agents are rewriting answers about refund windows or service limits, the model is missing context, and retrieval augmented generation or tighter grounding rules fix that directly. Fine-tuning and retrieval are different tools: fine-tuning alters the model's parameters for accuracy, style and specificity, while retrieval augments the prompt with external knowledge, and the choice depends on task type, data volume and goal (Google Cloud on fine-tuning versus RAG). A model that confidently states a wrong refund policy does not need more training on tone; it needs the policy in front of it.

Tone and format corrections are the opposite case. When reviewers keep approving the same kind of rewrite, for instance every escalation reply losing its opening apology and gaining a concrete next step, fine-tuning a small model on those approved pairs gives you persistent behavior change that prompt wording alone tends to drift away from (Heavybit's LLM fine-tuning guide). Supervised fine-tuning on demonstrations is the standard first method, and it is enough when your data is clean approved examples rather than preference rankings (AWS on fine-tuning with human feedback).

Practical guardrails for the training step:

• Split by ticket, not by sample. If two samples come from the same thread, keep them on the same side of the split or your evaluation leaks.
• Hold out a test set of at least 100 tickets before you touch anything, and never train on it.
• Start with parameter-efficient methods that adjust a small number of added parameters rather than the whole model, which lowers cost and reduces the risk of degrading general behavior (Google Cloud on PEFT).
• Watch for overfitting, where the model memorizes your examples and stops generalizing, and for catastrophic forgetting of behavior you did not train on (Oracle on training challenges).

If you are running several model families for drafting, classification and embedding, testing a fine-tuned variant against the baseline is easier when all of them sit behind one interface. Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible key, which simplifies side-by-side evaluation (Quanta Inference).

The metric that matters is the edit rate on held-out tickets, and it must be measured the same way before and after any change.

Define it precisely. The edit rate is the share of assisted tickets where the diff between draft and sent reply exceeds a threshold you set in advance, counted on tickets the model has never seen. Pick the threshold from your baseline distribution, not from intuition. If your baseline is a median edit ratio of 0.18 with a long tail, a threshold of 0.10 will count most tickets as edited; a threshold of 0.30 will count only the heavy rework.

A worked example makes the arithmetic concrete. Suppose you capture 1,000 assisted tickets over a measurement window and find that 380 have an edit ratio above your threshold, an edit rate of 38 percent. After prompt changes validated on the reviewed set, you measure the next 1,000 tickets and find 290 above threshold, an edit rate of 29 percent. That is a 9-point absolute reduction, roughly a 24 percent relative drop in rework. Now break it down by label. If factual corrections fell sharply but tone corrections barely moved, the prompt fixed grounding and your style problem remains a fine-tuning candidate.

Three cautions on reading those numbers. Ticket mix shifts, a product incident or a holiday can move the edit rate without any model change, so compare like-for-like categories. Agent mix shifts the same way, since a new hire edits more than a veteran. And reviewers change too, so keep a fixed sample of tickets double-scored to check that your labels stay stable.

Report the result to the people who generated it. Agents who see that their edits are being turned into better drafts, and who are told which of their corrections were approved, contribute more and label more carefully. That feedback loop between the briefing on policy and the model's behavior is the whole point of the exercise.