Using an AI Agent to Tag and Route Newsletter Replies

The taxonomy is the product decision. Get it wrong and no amount of prompt engineering saves you. Five labels cover the bulk of newsletter reply traffic:

Two rules keep this taxonomy honest.

First, unsubscribe and out_of_office must be handled by deterministic code, not by the model's judgment. Mailbox providers require a working one-click unsubscribe and expect senders to honor requests within two days per Yahoo's sender requirements. Do not let a language model sit between a reader and the unsubscribe button. Parse the List-Unsubscribe header on the way out, and treat any reply classified as unsubscribe as a trigger that a human confirms, not a suggestion the model can bury.

Second, resist adding labels. Intent classification works best when the label set is small and predefined for a bounded domain; open-ended label sets drift as VeGait's overview of intent classification explains. If you find yourself wanting a sixth label, ask whether it is really a sub-type (complaint about content vs. complaint about frequency) or a routing destination. Routing belongs in a separate field.

The AWS Comprehend walkthrough of automated email response makes the same architectural point: match the message to a known intent, answer automatically when there is a match, and send unmatched messages to a human per the AWS Machine Learning blog. Your five labels are that "known intent" set.

Ask for one object per message with a fixed shape. Constrained output matters here because you are writing to a database, not reading prose. The schema below is the whole contract:

{
  "intent": "interested | question | complaint | unsubscribe | out_of_office",
  "confidence": 0.0,
  "extracted_question": "string or null",
  "key_quote": "verbatim span, max 200 chars",
  "suggested_draft": "string or null",
  "requires_human": true
}

The prompt should do four things: state the label definitions, give one example per label, instruct the model to copy a verbatim span into key_quote, and forbid it from guessing when the message is ambiguous. Keep definitions tight. "Interested means the sender asks about buying, sponsoring, licensing or partnering" is checkable. "Interested means positive sentiment" is not.

A caution on structured outputs. Forcing a schema can make a model more likely to fill every field confidently even when the input does not support it as discussed in this Hacker News thread on false confidence in structured outputs. Two mitigations work well. First, allow null explicitly for extracted_question and suggested_draft and say in the prompt that null is the correct answer when the message does not contain one. Second, score trust separately from the model's self-reported confidence: Cleanlab's benchmark of structured output error detection found that dedicated trust scoring caught bad fields with meaningfully better precision and recall than self-reported log probabilities, which matters when you are deciding what a human must read per Cleanlab's structured outputs benchmark.

If you are running several models to compare labeling quality, or want to swap providers without rewriting the pipeline, Quanta Inference exposes text and embedding models through one OpenAI-compatible API key, which keeps the labeling call and the embedding call behind the same client.

The cutoff is a business decision expressed as a threshold. Work it out from the cost of each mistake.

Suppose you get 120 replies per week. At a 0.70 cutoff you might route 55% to auto-draft and 45% to review. At 0.85 you might route 30% and review 70%. The auto-draft bucket is not free: a wrong interested label that reaches a founder as a "hot lead" wastes ten minutes; a wrong complaint label that never reaches anyone costs you a churn signal.

A practical starting policy:

• 0.85 and above: auto-label, auto-draft, queue for a one-click approve.
• 0.60 to 0.85: auto-label but no draft; human writes from scratch with the label as a hint.
• Below 0.60: route to the review queue with no label attached, so a human is not anchored by a bad guess.

Then measure. Each week, sample 20 reviewed messages, record the human's final label, and compute agreement by bucket. If the 0.85+ bucket agrees with humans 95% of the time, you can lower the cutoff to 0.78 and pick up volume. If it agrees 80% of the time, raise it. Track accuracy per label separately, because models tend to be confident about unsubscribe and shaky about complaint.

Two operational rules protect the queue. First, never let the review queue grow past a day: replies are time-sensitive, and a two-day backlog is worse than no automation. Second, make the review UI show key_quote next to the label. Reviewers approve or correct in seconds when the evidence is visible, and the quotes double as your audit trail.

A draft is only useful if it is close enough to send. That requires two inputs the model does not have by default: voice and facts.

The tone guide is short and negative. "Write in the first person as the newsletter's author. Sentences under 20 words. No exclamation points. Never promise discounts or features that are not in the facts block. If the question is not answered by the facts block, write one sentence saying you will follow up and stop." The last rule is the important one, because it converts a hallucination risk into a deferral.

Product facts belong in a versioned file that sits next to the prompt, not inside it. Keep it to bullets a support person could defend: what the product does, what the team plan includes, where the pricing page lives. When pricing changes, you edit one file. This is the same discipline the deliverability world applies to authentication: one authoritative record, checked on every send, rather than ad hoc text as Gmail's guidelines describe for authentication and MXToolbox explains for DMARC.

A worked example. A reader writes: "We have a team of nine editors. Does the plan cover seats, and can you invoice us?" With facts stating that the team plan covers seats and that invoicing is available on request, the draft is two sentences: "Yes, the team plan covers seats for editors and we can invoice on request. Want me to send the details?" The extracted_question field is "Does the plan cover seats, and can you invoice us?" and confidence is high because both clauses map directly to facts. A reviewer approves in one click.

Contrast that with: "Loved the issue, made me think about how we handle this." The label may be interested at 0.61, which sends it to review with no draft. That is correct behavior. A generic compliment is a relationship signal, not a request, and a bad auto-draft would be worse than silence.

The approve step is where the agent stops being a toy. Three implementation details matter.

Identity and the draft folder. Write drafts into the mailbox as real drafts, not as rows in a database. Reviewers already know their mail client, and a draft in the folder keeps threading, signatures and reply headers intact. The agent never holds send credentials.

Idempotency. Every message gets a stable ID (the Message-ID header works) and the agent records what it did. If a job reruns, skip anything already labeled. This is the same principle as not sending the same email twice, and it prevents a loop from drafting five replies to one thread.

Suppression. A unsubscribe classification should create both a one-click unsubscribe event and an internal suppression record in the same transaction. Honor unsubscribes within two days, which is the stated expectation for bulk senders per Yahoo's requirements, and keep spam complaints low: below 0.3% is the threshold Gmail and Yahoo enforce per Gmail's guidelines and AWS's summary. A reply-based unsubscribe is a gift compared with a spam complaint, so process it fast.

Finally, feed decisions back. When a reviewer changes a label, log the original label, the new label, the model version and the quote. After a few hundred corrections you have a labeled set to regression-test any prompt or model change against. That set is worth more than any prompt tweak.

If you would rather not run the pipeline yourself, Quanta takes a described workflow (mailbox connection, labels, cutoff, draft folder) and builds it with you.

Track four numbers weekly and put them next to each other:

The first metric can rise while quality falls, which is why you watch the second. Approve-without-edit is the real quality signal: it tells you the tone guide and facts block are working. If it stalls below 60%, the problem is almost always the facts block, not the model.

One more measurement: count replies per issue and per intent over time. A rising interested count after a specific issue is a content signal you can act on. A rising complaint count about frequency is a warning that your list health is slipping, and complaint volume is the input to the 0.3% spam threshold every sender is judged on per Gmail's sender guidelines and PowerDMARC. Data you already collect can tell you what your readers want more of, and the growth tactics literature agrees that replies are a stronger signal than opens per Founderpath's newsletter metrics discussion.