When to Fix a Bad AI Answer Yourself vs. Escalate to the Vendor

These three failure modes account for most bad answers, and all three are cheap to repair once you know which one you are looking at. The table below is the working checklist.

Prompt defects are the most common and the cheapest to fix. Look for three signals. First, the answer ignores a specific constraint: tone, format, escalation paths, warranty conditions. In a customer support retrieval system, instructions usually need to carry troubleshooting workflows, escalation paths and product-specific rules that a short system message cannot hold (Microsoft's RAG prompt engineering guide). Second, the model answers when it should refuse; Microsoft recommends defining in the system message how the model handles situations where the context does not contain enough information. Third, the same ticket produces different answers on different runs; check whether your prompt relies on implied ordering of context chunks. A structured framework helps: Supportbench describes a CARE pattern (Context, Ask, Rules, Examples) and recommends one well-chosen few-shot example, ideally the edge case you are already afraid of (Supportbench's prompt checklist).

A worked prompt example. A billing assistant answers "yes, you can pause your subscription" to a customer on an annual plan. You test by pasting the plan terms into the prompt manually. The model still says yes. Then you check the instruction: it says "be helpful and answer the question" with no rule about plan eligibility. You add a rule that eligibility questions must be answered only from the plan-terms document and must state the plan name. The manual test now returns "annual plans cannot be paused; here is the refund path". You fixed it in an afternoon with no vendor involvement.

Retrieval failures look like model failures because the answer is confidently wrong about a fact your knowledge base contains. The tell is that the correct document exists and is current, but it never reached the context window. Check in this order: is the document indexed at all, is it chunked so the answer is split from its heading, does the query use vocabulary the document does not (customers say "cancel", the policy says "terminate"), is the top result crowded out by a near-duplicate from an older version. Evidence that grounding matters comes from clinical triage research, where evaluation of retrieval-augmented generation centered on models being vulnerable to hallucination and whether retrieved evidence improves reliability (a retrospective evaluation of RAG for emergency triage). The same logic applies to support: if the right evidence is not retrieved, grounding cannot help.

A practical retrieval test: take ten bad answers from the last week, run the retrieval step alone for each, and record whether the correct passage appears in the top results. If it appears for most cases and the answers were still wrong, retrieval is not your problem and you move to the next check. If it is missing for most cases, stop reading the model's output and go fix indexing, chunking and synonyms. That single test will save your team from filing tickets the vendor will correctly reject.

Data defects are the failure mode most often misattributed to the model. The assistant quotes a price that changed last quarter, or gives two different refund windows because two policy documents disagree. The scale of this problem is documented. KPMG reports that 55 percent of companies cite data quality as a major barrier to AI adoption, and that organizations spend up to 80 percent of project time just preparing data (KPMG on build versus buy). If most of the effort goes into data before launch, it should not surprise anyone that data causes most of the failures after launch.

Triage actions for data faults are unglamorous and effective: assign an owner and a review date to every policy document in the index; add a freshness check that flags documents older than their review cycle; archive superseded policy versions rather than leaving them retrievable; and add a rule that when two sources conflict, the model must surface the conflict rather than pick one. Keep in mind that fixing data is not free, and that matters for the build versus buy argument. Total cost of ownership analysis puts license fees at only 20 to 40 percent of true AI implementation costs, with data preparation and quality work often exceeding the initial software investment (Pertama Partners on AI TCO). A support team that treats data quality as someone else's job will spend its savings on repeated bad answers.

Now the case that belongs to the vendor. You have reproduced the failure with clean, current, correctly retrieved context and a prompt that states the rules clearly. The model still gets it wrong, repeatedly and deterministically enough to demonstrate.

Before you write the ticket, build the evidence package. A vendor cannot act on "the bot gives bad answers". They can act on a minimal reproduction: the exact instruction text, the exact context block, the query, the response, the timestamp, and the API parameters. Run it three times to show it is not a sampling fluke, and once more with a second, simpler prompt that isolates the behavior.

This is where a multi-model setup helps your triage rather than confusing it. If the same context and instruction produce a correct answer on one provider's model and a wrong one on another, you have evidence of a model-specific behavior rather than a systemic problem in your pipeline. Running that comparison across providers is easier when you call several models through one interface, which is what Quanta Inference provides with an OpenAI-compatible API for text, image, video, audio and embedding models: Quanta Inference. Keep the comparison in your triage procedure, not just in your incident response.

A worked example of the escalation threshold. Out of 400 answers where the context was verified correct, 31 are wrong and 24 of those 31 fail the same way: the model drops a stated exclusion clause when summarizing a policy with more than three conditions. Reproducible, patterned, context clean. That is a vendor ticket with a sample of 24. One wrong answer out of 400 with no pattern is a documentation entry for your own evaluation set, not a ticket.

A single prompt can be badly written. The same failure across prompts that share almost nothing is a signal about the model's behavior, not your instructions. Define a repetition threshold before you need it: three or more distinct prompt and context pairs from different product areas, failing in the same way within a defined window, converts the issue from a local bug into a pattern. That number should be in your triage runbook and, ideally, in the contract, because it is the point at which the provider's own quality process can help.

Support ticket triage research describes how misclassified tickets cascade: a meaningful share of manually triaged tickets get reassigned at least once, and each reassignment adds resolution time (DevRev on AI support ticket triaging). Bad AI answers behave the same way. A pattern that is not escalated gets handled repeatedly by individual agents who each think they are seeing a one-off. The practical tool is a failure taxonomy with counts: tag every bad answer as wrong instruction, retrieval miss, stale data, model behavior, or safety, and review weekly. When "model behavior" crosses your threshold with clean samples, escalate. When it stays near zero while "stale data" dominates, your roadmap is a knowledge base cleanup, not a vendor conversation.

The hardest failure to triage is the one where nothing on your side changed. Answers that were fine last month are wrong now: tone shifted, refusals appeared, formatting broke, latency spiked.

This is a vendor-side event whether or not the provider calls it a change. Your response should be procedural, not investigative. Maintain a fixed evaluation set of 30 to 50 representative prompts with known good answers, run it on a schedule, and record the pass rate. When the pass rate drops materially between two runs with no change on your side, that is your evidence. Without the baseline you have an anecdote; with it you have a dated before-and-after that a vendor cannot dismiss.

The contract question follows immediately: what did the provider promise about notice, and what does that promise actually cover? Service level agreements in adjacent vendor categories offer a useful model. An email threat protection SLA is evaluated on six elements (threat scope, service scope, measurement period, exclusions, remedies and customer obligations) and is expected to set availability and detection performance as separate metrics (Adaptive Security's SLA guide). Your AI vendor agreement should separate uptime from answer quality in the same way, because a fully available model can still answer badly.

This is the insight most support teams miss: the contract is only useful if its escalation triggers are the conditions your team already detects. If the agreement says "material degradation of service" and your runbook says "pass rate below 90 percent on the evaluation set", nobody can invoke anything.

Rewrite the triggers as testable conditions. A clause that adjusts obligations on a specified trigger, stated with a defined formula and limits, is the general form of what you want (ContractKen on escalation clauses). Applied to AI quality, that means numbers, measurement methods and named evidence.

Two clauses deserve specific attention. First, the evaluation method: who owns the test set, how often it runs, and whether the provider accepts your results as evidence. Second, the remedy: service credits are the usual remedy for a missed target in service agreements, and they do not cover downstream loss, which keeps human review essential (Adaptive Security's SLA guide). Do not model your AI escalation clause as compensation; model it as an obligation to investigate and remediate within a defined window.

Procurement discipline helps here. Buyers evaluating contract management platforms are advised to separate vendor service commitments (uptime, support response, documented escalation path, remedies) from the platform's own ability to track obligations, and to request the actual service-level schedule rather than accept marketing claims (Sirion's guide to vendor SLA support). Apply the same test to your AI provider: ask for the schedule, not the brochure.

There is also a build versus buy dimension worth naming. Vendor contracts cap what the provider owes you, so the more of the answer pipeline you control, the more of the failure surface you can fix without a negotiation. If a large share of your failures land in the "vendor" column, that is a strategic signal about dependency, and it is worth a structured conversation about your goals and architecture. A free 30-minute discovery call is a reasonable place to have it: Quanta Consulting.

The rule only works if it is practiced. Run a 30-minute weekly review with four standing items. Review the tagged failures by category and compare counts against the previous week. Pick two bad answers at random and walk them through the four checks in order, verifying that the tag was assigned correctly. Check the scheduled evaluation pass rate and note any change against the baseline. Review open vendor tickets against the contract's response commitments.

Keep a one-page runbook that every agent can follow at the moment they see a bad answer.

1. Capture: prompt text, context sent, response, timestamp, ticket ID.
2. Reproduce with clean, current context, hand-fed.
3. If correct with clean context -> classify as instruction, retrieval or data. Fix locally.
4. If wrong with clean context -> run twice more. If stable, tag as model behavior.
5. Check whether the same failure appears in an unrelated product area.
6. Escalate when: stable wrong answer on clean context, OR pattern across 3+ unrelated prompts, OR evaluation pass rate dropped without a change on your side.
7. Attach the reproduction package to the ticket. Cite the contract clause.

The value of this loop is not the paperwork. It is that your team stops spending engineering hours on vendor faults and stops filing tickets for its own data problems. That is the whole point of a triage rule: to move each failure to the party that can actually fix it, on the first attempt.