The Prompt Fix That Stops Invented IDs

Apply the changes one at a time, and measure after each. The ordering below is deliberate: the enum removes the largest share of errors, the examples teach the copy behavior, and the refusal token cleans up the remainder.

The token counts are the ones you would measure with a tokenizer on the actual prompt text: the enum block for 30 tickets, three worked examples, and a two-line refusal instruction. They are input tokens, which are the cheaper half of the bill on almost every provider, and they recur on every call. That is the trade you are making: a fixed surcharge per request in exchange for removing a failure mode.

Worked example: the ticket router

Take a support router that reads an incoming message and assigns it to one of 30 predefined tickets. The baseline prompt said: "Identify the ticket this message belongs to and return its ID." On a 500-ticket evaluation set with hand-labeled ground truth, 90 responses contained an ID that did not exist in the ticket table, which is 18 percent. Those were not parse errors. They were well-formed, confident, wrong identifiers, the kind that pass a JSON schema and fail a database lookup.

After adding the enum list, that fell to 30 invented IDs, or 6 percent. Adding three examples where the output copies an ID straight from the input brought it to 10, or 2 percent. Adding a NO_MATCH instruction brought it to 5, or 1 percent. At the same time, the share of cases where the model correctly returned NO_MATCH on genuinely unmatched messages rose, which matters because a refusal that is never used is just a silent misroute.

The arithmetic on cost: 360 extra input tokens per call. On 500 tickets that is 180,000 extra input tokens for the evaluation run, and in production it is 360 tokens multiplied by your monthly call volume. Compare that with moving the same workload to a larger model tier, which typically multiplies the per-token rate for every call, not just the failing ones, and still leaves the model free to invent an ID when the allowed set is missing from the prompt. Cost optimization write-ups consistently find that a large share of production token spend goes to premium models doing work a cheaper configuration could handle the LLM cost optimization analysis, and one vendor guide puts the share of production prompts that do not need frontier capability at 60 to 80 percent, with routing to appropriately sized models cutting effective inference cost by 40 to 85 percent the model selection guide. Fixing the prompt first is the version of that saving you can bank without changing your model at all.

The enum only helps if the model can see it as a closed set and if the format makes copying easy. Three rules cover most of it.

First, put the list where it will be attended to, near the instruction, not buried in a system prompt that also contains brand voice guidance and tool definitions. Second, number or delimit each entry clearly, one ID per line, with the human-readable label beside it. An ID with no label forces the model to match on the digits alone. Third, state the constraint in the imperative: "Choose exactly one ID from the list below. Do not output any ID that does not appear in this list."

ALLOWED_TICKET_IDS:
T-1001  Billing: duplicate charge
T-1002  Billing: refund status
T-1003  Access: password reset
...
T-1030  Shipping: address change

Choose exactly one ID from the list above.
Do not output any ID that does not appear in the list.

If your provider supports schema-constrained generation or enum fields in a JSON schema, use them as well. MLflow's guidance on structured outputs recommends keeping schemas minimal with required fields, strict types and enums to prevent ambiguous outputs, and warns that schema-constrained generation guarantees syntactic validity but does not guarantee value correctness the structured output guide. That distinction is exactly why the enum belongs in both places: in the schema, where it can be enforced by the decoder, and in the prompt, where the model reads it as part of the task description.

A caution on size. An enum of 30 tickets costs about 120 tokens. An enum of 5,000 product IDs costs thousands of tokens on every call and will crowd out the actual message. Above a few hundred entries, switch to retrieval: fetch the top 20 candidates by embedding similarity, then present those as the enum. The prompt pattern stays the same, the list just gets built per request.

The purpose of the examples is not to show the model what a good answer looks like in general. It is to show it that the ID in the output is the same string as the ID in the input, character for character. That is a different lesson, and it is the one that survives contact with a new ticket the model has never seen.

The demonstration format matters more than the content. Work on few-shot prompting notes that the label space and the distribution of input text in the demonstrations both matter, and that format itself plays a key role even when labels are random the few-shot prompting guide. For ID copying, that translates into keeping the input block and the output block visually distinct and always in the same order.

INPUT: My card was charged twice for the same order.
ALLOWED_IDS_SEEN_IN_INPUT: [T-1001, T-1015, T-1022]
OUTPUT: {"ticket_id": "T-1001"}

INPUT: I moved house, can you change the delivery address?
ALLOWED_IDS_SEEN_IN_INPUT: [T-1008, T-1030]
OUTPUT: {"ticket_id": "T-1030"}

Two or three of these are enough. More than five starts to cost real tokens for diminishing returns, and very long example blocks can push the actual message out of the effective context. If you find yourself wanting ten examples, the problem is probably the enum list or the task definition.

This is also the point where prompt versioning stops being optional. Once you have three edits stacked on top of each other, you need to know which version produced which output when a regression appears. Tools in this space treat prompts as immutable versioned artifacts with diffs and environment promotion the prompt versioning comparison, and the same discipline applies to a five-line enum block: a single changed ID in the list is a behavior change that should be reviewable the prompt management guide.

The final percent is stubborn because it comes from cases where the correct answer is not in the list. A message about a problem you have no ticket for will still tempt a model to pick the closest ID, and the closest ID is wrong in a way that looks right.

The fix is a fixed sentinel with an explicit instruction: "If no ID in the list matches the message, output NO_MATCH. Do not choose the closest ID. Do not guess."

Three details make the sentinel work:

• Use one exact string, in capitals, and reuse it in your parser. NO_MATCH, not "none", "N/A" or "no match found".
• Say what happens next. A line like "NO_MATCH routes the ticket to a human queue" gives the model a reason to prefer the honest answer.
• Count refusals as a first-class metric. Track the NO_MATCH rate alongside the invented-ID rate. A router that refuses 30 percent of traffic is telling you the enum is too small, not that the prompt is working.

LLM evaluation write-ups describe verifier-based and judge-based checks as the two practical ways to score outputs like this at scale, since neither depends on the model grading itself the LLM evaluation overview. For ID checking you need neither: a set membership test against your ticket table is deterministic, free, and catches every invented ID. Run it on every response in staging, and log the failures with their prompt version.

The same refusal pattern generalizes. Anywhere a model picks from a fixed vocabulary (categories, SKUs, employee IDs, country codes, enum fields in an API), the three changes apply in the same order: close the set, demonstrate copying, and legalize refusal.

Prompt fixes handle the case where the model has what it needs and is choosing badly. They do not handle the case where the model lacks the information. If the correct answer requires reading a database row the prompt never included, no enum will conjure it, and the model's only honest options are refusal or retrieval.

Use that as your decision rule. If invented IDs persist above a couple of percent after enum, examples and refusal, check whether the failing cases are ones where the input genuinely does not contain the needed identifier. If they are, add retrieval and put the retrieved candidates into the enum. If the failing cases are ones where the ID was right there in the input and the model still got it wrong, the problem is model capability or task framing, and that is when a stronger model or a different architecture earns its cost.

Routing literature makes the same point from the other direction: routing decisions should be based on measured requirements per query, not on provider reputation, and a static routing rule is usually the right first implementation because it is easy to evaluate the LLM routing guide. Task-aware routing systems that model cost and quality per query exist precisely because performance varies by task type the task-aware routing paper, and task-based routing with fallback logic is a mature pattern at the gateway layer the task-based routing overview. None of that changes the ordering: measure the failure, fix the prompt, then decide whether the model tier is the next lever.

If you are running this across several models to find which one handles your enum-and-examples prompt best, a single endpoint for every provider saves you from wiring each SDK separately. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key Quanta Inference, which makes the A/B comparison a configuration change rather than a migration.