Training Staff to Spot a Hallucinated Number

Build the exercise from answers your own organization actually received. Real prompts beat synthetic ones because staff recognize the phrasing and the stakes. Ask five or six colleagues to paste in an AI answer they used or nearly used in the past month, then pick three that fit the pattern.

For slot B, prefer a wrong figure that is close to the truth rather than absurd. A percentage that is off by a few points, or a total that looks like a rounding variant, teaches more than a number that is obviously impossible. To build it, ask a model for a statistic, then check the real value in a primary source and keep the model's version. For slot C, take a genuine figure and move it to a neighboring period, or ask for a figure "as of" a date and let the model answer for the wrong one.

Two rules keep the set honest. First, every example must have a findable source, because the learner's task includes citing it. Second, do not mix in a fabricated citation exercise here; that is a separate skill. If you want a model for what to watch for in generated material generally, Articulate's review checklist covers bias, gaps, outdated advice and invented references (Articulate).

Minutes 0 to 5: set the contract. Tell the group the only acceptable outcome is a source, not an opinion. Anyone who says "that looks about right" has not finished the task. Mention that the exercise contains one clean answer, so flagging everything as suspicious is also a failure.

Minutes 5 to 15: mark up individually. Hand out the three answers on paper or in a shared document. Each learner circles every checkable claim, following the fractionation step in the University of Maryland's fact-checking guide, which recommends breaking AI output into small searchable claims before evaluating any of them (University of Maryland). They then leave the document and search elsewhere, a practice the guide calls lateral reading: open a new tab and look for sources that can confirm or contradict each claim.

Minutes 15 to 25: hunt the sources. Learners must locate the primary or authoritative record for each number and date. VCU Libraries' fact-checking guide is blunt about the standard: trace the claim back to a source that would be in a position to know, and check the source itself rather than a summary of it (VCU Libraries). Post the rule on the wall: no citation, no credit.

Minutes 25 to 35: pair review. Pairs compare verdicts. Where they disagree, each side shows its source. This is where most of the learning happens, because the person who missed the wrong date usually missed it for a specific, correctable reason, such as trusting a secondary summary or not checking the period the figure covers.

Minutes 35 to 42: whole-group debrief. Reveal the three answers and the intended verdicts. Ask one question per slot: how did you find it, and how long did it take?

Minutes 42 to 45: write the rule. Each learner writes one sentence describing what they will do before acting on a number from an AI answer. Collect them.

The timing is deliberately tight. A 45-minute drill with a hard stop signals that verification is a quick routine, not a research project, which is the belief you are trying to install.

Staff who understand what a model is doing are better verifiers. For the debrief, IBM's explainer covers why hallucinations occur, and the AI for Education write-up describes a classroom demonstration in which a model produces a labeled map of US states and capitals and mangles the labels while sounding entirely confident (AI for Education). That account also notes the model's tendency to praise its own faulty output when asked to rate it. A short version of this demonstration makes an efficient opener for a skeptical room.

The broader reason to run the drill at all is that verification is a skill, and skills need practice rather than policy. Great Place To Work's survey of training practice at the 100 Best Companies describes organizations that build cross-functional exploration teams and gather use cases from every department, on the principle that AI explored only inside the technology organization will fail (Great Place To Work). A number-checking drill is a cheap way to give non-technical teams a real stake in that effort.

There is a receptive audience. LinkedIn data cited by Great Place To Work found that four in five US employees want more AI training, while only 38 percent of US executives were providing it. Meanwhile 57 percent of CEOs reported investing in the technology against 43 percent investing in workforce skills. The gap between appetite and provision is where a 45-minute exercise fits without competing for a training budget.

A simple rubric keeps the scoring fair and gives you numbers you can compare across quarters.

Two scoring rules matter more than the point values. First, a learner who declares all three answers unreliable loses the points for slot A, because over-flagging is the failure mode that makes a verification habit unusable. Second, a learner who found the right verdict by guessing, with no source, scores 2 out of 5. You are grading the process, not the outcome.

Track three aggregates: the percentage who caught slot B, the percentage who caught slot C, and the percentage who cited a primary rather than secondary source. Expect slot C to be the weak one; a correct number with a wrong date survives most people's sense of plausibility. If your first run shows most of the room catching B but half missing C, that tells you where the next drill's emphasis belongs.

One design warning. Do not let the session become a debate about whether AI is good. Organizations that treat adoption as a feature rollout rather than a change process see initial curiosity followed by a sharp drop in usage, according to Netrix Global's account of stalled rollouts (Netrix Global). A drill that stays practical, with sources on the table, keeps the conversation on behavior.

Models change, and so do the prompts your staff use. A drill built from last quarter's examples will slowly stop teaching anything, because the answers get passed around. Rebuild the three examples each quarter from the submissions you collect in the preceding weeks. Two hours of preparation per quarter buys a session that stays sharp.

If you have access to a single API for several model families, generating candidate examples becomes a short script: send the same prompt to several models, diff the numbers against your primary source, and keep the mismatches. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key with prepaid pricing in US dollars (https://quantaintelligence.ai/inference), which turns that comparison from a manual chore into a batch job. The point of the batch job is not to find villains; it is to harvest realistic wrong numbers faster than colleagues can submit them by hand.

Follow-up is where most programs leak value. Three habits pay for the quarter:

• Add the drill's citation rule to any internal AI use policy. The questions in CTR's policy checklist are a good starting point, particularly the one on when human review is required before AI-assisted work is used or distributed (CTR).
• Publish a small checklist alongside any AI-assisted deliverable, following the practice George Veletsianos proposes for AI-generated research summaries: state what tool produced the text and list the items a human verified, including that no fabricated data appears (Veletsianos).
• Give the drill a visible owner, typically one person per department, so the quarterly rebuild does not quietly lapse.

If you want the drill built and scored for several lines of business, or the follow-up checklists drafted to match your own policies, Quanta Experts can take that on and return finished materials (https://quantaintelligence.ai/agents).

Here is slot B built concretely, using the percentage-change trap described in the verification guide. A model is asked to report an increase from 40 to 50 and returns 20 percent. The arithmetic is (50 minus 40) divided by 40, times 100, which is 25 percent, so the model's 20 percent is wrong. The deeper lesson is that the correct denominator is the baseline figure, and a learner who only checks "did the model divide" will miss a case where the formula itself does not match the quantity being asked about (Manuel B. Garcia).

Run the same trap with your own figures. Pick a real metric from a dashboard, ask a model to compute a percentage change, then check both the arithmetic and the denominator. Half the room will reach for a calculator and stop there. The debrief question is the one that transfers: which number was the baseline, and does the sentence around it say so?

Dates reward the same treatment. Ask a model for a figure "as of" a specific month, then verify the month in the original release. When a learner finds the right number attached to the wrong period, have them read the original sentence aloud. The correction is usually a single word, which is precisely why it slips through.