How to Measure Whether AI Training Changed Anyone's Work

Good measures share four properties. They should already exist somewhere in a system you own, so collecting them is a query rather than a survey. They should be per person, not per team, so you can see who changed. They should be readable at both baseline and review without changing the definition. And they should be hard to game in the short term.

The last point matters more than it sounds. "Number of prompts sent to the assistant" is easy to collect and easy to inflate. "Share of replies that passed supervisor review without correction" is harder to collect and much harder to fake, because a supervisor has to sign off.

Here is how the three measure types map onto common AI training targets:

The optional fourth row is worth including if the training's real goal was adoption rather than performance. Be honest about which one you are measuring. A program that gets 80 percent of a team using the assistant daily but produces no quality or speed change has succeeded at adoption and failed at impact, and those are different conversations.

For engineering teams there is a specific trap: AI-assisted code can raise the volume of pull requests while lowering review quality. METR's work on measuring how long a task an AI can complete makes a related point about capability measurement: it is far easier to measure whether a model can do something at all than whether the result is useful in a real workflow. Your review-catch rate is a better signal than your pull request count.

The baseline is the part people skip, and it is the part that makes the whole exercise worth doing. Management Concepts' guide to measuring training impact puts a pre-training baseline assessment near the top of its list, and the reason is simple: without it you cannot distinguish training effect from seasonal variation, staffing changes, or a product launch that moved the workload.

The baseline window matters. Four weeks of data immediately before the training is usually right. One week is too noisy, especially for per-person daily counts. A full quarter is more stable but risks including a period when the workflow was meaningfully different.

Record the baseline per participant, with a participant identifier that survives the review period. If someone changes teams or leaves, you need to know that at review time rather than discovering it while comparing numbers. Keep the raw counts as well as the ratios, because at review you will want to recompute with and without the people who left.

Here is a minimal baseline record you can keep in a spreadsheet or a table in your warehouse:

participant_id, team, cohort, measure, baseline_value, baseline_window_start, baseline_window_end, source_system, definition_version
1042, support-tier1, cohort-a, tickets_per_day, 21.4, 2026-02-02, 2026-02-27, helpdesk, v1
1042, support-tier1, cohort-a, correction_rate, 0.31, 2026-02-02, 2026-02-27, helpdesk, v1
1042, support-tier1, cohort-a, minutes_per_reply, 18.2, 2026-02-02, 2026-02-27, helpdesk, v1

The definition_version column looks fussy until the first time someone changes how "correction" is counted halfway through the review period. Version the definition and freeze it.

One more baseline decision: whether to include a comparison group. If only one of three support pods attends the training, the other two give you a rough control. This costs nothing and prevents the most common attribution error, which is crediting training for an improvement that hit everyone because the ticket queue got quieter. Docebo's guide to measuring training effectiveness frames the same logic through the Kirkpatrick levels, where behavior change on the job sits above satisfaction and knowledge and is where most programs stop collecting data.

Pick the date in advance and put it on the calendar of everyone who will attend the review. Four weeks after the session is the default for a good reason: shorter and people are still learning the tool, so you are measuring the learning curve rather than the plateau; longer and other changes start to contaminate the comparison. Faye Digital's account of structured AI training programs notes that structured programs can show measurable results such as hours saved and faster turnaround within about 30 days, which lines up with the four-week window as a practical first read.

The review date is not the end of measurement. It is the first read. If the numbers are flat at four weeks, that is information: either the training did not transfer, or the workflow did not change enough for the measure to move, or the measure was wrong. All three are fixable, and all three are invisible without a scheduled read.

Write down at baseline what you would consider a meaningful change, so the review is a comparison against a stated expectation rather than a negotiation after the fact. For a support team with a 31 percent correction rate, a drop to 24 percent on the trained cohort while the untrained pods stayed near 30 percent is a result worth acting on. A drop to 29 percent is noise.

The four-week read should answer four questions, in this order:

1. Did the measure move, and by how much, per participant?
2. Did it move for the comparison group too?
3. Which participants moved and which did not, and is there a pattern (team, tenure, task type)?
4. Did anything else change in the workflow during the window that could explain the movement?

Question three is where the useful insight lives. A training that moved half the cohort a lot and left the other half untouched is a different problem from one that moved everyone slightly. The first calls for targeted follow-up with the people who did not shift, usually because their task mix did not match what the session covered. The second may mean the tool is helping but the session content is thin.

Take a support team of 12 agents handling tier-one tickets. The training taught them to draft replies with an AI assistant, edit, and send. Three measures: tickets handled per person per day, share of replies needing supervisor correction, and average minutes per reply. Baseline is the four weeks before the session; review is four weeks after.

Baseline: 21 tickets per person per day, 31 percent correction rate, 18 minutes per reply. Across 12 agents over 20 working days, that is roughly 5,040 tickets in the baseline window.

At review, assume the team is now at 24 tickets per person per day, 26 percent correction rate, and 16 minutes per reply. The naive read is a clear win: volume up 14 percent, corrections down 5 points, time down 11 percent. Now check the comparison group. If two untrained pods also moved from 21 to 23 tickets per day because a backlog cleared, then most of the volume gain is not attributable to training. But if the untrained pods' correction rate stayed at 31 percent, the 5-point quality improvement looks like training effect.

That combination, volume gain that is partly ambient and quality gain that is cohort-specific, is the most common real result. Report it that way. A finding like "the trained cohort's correction rate fell 5 points while the untrained pods held flat, and roughly two-thirds of the throughput gain appears to be queue-driven" is more useful to the next budget conversation than a blended percentage.

Time per reply needs the most care, because agents who handle more tickets often spend less time per ticket for reasons unrelated to the tool. Cross-check: if minutes per reply fell but correction rate rose, you probably traded quality for speed, which is a bad trade in support. If both fell, the tool is plausibly doing real work.

The fourth measure, share of replies drafted with the assistant, tells you whether the effect is coming through the mechanism you trained. If 90 percent of the trained cohort drafts with the assistant and the correction rate still did not move, the assistant is not the lever. If only 30 percent draft with it, the training did not stick and the follow-up is coaching, not a new measure.

Measurement design and training design are the same project. Training that never touches the real workflow cannot produce a workflow change, no matter how good the measurement. Great Place To Work's account of AI training at large employers describes companies building cross-functional teams and gathering use cases from every department before designing programs, which is exactly the input that tells you which tasks to measure. Guild's employer guide to AI training makes the adjacent point that different roles need different content, which means different measures per role rather than one company-wide dashboard.

Two design choices do the most to make measurement possible. First, train on the actual tool with the actual data the participant will use on Monday. A session on generic prompting followed by a tool rollout two weeks later produces a measurement gap you cannot close. Second, build a short practice loop into the session where each participant completes the real task once, so the first post-training attempt is not also their first exposure.

NAVEX's outline of AI at work training covers risk, privacy, and ethical use alongside practical tool use. If your training is mostly compliance-oriented, your measures should reflect that: incidents reported, escalations handled correctly, and policy questions raised during the review window are more meaningful than time per task. Udemy's guide to building AI literacy makes the case that teams without literacy spend hours fixing AI-generated errors, which is a quality measure you can count directly as rework.

A short checklist before the session runs:

Three outcomes are possible at the four-week read, and each has a different next move.

The measures moved and the comparison group did not: the training worked for this task. The next step is not more training on the same thing; it is widening the cohort or moving to the next task with a fresh baseline. Document the measure definitions so the next program can reuse them.

The measures moved but so did the comparison group: you cannot separate training from ambient change. Do not report a win. Instead, extend the review or find a measure that is specific to the trained behavior, such as share of replies drafted with the assistant.

The measures did not move: check adoption first. If participants are not using the tool, the problem is the workflow or the coaching, not the session content. If they are using it and nothing changed, the task may be one where the assistant is not yet useful, which is worth knowing before you train anyone else on it.

Guild's chapters on employee sentiment and Udemy's summary of costly AI mistakes both point at the same failure mode: teams that avoid a tool entirely or use it badly will show up in your data as low adoption or high rework, not as a flat performance trend. Read the adoption measure before you conclude the training was weak.

One habit makes all of this compound: keep the baseline records. After three or four programs, you will have a library of task-level measures with definitions that you can reuse, and the fourth program's review takes an hour instead of a week.

If the team running the training needs help turning the tasks into a working assistant and a clean data pull, an expert engagement can produce the workflow and the measurement scaffold together. For teams building the tool itself, putting every model behind one API key removes a category of variation from your review, because the same assistant reaches text, image, and embedding models without separate integrations. When the measures are agreed and the review date set, the simplest way to get an outside read on your plan is a short call with someone who has run the exercise before.