How to Instrument AI Workflows So ROI Calculations Are Possible

Every step of an AI workflow should emit one structured event. Keep the schema small and stable. If you change event names between versions, your historical data becomes unjoinable and every ROI query needs a translation table.

The run_id is the most important field. It is the only key that lets you take a provider invoice line, a tool call, and a human's 4 minutes of review and say they belong to the same completed task. Without it you are back to account-level averages, which tell you almost nothing about whether a specific investment paid off.

Name events by what happened, not by what the system did internally. workflow.step.completed and workflow.step.failed age better than llm_call_2 or agent_retry_v3. Treat event names as a public contract with finance: publish the list, and only add to it, never rename.

A minimal emitter in Python looks like this:

import uuid, datetime, json

def emit(event_name, run_id, properties=None, cost=None):
    record = {
        "event_name": event_name,
        "run_id": run_id,
        "ts": datetime.datetime.now(datetime.timezone.utc).isoformat(),
        "properties": properties or {},
        "cost": cost or {},
    }
    # append to your event store; one JSON object per line is enough to start
    print(json.dumps(record))

Cost tagging means recording the ingredients of spend at the moment of the call, not reconstructing them later. Mavvrik's guide to AI cost tracking lays out three connected layers: a usage layer (provider, model, input and output tokens, embeddings, retrieval count, tool calls, latency, retry count), an ownership layer (team, product, feature, workflow, agent), and a finance layer (allocation rule, budget, cost-to-serve) (Mavvrik). Instrument the first two at the call site; the third is a finance job built on top.

Capture these attributes on each model call:

• Provider and model, as separate fields, so you can swap either without rewriting queries.
• input_tokens and output_tokens, read from the API response, not estimated.
• cached_tokens or a cache hit flag. Cached input is billed differently from fresh input, and teams that ignore it overstate cost.
• retry_count and attempt. Retries are real spend with no output. A workflow that retries 30 percent of calls looks twice as expensive per task as one that does not.
• tool_calls, with the tool name and duration. Retrieval, search, and code execution carry their own cost signals (retrieval queries, embedding volume, vector store storage) that do not appear on a model invoice (Mavvrik).
• human_minutes, logged when a person edits, approves or rejects an agent output. Use a fully loaded hourly rate (salary plus benefits plus overhead, commonly 1.3 to 1.5 times base salary) rather than base pay (Pickaxe).

The OpenAI-compatible OpenTelemetry GenAI semantic conventions give you a naming baseline for these attributes, so your traces reconcile with standard tooling instead of inventing a schema from scratch (OpenObserve). Even if you do not run a full tracing stack, adopting the attribute names costs nothing and saves a migration later.

One worked example, using illustrative numbers. Suppose one completed support reply runs three model calls at an average of 1,200 input tokens and 300 output tokens each, plus two retries, plus one retrieval lookup, plus four minutes of human review at a $60 fully loaded hourly rate. The model tokens cost whatever your rate card says. The human four minutes costs $4.00. If total model and tool spend per task is $0.06, cost per completed task is $4.06, and almost all of it is human review. That single number reframes the ROI conversation: the automation did not remove the person, it moved them to a review step.

A baseline is the pre-automation measurement of the exact task you plan to automate, recorded before rollout. SS&C Blue Prism lists establishing a baseline as step two of eight in measuring AI ROI, right after defining goals (Blue Prism). One practitioner guideline is to set baselines 3 to 6 months before introducing AI, recording total labor cost and time taken for the specific task (Naitiv).

Snapshot at minimum:

• Volume: how many completed tasks per week, from the system of record, not from memory.
• Cycle time: elapsed time from task start to task done, measured from timestamps, not surveys.
• Human minutes per task, from time tracking or ticket logs.
• Rework: how often the output was corrected, reopened or rejected. Rework time is the part teams most often forget, and it can erase most of the apparent savings (Naitiv).
• Quality marker: error rate or change failure rate for the same period.

Store the baseline as a versioned dataset with a date range and a named owner, in the same warehouse as your live events. If the baseline lives in a spreadsheet and the live data in a tracing tool, the join never happens and the comparison never gets made. Also snapshot quality, not just speed: faster output that rework erases is not a saving, it is a reshuffle.

For coding workflows, the baseline is specific: cycle time, throughput, change failure rate and rework for each team over the 90 days before rollout (Waydev). For other workflows, pick the same four dimensions (volume, cycle time, human minutes, rework) and hold them constant across the comparison.

The join key is run_id. Build one fact table where every row is a completed task, with the run id, start and end timestamps, outcome, and summed cost attributes. Every event table feeds it. Finance queries the fact table; engineering writes to it. Nobody maintains a second copy.

A practical shape:

-- cost per completed task, last full month
SELECT
  COUNT(*)                                   AS completed_tasks,
  SUM(cost_model + cost_tools + cost_human)  AS total_cost,
  SUM(cost_model + cost_tools + cost_human) / COUNT(*) AS cost_per_task
FROM workflow_runs
WHERE status = 'completed'
  AND ended_at >= DATE '2026-01-01'
  AND ended_at <  DATE '2026-02-01';

Three rules keep the table trustworthy. First, reconcile monthly: compare the sum of token costs in your events against the provider invoice and record the variance. A 2 percent gap is normal; a 40 percent gap means your rate card or your token counts are wrong. Second, define "completed" once, in writing, and compute it from the system of record. Third, publish the metric definition alongside the number, because finance and engineering arguing about what counts as done is the most common reason a valid ROI calculation gets rejected.

The output of this table is cost per completed task, which is the building block of ROI: once you know what each outcome costs, ROI is a comparison against the value it delivers (AWS). Two refinements make it defensible. Model the J-curve: a temporary productivity dip during rollout is a real cost. A worked coding example counts a 5 percent dip for 3 months across 200 engineers at a $180,000 loaded cost, which adds $450,000 to first-year investment (Waydev). And report capacity, not cash: time saved becomes money only if you show where the hours went, otherwise it gets absorbed into meetings and busywork (Pickaxe).

Rule one: if a step has no event, it has no cost, and any ROI number that includes it is an estimate. Instrument the smallest set of steps that lets you compute cost per completed task, then expand.

Rule two: prefer cost per completed task over total spend. Total spend rises when usage rises even if unit economics improve. Cost per workflow, cost per inference and cost-to-serve are stronger operating metrics because they connect usage to margin and pricing decisions (Mavvrik).

Rule three: separate hard and soft ROI. Hard ROI is quantifiable money in or out; soft ROI covers satisfaction, skill retention, brand and agility (Techstack). Instrument hard ROI first; soft benefits need their own metrics and should not pad the arithmetic.

Rule four: use a 6 to 12 month window. A 90-day ROI window for an agent deployment is dominated by implementation and learning costs; meaningful measurement typically needs 6 to 12 months, and strategic deployments are modeled to 24 (Pickaxe).

Pitfalls that recur: counting task completions instead of outcomes, which is why some teams have moved to agentic work units that tie activity to value (Pickaxe); leaving out human-in-the-loop and monitoring costs, the "trust tax" of making sure the agent is not confidently wrong; and letting the saved hours vanish without a record of where they went.

Public ROI figures are directionally useful and rarely comparable. A benchmark report assembled 47 public metrics and found credible workflow gains, including 15 percent customer-support productivity gains and 55.8 percent faster coding task completion, while cautioning that many business cases are vendor-published, annualized or gross of implementation cost (Alice Labs). Instrument your own workflow; cite the benchmark only as context.

1. Pick one workflow and one completed-task definition. Write both down.
2. Snapshot the baseline for the last 8 to 12 weeks from existing systems: volume, cycle time, human minutes, rework.
3. Add the event emitter with the five-field schema and a run_id generated at task start.
4. Tag model and tool calls at the call site. Read tokens from responses; log retries and cache hits.
5. Build the fact table and the cost-per-completed-task query. Reconcile against one provider invoice.
6. Publish the definition and the number to finance in the same place.

If you are wiring several model providers and tools into one workflow, routing them through a single OpenAI-compatible endpoint removes a chunk of instrumentation work, because tokens and costs come back in one format you can log: Quanta Inference puts text, image, video, audio and embedding models behind one API key, prepaid and priced in dollars. Whether you build or buy that layer, the schema above stays the same.