Measuring AI Agent ROI

Most AI agent ROI claims are fiction - not malicious fiction, but the optimistic kind where someone multiplies hours saved by an hourly rate, ignores every cost, and declares victory. Careful measurement is rarer and far more valuable: it tells you which agent workflows deserve expansion, which need redesign, and which should be shut down. This guide gives you the full formula, the costs everyone forgets, and a tracking approach simple enough to actually maintain.

The full ROI formula

Return on investment is value created minus total cost, and both sides need full accounting. Value created is time saved multiplied by the value of that time, plus errors avoided multiplied by the cost of those errors, plus any quality improvements that translate to revenue or reduced risk. Total cost is build cost (the hours spent designing, prompting, and testing, with those hours priced at a real rate), plus run cost (inference, tools, infrastructure per execution), plus review cost (human time verifying outputs), plus failure cost (incidents, rework, and wrong outputs that slipped through). Most ROI calculations count one item on the value side and zero on the cost side. Run the full formula for your top three agent workflows and you will likely find one clear winner, one break-even, and one that is quietly losing money.

What people forget to count

• Build and iteration time. The two weeks of prompt engineering were real labor with real cost - amortize them over the workflow's expected lifetime.
• Review and verification time. If a human spends fifteen minutes checking each agent output, that is a per-run cost, and it often exceeds the inference cost.
• Failure remediation. The wrong outputs that slipped through have costs: customer apologies, rework, bad decisions made on bad data. Estimate from incident logs.
• Maintenance as things change. Models update, tools change, websites redesign - keeping the workflow working is ongoing labor, not a one-time cost.
• The cost of delay. An agent that takes twenty minutes where a human took five may still win on labor cost but lose on anything time-sensitive.
• Opportunity cost of the builder. The engineer perfecting prompts was not building something else - count the trade-off at least qualitatively.

Measuring time saved properly

Time saved is the heart of most agent ROI, and it is routinely measured badly. The right way: measure the baseline before automating - time five to ten manual executions, note the variation, and record what the human's time was actually worth (not their salary divided by hours, but the value of what they would otherwise do). Priced honestly, that baseline is the true cost of repetitive work the agent is meant to replace. Then measure the agent-assisted process end to end, including the human review time, the failed runs that needed redoing, and the setup time per run. Compare distributions, not anecdotes: the agent's median run against the human's median, with attention to the tails. And re-measure quarterly - workflows drift, humans get faster with practice too, and an ROI that was positive in January can be negative by June without anyone noticing.

When ROI is negative and that is fine

Not every agent workflow needs positive ROI to be worth running. Learning workflows - the ones teaching your team what agents can do - are investments, and their return is capability, not hours. Strategic workflows that build organizational competence with a technology you expect to matter are similarly justified as R&D. And quality-improvement workflows may show negative ROI on time while reducing error rates that matter more than the hours - a support agent that slightly costs more but halves complaint escalations is winning on a dimension the formula underweights. The key is being explicit: label these as investments with a thesis and a review date, not as savings. Negative ROI by accident is failure; negative ROI by decision, reviewed periodically, is strategy.

A simple tracking template

• Per workflow, per month: number of runs, total run cost, total human review time, failures and their remediation cost.
• Quarterly: re-measured time-saved figures, updated value-of-time estimates, and the resulting ROI calculation.
• One line per workflow stating the verdict: expand, maintain, redesign, or retire - with the evidence cited.
• A changelog of what changed about each workflow, so ROI shifts can be explained rather than wondered at.
• An explicit list of excluded benefits and costs with reasons - the exclusions list that keeps the numbers credible.

Where the run-cost line comes from

Run cost is the one line in the formula that teams often estimate when they could be reading it directly. Model providers report usage on every API response. Anthropic's Messages API returns a usage object with each response that reports the input tokens, output tokens, and cache token counts for that single request. OpenAI's APIs return a usage object as well, with prompt and completion token counts per request. If your harness records those counts against the run id at the moment each call returns, per-run cost stops being a guess: multiply the recorded token counts by the rates on your provider's pricing page, add the per-call charges for any paid tools the run used, and the run-cost line of the ROI formula is a sum of measurements rather than an allocation of a monthly bill.

This is also where ROI measurement meets observability. The same per-step record that prices a run - which model, how many tokens in and out, which tools, how long each step took - is the record you need when a run fails and someone asks what happened. Teams that build the run record once, for cost, find they have already built most of what debugging needs. The companion guide on agent observability covers what that record should contain and how to read it after a failure.

A worked example with invented numbers

The figures in this section are invented to show the arithmetic on a complete example. They are not benchmarks, and your workflow will differ in every input. The method is the part to copy. Imagine a workflow that drafts replies to support tickets for a human to approve. It runs 400 times a month. Timing the manual process first: a person needs 12 minutes per draft, start to finish. With the agent, the draft arrives written and a person spends 3 minutes reviewing and fixing it. Net time saved is 9 minutes per run, or 3,600 minutes a month: 60 hours. Price that time at an illustrative 40 dollars an hour and the monthly value created is 2,400 dollars.

• Run cost: the recorded usage averages an illustrative 11 cents per run across tokens and tool calls, so 400 runs cost 44 dollars a month.
• Build cost: the workflow took 30 hours to build and tune, priced at the same 40 dollars an hour, which is 1,200 dollars. Spread over a 12-month expected life, that is 100 dollars a month.
• Failure cost: about 2 percent of drafts slip through with an error that takes 20 minutes to put right. That is 8 runs a month, roughly 2.7 hours, or about 107 dollars.
• Maintenance: keeping prompts and templates current takes about 2 hours a month, or 80 dollars.
• Total monthly cost: 44 + 100 + 107 + 80, which is 331 dollars. Net value: 2,400 minus 331, which is 2,069 dollars a month. The workflow clears the bar comfortably.

Now stress the same example at its weakest point, which is review time. If reviewers slow down and take 9 minutes per draft instead of 3, net time saved falls to 3 minutes per run: 20 hours a month, worth 800 dollars, and the net shrinks to 469 dollars. If review takes the full 12 minutes, the value side is zero and the workflow loses 331 dollars a month no matter how cheap the tokens get. That is the sensitivity worth knowing before you scale anything: in most knowledge-work workflows the review minutes decide ROI, and the token bill is a rounding error beside them. Measure review time with the same care as the baseline, because a two-minute error there outweighs every other line in the table.

Careful ROI measurement will kill some of your agent workflows, and that is its purpose. The ones that survive scrutiny are the ones worth scaling, and scaling those with confidence - backed by real numbers - is how agent programs grow from experiments into infrastructure. Measure carefully, review quarterly, and let the losers go.

A second model: a Python-computed hypothetical with a sensitivity table

The worked example earlier used invented figures to show the arithmetic. This section builds a different hypothetical model in code, so every number below comes from a script you can run and change. It is a hypothetical for a document-triage style workflow. None of the inputs are measurements from a real deployment, and AgentsUse has not benchmarked any agent on this task. Replace every input with your own timings before drawing a conclusion.

The inputs

• Task volume: 400 tasks a month.
• Manual time per task: 12 minutes. Review time with the agent: 4 minutes, which includes opening the draft and fixing small issues.
• Error rate: 3 percent of agent outputs slip past review and need 25 minutes of rework later.
• Labor value: 40 dollars an hour, applied to saved time, rework, build and maintenance alike.
• Model usage per task: 8 model calls, each with 6,000 input tokens and 600 output tokens (an assumption about a multi-step run with tool results fed back in).
• Token price: 2 dollars per million input tokens and 10 dollars per million output tokens, the Claude Sonnet 5.5 list prices on Anthropic's pricing page and API pricing documentation, checked 2026-11-03. Prices change, so recheck before reuse.
• Build cost: 30 hours spread over a 12-month life. Maintenance: 2 hours a month.

Token cost per task is 8 times (6,000 times 2 plus 600 times 10) divided by one million, which is 0.144 dollars. That ignores prompt caching and batch discounts that the same pricing page lists, so it is a deliberately plain figure. The Anthropic tool use overview notes that tool definitions and tool_use and tool_result blocks also count as input tokens, which is why the input figure here is large relative to the output.

base = dict(volume=400, manual_min=12.0, review_min=4.0, err=0.03, rework_min=25.0,
            wage=40.0, calls=8, in_tok=6000, out_tok=600, p_in=2.0, p_out=10.0,
            build_h=30.0, life_m=12, maint_h=2.0)

def model(**o):
    p = {**base, **o}
    tok = p["calls"] * (p["in_tok"] * p["p_in"] + p["out_tok"] * p["p_out"]) / 1e6
    run = p["volume"] * tok
    value = p["volume"] * (p["manual_min"] - p["review_min"]) / 60 * p["wage"]
    rework = p["volume"] * p["err"] * p["rework_min"] / 60 * p["wage"]
    build = p["build_h"] * p["wage"] / p["life_m"]
    maint = p["maint_h"] * p["wage"]
    cost = run + rework + build + maint
    return dict(tok=tok, run=run, value=value, rework=rework,
                build=build, maint=maint, cost=cost, net=value - cost)

print({k: round(v, 2) for k, v in model().items()})
for r in (2, 4, 6, 8, 10):
    print("review_min", r, round(model(review_min=r)["net"]))

Running that script on 2026-11-03 printed the baseline: tokens 0.14 dollars per task, run cost 57.6 dollars a month, value of time saved 2,133.33, rework 200.00, build 100.00, maintenance 80.00, total cost 437.60 and net 1,695.73 dollars a month. The review loop then printed net values of 2,229, 1,696, 1,162, 629 and 96 dollars for review times of 2, 4, 6, 8 and 10 minutes.

Reading the baseline

The token bill is 57.60 dollars of a 437.60 dollar cost side, about 13 percent, and it is under 3 percent of the 2,133 dollars of value. Rework is the largest cost line at 200 dollars, then build at 100 and maintenance at 80. The headline is that the result is positive by a wide margin in this hypothetical. The useful question is how wide the margin really is, and for that the model needs to be pushed.

The sensitivity table: which input flips the result

Each row below changes one input and holds the rest at baseline. The break-even column is the value at which monthly net crosses zero, computed separately by bisection on the same model function. Each row also states how far that point is from the baseline input.

• Review time per task: baseline 4 minutes, break-even 10.36 minutes. Net at 8 minutes is 629 dollars and at 10 minutes is 96 dollars. Margin: review time must rise by about 6.4 minutes, to roughly 2.6 times the baseline.
• Manual time per task: baseline 12 minutes, break-even 5.64 minutes. If the manual task is really 8 minutes, net falls to 629 dollars. A baseline that was timed on slow or unrepresentative runs is the cheapest way to fool yourself.
• Task volume: baseline 400, break-even 38 tasks a month, because build and maintenance are fixed costs. At 100 tasks net is 289 dollars and at 50 tasks (with review at 4 minutes) it is 54.
• Error rate: baseline 3 percent, break-even 28.4 percent. At 15 percent net is still 896 dollars. In this hypothetical the error rate moves the result far less than review time does, but only because the rework cost per error is modest.
• Rework minutes per error: baseline 25, break-even 237 minutes. At 100 minutes net is 1,096. A rework cost this large would come from an error that reaches a customer, which belongs in a risk estimate, not a labor line.
• Token price: both prices multiplied together, break-even at about 30.4 times the Sonnet 5.5 list price, which would be roughly 4.38 dollars of tokens per task. At 10 times the price net is 1,177.
• Number of model calls per task: 8 at baseline. At 80 calls net is 1,177 and at 800 calls it is negative 4,007. Loops that multiply calls hurt only when they are very large.

Two inputs flip the sign within a plausible range of mistakes: review time and the manual baseline. Together they set the minutes saved per task (manual minus review), and the two-way grid shows how quickly the margin disappears when both are off. Net monthly dollars, with manual minutes down the side and review minutes across (2, 4, 6, 8, 10):

• Manual 6: 629, 96, negative 438, negative 971, negative 1,504.
• Manual 8: 1,162, 629, 96, negative 438, negative 971.
• Manual 10: 1,696, 1,162, 629, 96, negative 438.
• Manual 12: 2,229, 1,696, 1,162, 629, 96.
• Manual 14: 2,762, 2,229, 1,696, 1,162, 629.

Net depends on the difference between the two columns, so each diagonal of the grid is constant. Saved minutes per task of 2 give 96 dollars a month (manual 6 with review 4, or manual 12 with review 10), and break-even is at about 1.64 saved minutes per task at this volume. A workflow that saves under roughly two minutes per task at 400 tasks a month is on the line in this model, whatever the model vendor charges.

What the table says about model choice

Swapping the price tier barely moves the answer here. Using the Claude Haiku 5.5 list price for prompts up to 100,000 tokens (0.10 dollars input and 0.50 output per million, same pricing page, checked 2026-11-03) cuts token cost to about 0.0072 dollars per task and raises net from 1,696 to 1,750 dollars, a gain of 54. Using Claude Fable 5.1 (10 dollars and 50 dollars per million) raises token cost to 0.72 dollars per task and lowers net to 1,465 dollars, a loss of 231. The cheaper model only wins on ROI if its outputs need no more review than the expensive one. If a cheaper model adds one minute of review per task, that costs about 267 dollars a month at this volume, which exceeds the whole price difference between the tiers. Model choice belongs on the quality side of the ledger first.

How to use the model on your own workflow

• Time ten manual runs and ten agent-assisted runs before filling in the manual-time and review-time inputs. Use medians and keep the ranges.
• Log tokens from the API usage field for a week to replace the 8 calls and 6,000 tokens assumptions, then price them from the vendor page on a stated date.
• Count rework from incident notes, not from memory.
• Run the one-at-a-time table, find the input with the smallest margin, and spend your measurement effort on that input.
• Rerun after any model, price, volume or reviewer change, and record the date of the pricing page you used.

The pilot trap: why small trials mislead

Pilots systematically overstate agent ROI, and the reasons are structural. The pilot runs on the friendliest tasks, hand-picked to succeed. It gets the best people's attention - the team's strongest operators shepherd it, masking the usability problems average users will hit. Hawthorne effects inflate effort: everyone knows they are being measured, so they try harder. And pilots rarely run long enough to encounter the failure modes that dominate production: the weird edge cases, the data drift, the Monday-morning surprises. Then the rollout hits reality and the numbers collapse. Counter the trap deliberately: pilot on representative tasks, not showcase tasks; measure with the actual operators who will use it, not the champions; run long enough to see failures; and discount the pilot numbers explicitly - a common rule is to halve the time savings and double the cost estimate before projecting. A pessimistic pilot that still shows positive ROI is worth scaling; an optimistic one proves nothing.

• Representative tasks, not showcase tasks - the pilot should resemble Tuesday, not the demo.
• Real operators, not champions - the people who will actually use it, with their actual skill levels.
• Long enough to fail - edge cases and drift appear on week three, not day two.
• Discount explicitly: halve the savings, double the costs, then decide.
• Measure the failure paths too - pilot economics must include the escalations and rework.

Attribution: what the agent actually caused

The hardest ROI question is attribution: how much of the improvement came from the agent versus everything else that changed? Teams adopt agents alongside process improvements, training, and tooling upgrades, then credit the agent for the combined effect. Measure before and after on the same tasks: time the manual process before introducing the agent, with the same people, on comparable work. Where possible, run a holdout - one team or workflow without the agent - as a control; the difference between the groups is the agent's true contribution. Watch for confounders: if the agent rollout coincided with hiring better staff or dropping a difficult client, the numbers lie. Attribution does not need laboratory precision - it needs enough rigor that you would bet budget on the conclusion. Roughly right beats precisely wrong, but roughly right beats optimistically precise.

Non-financial returns that belong in the analysis

Strict financial ROI misses returns that matter. Consistency: the agent produces the same quality at 2 a.m. as at 10 a.m., which humans do not - the value of eliminating variance is real even when it is hard to price. Coverage: tasks that were never worth human time now get done - the backlog of small improvements, the documentation nobody wrote, the monitoring nobody staffed. Speed: turnaround measured in minutes instead of days changes what is possible, enabling workflows that were previously unthinkable. Morale: removing drudgery from skilled people's days has retention value no spreadsheet captures. None of these justify ignoring the financial math, but a project with marginal financial ROI and strong non-financial returns is often worth keeping - and a project with great financial ROI that burns out the team reviewing its output is not the bargain it appears.

Killing projects with bad ROI

The final ROI discipline is the kill decision. Sunk costs, champion egos, and the narrative that AI is the future keep underperforming agent projects alive long past the point the numbers turned red. Set the kill criteria before launch: the ROI threshold, the date by which it must be met, and who makes the call. When the date arrives, honor it - the criteria were set by your past self precisely because your present self would rationalize. Killing is not failure; it is the mechanism by which resources flow to the deployments that work. Document what the killed project taught you: the failure modes, the wrong assumptions, the tasks that turned out to be unsuitable. Organizations that kill cleanly learn faster than organizations that persist hopefully, and their agent portfolios compound while others stagnate.

Reporting ROI to skeptics

The numbers are only half the battle; the other half is presenting them so skeptics believe them. Lead with the methodology, not the result: how you measured, what you counted, what you excluded, and where the uncertainty lies. Skeptics distrust impressive numbers with vague methods far more than modest numbers with transparent ones. Show the range, not just the point estimate - best case, expected case, and the assumptions each depends on. Acknowledge the costs fully, including the ones that make the number worse: the failed pilot, the supervision time, the incidents. Counterintuitively, volunteering the bad news increases credibility for the good news. And separate one-time effects from run-rate: the first month's savings often include backlog-clearing that will not repeat. A report that a skeptic cannot poke holes in is worth more than a report with a bigger number - because the bigger number gets discounted to zero the moment trust breaks.

• Methodology first, results second - skeptics trust transparent modest numbers over vague impressive ones.
• Show the range: best case, expected case, and the assumptions behind each.
• Volunteer the bad news - full cost accounting increases credibility for the good news.
• Separate one-time effects (backlog-clearing) from sustainable run-rate savings.
• A hole-proof report beats a bigger number; trust discounted to zero is worth zero.

Sources and further reading

• Anthropic, Messages API reference (docs.anthropic.com/en/api/messages): the response object carries a usage field reporting input_tokens, output_tokens, and cache token counts for each request, which is what makes per-run cost a measurement instead of an estimate.
• AgentsUse, How Much Does It Cost to Run an AI Agent (this site): where agent spend actually goes - steps, context growth, tool calls, and retries - and the worksheet format the run-cost line above is priced from.
• AgentsUse, Agent Observability (this site): the run record that captures per-step model, token, tool, and timing data once and serves both cost measurement and failure debugging.