A cheap model can be the expensive choice.

That sounds backwards until an AI system has to do more than answer one prompt. An agent may make several model calls, call tools, recover from errors, retry a task, and carry an expanding conversation history into later turns. If it fails, all of that spend still happened. A token price tells you what one unit of model input or output costs. It does not tell you how much you must spend to get one piece of work that your system can actually accept.

That is the point of quality-adjusted inference cost: measure the resources needed to produce an outcome that meets a defined standard. You may also see it called cost per successful task, cost per correct answer, cost per accepted outcome, or cost-of-pass. The labels vary, but the engineering question is stable: what did it really cost to get one successful result?

AWS put that question into a concrete benchmarking harness published on September 11. Its evaluation measures cost per correct answer, agent trajectory cost, and rubric-graded professional deliverables while keeping evaluation logic constant across tested model backends. AWS’s description of the harness is timely because it treats an agent run as a complete operating path rather than a single inference request.

Plain English: Don’t ask which model produces the cheapest request. Ask which configuration produces accepted work for the least total spend.

The denominator is the important part

The simplest version is a ratio:

total directly attributable execution cost / number of accepted outcomes

The numerator includes every attempt, including failures. The denominator includes only outcomes that passed the acceptance rule. Suppose configuration A costs little per run but succeeds infrequently, while configuration B costs more per run but succeeds much more often. A can easily have the worse cost per accepted result.

For independent repeated attempts, a useful approximation is single-attempt cost divided by probability of success. It is an approximation, not a law: retries often fail for related reasons. A bad tool integration, missing permission, ambiguous task, or broken retrieval source can make failures correlated. In those cases, simply retrying may add cost without materially increasing the chance of a pass.

This metric matters because an AI service is rarely buying tokens for their own sake. It is buying incident summaries that are correct enough to route, migration plans that satisfy a review rubric, tests that pass, or changes that are accepted after review. OpenAI similarly describes cost per successful task as depending on price, computational expenditure, and the likelihood of reaching the correct result. Its guidance on third-party evaluations recommends considering expected cost per successful solve across repeated attempts when success can be measured, rather than presenting only a success rate at one fixed token budget. OpenAI’s scorecard and evaluation guidance make the same underlying case: performance and expenditure belong in the same decision.

Define success before measuring cost

The hard part is not division. It is deciding what “successful” means without quietly optimizing for the wrong thing.

Start with an outcome criterion that is independent of the cost measurement. For a code-maintenance workflow, success could mean a deterministic test suite passes, required static checks pass, and a reviewer accepts the change. For a support workflow, it might mean the answer contains required facts, does not violate policy, and receives a passing human or rubric-based assessment. A rubric-based grader is a structured set of criteria used to assess an output; it is useful when there is no single deterministic answer.

Then capture the full attempt cost. That usually includes input and output tokens, any applicable reasoning expenditure, tool execution, retries, and other resources directly attributable to the run. Keep external operational costs separate at first. Engineering time, platform hosting, supervision, integration, and incident handling are real costs, but adding them prematurely can obscure which agent behavior changed. They belong in a broader total cost of ownership analysis later.

Next, record the execution path, not only its final answer. An agent’s trajectory includes its turns, tool calls, retries, recovery behavior, latency, and context growth. AWS evaluates multi-turn web-research trajectories and notes that client-managed histories can resend accumulated context on later turns. That makes the number of turns and the size of prior context material cost drivers, not harmless implementation details. AWS explains the context effect here.

Plain English: A successful final answer can hide an inefficient journey. Record the journey, because repeated context, detours, and retries all appear on the bill.

Finally, compare configurations at an explicit acceptance threshold. The right choice is commonly the least expensive configuration that meets the required quality, latency, and reliability target. It is not automatically the cheapest model, the highest-scoring model, or the model with the lowest average request cost.

What this is not

Quality-adjusted inference cost is adjacent to several useful metrics, but it is not a replacement for all of them.

It is not token pricing. Token pricing is a marginal rate for model input or output. It helps estimate a request, but it cannot capture whether a model reaches a valid result, how many turns it takes, or how often it needs to try again.

It is not average cost per request. That average mixes passing and failing work, which is exactly what can hide an apparently cheap configuration’s poor outcomes. It is also not pass rate. A pass rate tells you how often something worked, but says nothing about whether it required a long, expensive trajectory to work.

Nor is it trajectory cost alone. A trajectory-cost metric tells you what a particular path consumed. Quality-adjusted cost asks the additional question that matters to a production owner: did that path yield an accepted outcome? Finally, it is narrower than total cost of ownership. It is a workload-level execution metric, not a complete financial model of an AI product.

A practical engineering example

Consider a hypothetical internal agent that prepares pull requests for a routine service migration. It reads a repository, proposes edits, runs available checks through tools, and produces a summary for a senior engineer.

A useful evaluation set would contain representative migration tasks, including straightforward changes, tasks with incompatible dependencies, and tasks where the correct response is to stop and explain a blocker. Define a pass before testing models: the change must satisfy deterministic checks where available; the summary must identify changed files and remaining risk; and reviewers must judge it acceptable against a fixed rubric.

For every attempt, log model input and output, tool calls, tool failures, retry count, elapsed time, and the amount of context carried into each turn. Also log the final acceptance result and the reason for rejection. Run each candidate model and policy over the same task set. A policy here means the operational rules around the model: its Prompt, whether it can retry, when it calls a tool, and how it manages Context.

Imagine one configuration produces inexpensive first attempts but frequently loops after a failing test, repeatedly resending repository material. Another spends more on its first attempt but stops after a focused diagnosis and produces accepted patches more often. Average request cost may favor the first configuration. Cost per accepted migration may favor the second. The trace explains why: the first configuration’s cheap start did not survive its retries and expanding context.

This setup also enables better routing. You might send easy migrations to the lower-cost configuration and route cases with certain dependency patterns to the stronger one. But do not declare routing a win from an average alone. Evaluate the mixed policy on the same acceptance criteria and confirm that it meets its latency and reliability constraints.

Plain English: Measure the whole workflow against work that matters. Then you can see whether a lower model bill is actually saving money or just buying more failed runs.

Where the metric breaks

A precise ratio can create false confidence. Its biggest weakness is the acceptance criterion. If the grader rewards superficial answers, a low cost per success may merely show that the system learned to satisfy a weak test. Binary pass/fail rules can also hide important quality differences above the pass threshold. Add rubric scores, human-review outcomes, downstream rework, or separate safety checks when those differences matter.

Aggregates can hide distinct task populations. A model may be excellent on simple repository edits and unreliable on dependency changes. Report results by task family, difficulty, failure mode, and relevant user segment rather than publishing only one portfolio ratio. Small samples are especially dangerous when pass rates are low or two candidates are close. Repeated runs and uncertainty estimates are necessary before making a consequential procurement or architecture decision.

Comparisons are deployment-specific. Different reasoning settings, context limits, tool implementations, caching, regions, provider infrastructure, and timeout policies can change the observed result. The metric is not an intrinsic property stamped onto a model. It is evidence about a model-plus-policy-plus-tools configuration under a particular workload.

There is also a temptation to force every outcome into dollars. Resist it. Latency, concurrency, reviewer burden, failure severity, and user trust may be constraints rather than ingredients in one blended score. A configuration with the best cost per accepted task can still be unusable if it misses an interactive latency target or produces rare but unacceptable failures.

What a senior engineer can do this week

Choose one bounded agent workflow with an existing source of truth for success. Do not begin with an open-ended assistant. A test-backed code task, document extraction task, or known-answer support classification is more useful.

Write the acceptance rule before choosing a model. Keep it stable for the first comparison. Instrument every attempt so you can reconstruct the trajectory and attribute its cost. Then run at least two model or policy configurations against the same representative tasks. Report four views together: acceptance rate, cost per accepted outcome, latency distribution, and failure reasons.

Use the results to change one operational decision: a retry cap, a Context-management rule, a model-routing boundary, or a tool permission policy. Re-run the evaluation after the change. This turns evaluation from a launch-time benchmark into an engineering control loop.

The useful shift is modest but consequential. Token prices still matter. They are just inputs to a larger calculation: the cost of getting reliable work done.