Building agents for on-call and incidents?

Why Resolve AI: Token Efficiency at Scale

Share:

Why Resolve AI: Token Efficiency at Scale

Why did token usage become the way the industry measures AI progress?

Because there was no agreed way to evaluate what an agent actually produced. Tokens are countable and outcomes were not, so consumption became the scoreboard by default. A token count tells you how hard a model worked on a problem. It tells you nothing about whether the problem got solved, and for anyone running agents against production systems that second question is the only one with a budget attached to it.

What is token efficiency?

Token efficiency is the least number of tokens you can spend per completed outcome. The definition only works if the outcome is a specific unit of work rather than a vague notion of usefulness. For Resolve AI the units are investigating an alert and getting you to the root cause of an incident. Fixing the outcome as the denominator means the number moves only when the agent gets better at reaching the same conclusion with less work, which is the property you want to optimize against. Cost per investigation becomes predictable, and quality gets measured on its own terms rather than being inferred from spend.

Where do agents actually waste tokens?

Context. The token bill is won or lost on what the agent decides to look at, how much of it it pulls into the window, and how many times it goes back for more. Two agents can run on the same model against the same incident and end up with very different costs, entirely because one made precise calls into code, infrastructure, and telemetry while the other pulled broadly and repeatedly.

What does an agent over-exploring or under-exploring look like?

The useful way to picture it is a detective.

  • An under-exploring agent takes the first plausible explanation it finds and closes the case. It produces a fast answer that is frequently the wrong one, and it looks efficient on a token report right up until someone checks the conclusion.
  • An over-exploring agent interviews everyone in town. It spends heavily, and it often still fails to commit to a conclusion at the end of it.

A good one works from a map of the city. It knows which sources matter for this class of problem, asks different kinds of questions across code, infrastructure, and telemetry, and stops when the evidence converges rather than when it runs out of places to look.

Why does agent quality peak instead of improving with more context?

Plot agent quality against the amount of context you give it and the shape is a hill, not a line.

  • Quality is low on the left because the agent has not seen enough to be right.
  • Quality is also low on the right because the relevant evidence gets diluted by irrelevant evidence and the reasoning degrades along with it.
  • The peak sits somewhere in the middle, and its position is not fixed. It moves with the class of problem, the model, and the quality of what was retrieved. Most of what Resolve AI's research team works on is finding that peak and staying on it.

How do you engineer an agent to stay near that peak?

Three things carry most of the weight.

  • Model selection is the first. Running the most capable model on every subtask is waste, so cheap tasks route to cheaper models and the expensive reasoning is held back for the steps that need it.
  • Tool calls are the second. Every call pulls its result into the context window, so an imprecise call costs tokens twice: once to make it, and again on every subsequent turn that carries the result forward. Precision at the call site is where budgets are protected.
  • Reuse is the third. An investigation that runs today should become context for the investigation that runs next month. Without that, the system pays repeatedly to rediscover the same service topology and the same failure patterns.

Why does Resolve AI charge on outcomes instead of tokens?

Because the vendor should carry the token risk, not the customer. Resolve AI charges for the amount of work completed, which means every inefficiency in the agent comes out of Resolve AI's margin rather than a customer's invoice. It is also the pricing model that keeps the research incentive honest. If a better retrieval strategy halves token spend on an investigation, that is a direct gain. If the agent over-explores, that is a direct loss.

What does outcome-based pricing change for a buyer building a budget?

The line item gets sized against work rather than consumption. A platform team budgeting for incident investigation can plan against the number of alerts and incidents they expect to handle, without modeling token spend per investigation or padding the number for a month where the agents run hot. How many tokens an investigation costs stays Resolve AI's problem.

See the agents that run and fix software in action

See the agents that run and fix software in action

Join our engineering leads for "Behind the Build", a webinar series deep-dive into how we built agents that run software.

Watch now