6 reasons DIY agents fail at investigating alerts and incidents

Why does AI flounder with production when it is so great with code? The answer is simple to state but hard to build for: Production is a distinct problem. Not harder; an entirely different problem.

The problem of production

Running software is where engineering time actually goes. About 70% of it, spread across maintaining platforms, scaling infrastructure, and monitoring deployments, as well as debugging incidents, shipping hotfixes, responding to alerts, updating runbooks, patching vulnerabilities, and restoring service.

None of that work belongs to a single team, either. One engineer can deploy a fleet of agents to build a new product in a day, but a single production incident routinely spans app, infra, database, and platform ownership before anyone can find the cause. A typical Sev1 pulls in eight senior engineers over 100 minutes: L1 support, infra, two app teams, database, an eng director, a comms manager, or an incident commander. No one has the full picture.

So why have we “solved” code with AI, while leaving production alone? Because code cooperates. It's self-documenting: names, types, and tests say what everything is supposed to do. It's modular, bounded, structured units a model can hold in context and reason about as a whole. It's one domain, the same material in every repo. And the feedback loop answers in seconds: write, run, pass, or fail.

Production is different on all four counts:

• Telemetry is infinite. A production system emits more log lines in a day than any context window will hold, and the volume scales with the system, not the problem. The evidence for one incident might be a few hundred lines hidden in tens of millions by a team member who’s no longer there or by coding agents. Intelligence is knowing what to query, when, and how to filter, not holding more context.
• Workflows are stateful across time and teams. An investigation accumulates state across services, handoffs, and hours, and every handoff leaks context. An agent with no memory of what it learned at 2 am starts from scratch at 4 am. So does the engineer who inherits its chat session.
• Tribal knowledge is load-bearing but invisible. The batch-job correlation, the node-pool memory leak: not in any log or runbook. They live with the engineer who's been on call for 18 months, and when that engineer isn’t on rotation that week, the gap shows up in time-to-resolution.
• Mistakes have asymmetric cost. A wrong answer in a code review costs a comment. A wrong action during a Sev1 costs revenue, customer trust, and sometimes a second incident. And there's no learning by trying when the stakes are high, and failure isn’t an option.

None of the four shows up in an afternoon prototype where it’s one engineer, one session, a narrow alert set, and nothing at stake.

What's below the waterline

A DIY agent can solve your trivial production problems. Point a frontier model at prod, give it more context, more tokens, more reasoning on every problem, and it demos beautifully, because a frontier model can do almost anything passably once.

However, production is where “passably” and “once” aren’t good enough. The answer has to be right, not plausible. There are no shades of grey, and it has to be right again tomorrow, on a different service and after a handoff. And if you err on the side of a plausible answer, production is unforgiving and where costs and mistakes compound.

Getting from passably once to reliably every time takes a purpose-built agentic harness, the layer of systems around the multiple models that a demo never needs. DIY builds skip it, and the six resulting gaps sit below the waterline.

1. Scattered knowledge

Tribal knowledge lives in individual team prompts, skills, code repos, knowledge sources, and with individuals, and the scattering follows the org chart. Each team owns a set of services, so each team captures the operational knowledge for those services in its own runbooks, context docs, and, these days, markdown files written for agents. Every team running its own AI setup makes it worse: in 2026, that's a CLAUDE.md, a prompt folder, and a skills directory per team. Your agent can only see the pile that your own team gave it.

Fixing it means two jobs:

• Gathering what's written down. Tedious, but a determined team can do it: crawl the repos, index the runbooks, and merge the prompts.
• Capturing what was never written down. The batch-job correlation in that one engineer's head only gets captured during investigations, by recording every correction an engineer makes and using it in the next one. That has to be a running system, not a one-time setup.

The gathering isn’t a one-and-done type of task, either. Services get renamed, deploys rewire the dependency graph, and the runbook you indexed in March is wrong by April. A knowledge base that has drifted is worse than none, because the agent trusts it.

Production needs a graph of services, dependencies, deploys, and team knowledge that stays current, and most DIY builds don't survive that requirement.

2. Shallow reasoning

Frontier models are designed to produce the most coherent answer to any prompt. In general tasks, that's exactly what you want, because there are many right answers and coherence is the quality bar. Production investigations are the opposite case: there's only one right answer, and what proves you found it is a causal chain of citable evidence leading from symptom to root cause, not a fluent explanation.

Without an architecture built to force that chain, anchoring bias takes over. The model over-indexes on the first plausible answer, stops looking, and presents the confident wrong answer. In some cases, the on-call engineer applies the wrong fix, which can become catastrophic. But in most cases, the on-caller becomes the tester of the agent's first guess: the human is the investigation loop. Verifying a confident hypothesis against production can take longer than investigating unaided. In both cases, trust quickly erodes, and teams begin to quietly abandon the tool.

A multi-agent system that actually tests alternatives leaves a different artifact:

• An evidence timeline, so you can see what was checked and when. It serves both sides: different agents reason over the evidence against the hypothesis they are testing, and the human verifies against it.
• A causal chain, every link tied to its evidence. A strong chain also allows the system to hold its ground under pushback instead of folding the way a chat model does.
• Disconfirming checks, where an independent agent spends queries trying to kill the hypothesis before the system commits to one.

A first-guess system produces none of that.

3. Models are not trained for this domain

General-purpose models aren't post-trained for production troubleshooting. Their weights come from documentation and tutorial-grade fixes, so when a symptom matches a common cause, they anchor on it. For the tasks at the core of an investigation, no frontier model cuts it. Those need models purpose-built for production reasoning, where no two production environments are the same.

Building custom models is a research project in its own right. An open-ended investigation has no answer key to score against, so you need custom verifier models to judge them. The verifier itself is a trained system built on extensive investigation data most teams don't have.

Then there's selecting between models. Every choice trades off quality, latency, and cost, and production investigation is many tasks, not one: reasoning over a dashboard image is different work from reasoning over logs, and a routine triage doesn't justify the model a novel Sev1 needs. Getting all three right means breaking the problem into tasks and matching each to the model that wins its tradeoff. And the picks don't stay right, either. Models change every month, so the matching is continuous.

What the domain needs is all three running as one system: models post-trained on production investigations, verifiers trained to judge them, and an orchestrator that updates as models change. A DIY build takes each on as a permanent job.

4. No evals

With no way to measure whether a change improved or degraded the agent, teams change blindly. You can't tell whether you're at 50%, 70%, or 90% accuracy without measuring, and every model upgrade becomes a blind migration.

The fix is measurement infrastructure, not a prompt change. A production eval platform needs to confirm five things:

• It got the answer right.
• It avoided the confident wrong answer, abstaining when the evidence isn't there.
• It took the right steps to get there, not just the right conclusion from the wrong path.
• It expressed the right confidence in the answer it gave
• It still holds SOTA quality as models and architectures change underneath it.

It can do that through positive and negative evals, alongside calibration. Positive evals let you know the agent correctly identifies the root cause; negative evals and calibration catch different failures. Negative evals capture cases where the system should have said it doesn't know. Calibration catches the case where it says 90% confident and is right 60% of the time.

Evals also become rot. Models get stronger and saturate the cases, and your environment drifts away from the conditions they encode. Eval generation has to run continuously, regenerating cases to track what production looks like now, not a golden set you build once.

5. Token explosion

Agents spend tokens at every step of an investigation.

• Context loads first with the system prompt, runbooks, and service metadata.
• Then tool calls, a lot of them. Production evidence is fragmented across observability, infra, code, and deploy systems, so each needs a tool call. Then every result means more context.
• Then reasoning, taking in everything within the context and tool calls.
• Finally, action, with tool calls again to implement fixes.

The bill also arrives twice. Model usage grows because the logs are the tokens, and observability costs grow because the agent is hammering the same metered query APIs your engineers use.

At demo volume, ten investigations a week, nobody notices. At production volume, a frontier model on every task does not survive a budget review. The pattern has a name, tokenmaxxing: more context, more tokens, more reasoning on every problem, because that's what made the demo work.

The build that survives that review targets a flat cost per investigation, whether you run ten a week or thousands. That means retrieving only what the current hypothesis needs, rather than stuffing the context window. Telemetry is infinite, but your budget isn't.

6. Security and compliance

Agents with production access are new principals in your security model, and most DIY builds treat them as scripts running under a human's credentials. Each agent needs its own identity, with permissions scoped at two levels:

1. What the agents themselves are allowed to touch
2. What they can touch on behalf of the engineer who invoked them.

An engineer with read access to one namespace shouldn't inherit cluster-wide reach because the agents' service accounts have it.

Integrations are where that scoping gets hard. The fastest DIY path is to wire up general-purpose connectors, and for a demo, it works. But each integration needs to run with least-privilege credentials scoped by your team, revocable at any time, with reads and writes on separate rails. Every integration that bundles those together or defaults to broad OAuth scopes creates a surface area that credential scoping alone doesn't close in a system that already has production access.

Write operations are where the architecture either holds or doesn't. The end state of any production agent worth building is closed-loop remediation, where the agents don't just surface a fix but execute it. Getting there safely requires a hard separation between the models that reason and the engine that acts, plus a policy layer that sorts every action into one of three categories:

1. Autonomous: Safe enough to execute within predefined guardrails, with no human involved.
2. Approved: Requires a human-in-the-loop to sign off before execution begins.
3. Observed: Not interruptible, but a human monitors what the agents did rather than authorizing each step.

A DIY build that collapses those into a single approval gate, or skips the policy layer entirely, can never safely expand its autonomy. The categories have to be designed from the start, because the system's ability to earn trust over time depends on the precision of that boundary.

Secrets, tokens, and customer data leak into logs despite best efforts, so telemetry needs a redaction layer before any of it reaches the models. The layer sits at the boundary, stripping sensitive fields before data transmits anywhere, with customer-configurable patterns on top of the defaults.

Audit is where DIY builds fail first. Most harnesses log tool calls, and auditors ask why an action was taken, not just what it was. The system that passes can produce, on demand, every action the agents took in production, the reasoning behind each one, and the human who approved it.

Underneath all of it sits infrastructure governance. A DIY system can send production telemetry to model providers on every investigation, so data residency, retention, and key ownership questions don't go away; they move to your LLM vendor relationships. None of this is a milestone you pass once. Credential rotation, access reviews, and evidence collection are standing work, which is why a compliant AI layer is a build measured in quarters, not sprints.

3 common ways it breaks in practice

A team that has read this far might still build anyway. Gather the knowledge, add the evals, and then watch the token bill. Each gap looks like a line item you can work through. The problem is that the gaps don't fail one at a time. They combine during incidents and form the same three shapes every time.

The failures below aren't hypotheticals. They're the pattern we see when teams run DIY agents against real production, and each one traces back to several gaps at once.

The confident wrong answer

The system surfaces a hypothesis. Well-formatted, cites real log evidence, sounds authoritative. The junior engineer on call acts on it. The actual root cause was three services upstream. Cost: 40 minutes of a live incident.

Nothing here is hallucinated. Every citation checks out. So spot-checking the evidence, the standard verification habit, passes. A senior engineer has seen enough incidents to distrust a tidy answer. A junior engineer checks the citations, verifies they are real, and acts. The failure selects for the person least equipped to catch it.

The root cause sat three services upstream because the agent reasoned one hop at a time. Each piece of evidence was correct for its service, but the conclusion was wrong for the system. Models optimize for coherent output, and in production, coherence and accuracy diverge under pressure and incomplete information.

The investigation that dies on a laptop

The tool works for Engineer A on their services in their session. The investigation lives on one laptop. Context resets on every handoff. Even when it works, it only works in a very limited form.

Production is a multiplayer problem, and most AI tooling is single-player by design. Local tools scope state to a session on purpose, for privacy and simplicity. Making an investigation durable and shared changes the data model, the permissions model, and the product. That's a rebuild, not a feature request.

A shared investigation has to persist on two timescales:

• Within an incident: what the agent learned at 2 am has to survive the 4 am handoff
• Across incidents: this month's outage has to be able to find a similar one from March

Either way, the investigation is a team artifact: an evidence trail, conclusions, and a causal chain that outlive the session and stay accessible to the next on-call engineer.

The action that didn't know what it didn't know

An agent with tool access and a planner executed an action based on an incomplete understanding of the system. It missed a nuance in a critical batch job that another team had mentioned on Slack months earlier. The "reasonable" action caused a system failure because the deciding context was institutional and never in the logs.

The action was defensible on every piece of data the agent could see. The failure was a lack of knowledge the agent had no way to know existed, and a smarter model doesn't fix that.

Getting the right context in is a knowledge architecture problem: raw APIs are messy and not designed for machine consumption. Keeping bad actions out takes approval gates, which are harder to build than the AI itself and which most teams treat as afterthoughts.

What building AI for production actually costs

DIY can be done in a day for a few hundred dollars in tokens. An actual AI for production agent takes months and millions of dollars to build.

Let’s price it for a mid-size company: 200 engineers, a regulated industry, and standard alert volume. Year one lands around $5M, and steady state runs about $4M a year after that. Where the money goes matters more than the total:

Line item	Ballpark	What it covers
People	~$1.8M year one	A 5-person build team (3 AI engineers, platform/security, UI/UX) for a three-month v1, then 2.25 FTE of maintenance
Evals	~$1.8M year one	~$300K to build the platform, then a 2.25 FTE program (eval platform eng, quality/ML eng, SRE annotation time) plus compute and tooling
Infrastructure	~$1.2M/yr	Cluster compute times a 2-2.5x multiplier for storage, transfer, and managed services
Ops tooling	~$350K/yr	Observability, security, and compliance tooling, each around 15% of infra spend
LLM tokens	<$100K/yr	~$60 per alert and ~$150 per incident investigation at production cache rates

Tokens, the line everyone budgets for, is the smallest number on the list. A hundred investigations a month run about $6K. That assumes the context discipline from the token explosion section, though. Without it, this line stops being small.

The real spend is on people. Steady state is roughly 4.5 FTE of specialized engineers before anyone touches a feature. The maintenance half is required regardless of how well v1 ships:

• Model swaps: providers deprecate models once or twice a year, and each migration costs 2-4 weeks of senior eng time
• Prompt drift: monitoring and rewriting as usage patterns shift, never done
• On-call for the AI itself: when the system hallucinates at 2 am, someone gets paged. About 20% of an FTE
• Data pipeline maintenance and multi-provider failover

The eval half is where hiring gets hard. Building agentic evals is research-grade ML systems work: trace-replay infrastructure, LLM-as-judge calibration, statistical significance testing, and meta-eval to ensure eval scores track production outcomes. The people who've done it work at frontier labs and on senior FAANG infra teams, at $600K+ fully loaded. Hiring juniors at startup salaries gets you flaky eval runners nobody trusts.

And all of this buys a functional system, one that investigates alerts, calls tools, and produces reasonable triage. Reaching state of the art adds what the line items don't price:

• Post-training models on your production data, which runs into the millions
• A knowledge graph of services, ownership, and dependencies that stays current
• Multi-agent architecture, with specialized agents across logs, metrics, traces, and code, and a separate agent checking the result

Teams that get there typically spend 18-36 months past "functional" before the system is trusted to act on its own. The demo is the easy 80%. Investigating production is the 20% that's 100x the work.

What Resolve AI builds so you don't have to

That harness is what Resolve AI is. Post-trained models, a knowledge graph that stays current, causal reasoning with the evidence attached, governed actions behind approval gates, a learning loop, and the eval platform underneath, running as one system instead of six projects.

It's working in demanding production environments. Zscaler runs 150K+ monthly alerts through it. Coinbase cut MTTR by 72% across 100+ engineers. DoorDash took investigations from 40 minutes to 5.

The operating model is the one this whole article points at: agents should run and fix your software, and engineers should run the agents. That only works when the agents have earned it.

Wire a model to your logs, and you'll have a demo by Friday. That part was always true. Whether it becomes something an engineer trusts at 3 am depends on the harness around the model, and now you know what that system costs. If you'd rather not spend 18 months building it, we should talk.