Every observability vendor now ships something called an “AI SRE.” Cloud providers have one. Incident-management platforms have one. Your monitoring tool probably added one last quarter.
The question for most engineering teams is no longer whether to bring generative AI into incident resolution. It’s how to do it without ending up with an expensive chatbot that engineers stop trusting after the second wrong answer.
That second wrong answer is the whole problem.
GenAI in operations doesn’t fail because the technology is useless. It fails when teams use it for the wrong job, adopt it in the wrong order, or skip the guardrails that make its output safe to act on at 3am.
The key is knowing what to automate first, what to verify, and when it’s safe to let AI take action.
First, Be Honest About What GenAI Is Good At
The results coming out of the industry are real. Microsoft’s Azure SRE Agent has collected diagnostics on more than 18,000 incidents and reports over 10,000 engineer-hours saved. Datadog’s Bits AI SRE reports up to a 95% reduction in time-to-resolution by testing hypotheses against live telemetry. PagerDuty built persistent memory into theirs because remembering what fixed an incident last time compounds with every new incident.
But there’s an equally important result: a rigorous 2025 evaluation by ClickHouse found that today’s LLMs, used naively, do not outperform expert SREs at diagnosis. On raw diagnostic accuracy, the experienced human still wins.
Where AI clearly helps is synthesis speed – turning forty noisy alerts, three dashboards, and a log stream into a coherent timeline, drafting the postmortem, and pointing to where to look first.
That’s the twenty minutes of legwork every incident begins with, and it’s the part that’s genuinely automatable today.
So our approach to GenAI in incident resolution is straightforward:
Accelerate the investigation. Draft the analysis. Keep a human on the decision.
Start with AI-Powered Telemetry Analysis
The lowest-risk place to introduce GenAI into incident management is simple: let it answer questions about data you already have.
A read-only agent can answer questions from your real event history:
“How many critical alerts did we have last week on production?”
“How was KubePodNotReady resolved last time?”
“Which rule fired most this month?”
This is where trust starts – and where the first failure mode appears: a chatbot with no grounding.
If the agent answers from a generic model of what a typical Kubernetes cluster looks like instead of your Prometheus, your alert history, and your runbooks, engineers quickly learn that it’s guessing.
The test is simple:
Can it answer a question your dashboards can’t, using data you already have?
Move to AI-Assisted Incident Investigation
Answering questions is useful, but incident resolution requires investigation.
The next step is having the agent investigate an alert like a good engineer: form explicit hypotheses, check each against live metrics, logs, and your actual SOPs, and show its reasoning.
The agent shouldn’t wander from tool call to tool call until it gets lucky. It should have a hypothesis, gather the relevant evidence, and explain how that evidence changes its assessment.
The failure mode here is the plausible story.
Ask a language model, “Why is the payments service down?” and it can produce a confident explanation whether or not the evidence supports it.
Plausible and correct are different things.
At this stage, the goal isn’t to have AI make the final call. It’s to make the investigation faster and more structured while keeping the evidence visible to the engineer handling the incident.
Verify AI-Generated Root Cause Analysis Before Acting
This is where the difference between a demo and a tool engineers trust becomes clear.
An AI-generated RCA needs to be verifiable.
First, use a citation contract: every factual claim – a metric value, pod name, deployment timestamp – must reference the specific tool call that produced it. Anything the agent believes but didn’t retrieve evidence for must be labelled as an unverified hypothesis.
A confident fabrication is more dangerous than an admitted gap because an on-call engineer may act on it.
Second, use a verifier that can’t investigate.
A second agent, with no tools, checks each claim against the recorded execution trace and classifies it as supported, unsupported, contradicted, or correctly labelled as a hypothesis.
A verifier with tools is another opinion. A verifier without tools is an auditor.
The result is a confidence badge on the RCA that means one precise thing:
These claims are traceable to evidence.
Not “this conclusion is correct.” Those are different statements.
Introduce Governed AI Remediation
Only once output is verifiable does it make sense for the agent to do anything.
Silencing a noisy rule, restarting a workload, or rolling back a deployment should become a proposal with a dry-run preview, approved by a person, and executed with a full audit trail.
The biggest risks here are architectural.
The investigation agent must be read-only by construction – separate credentials and a separate tool set from the agent that acts, rather than a permission flag on the same agent.
Actions should come from a fixed, audited catalog, not arbitrary scripts generated by the model.
Rare exceptions where automation should not wait for approval – such as regulatory-latency actions where hesitation itself is the risk – should be deliberate, documented, and few.
The more power you give an AI system, the more narrowly that power should be governed.
What Makes GenAI Adoption Work in Incident Management
Whatever stage you’re at, these are the questions we’d ask any vendor – including ourselves.
Where does alert content go, and is it treated as untrusted?
Alert labels and annotations originate in client clusters and reach the model verbatim. That makes them attacker-controllable input – a prompt injection in an annotation is a real attack surface.
Input guardrails matter, but so does tenant isolation enforced through credential scoping per environment, rather than an instruction in a prompt.
What does it cost per resolved incident?
Sending every alert to an LLM is a fast way to turn an AIOps rollout into a cloud-bill surprise.
The answer is statistical triage before the model gets involved – deduplication, entropy scoring, and Bayesian priority – so only alerts that deserve investigation consume AI spend.
Better triage is a measurable lever on cost per incident.
We’ll go deeper into AI-powered alert triage in the next post.
Does it remember?
Most deployments store every past RCA and never retrieve one. Every incident starts from zero.
The compounding value of persistent memory only exists if the outcome of each investigation – root cause, evidence, and what actually fixed it – becomes context for the next one, alongside related tickets already sitting in Jira.
Can you turn it on without it changing anything?
Any capability that could suppress an alert or modify an incident should be able to run in shadow mode first: classify, record, change nothing, and let you compare its verdicts against what your operators actually did before allowing it to act.
Where We Are With Our AI SRE Agent
Our conversational agent over real event history and hypothesis-driven RCA grounded in Prometheus metrics and SOPs are already live.
The next step is verified AI-generated RCA. The citation contract and tool-less verifier are designed and in progress and will ship in shadow mode first, allowing us to calibrate results against engineer judgement before introducing confidence badges.
We’re also designing governed remediation, with strict read/write separation as a hard prerequisite. The goal is for the agent to propose actions, show exactly what will happen, and execute only after the appropriate approval.
Long-term memory – including semantic recall of past RCA episodes and Jira similar-issue lookup – is on the same roadmap.
We’d rather show you where we actually are than claim the whole system is already solved.
How to Start with GenAI for Incident Resolution
Start small.
Connect what you already run – Prometheus, Alertmanager, and your runbooks – and measure whether the agent can answer questions your dashboards can’t.
Then move into grounded investigation. Once the investigation is reliable, add verification. Only after that should you consider giving the system permission to act.
At every stage:
Don’t ask AI to make a decision you haven’t yet learned to trust it with.
That’s how we think GenAI should enter incident resolution – not as a leap from monitoring to autonomous remediation, but as a gradual process where every new capability has to earn its place.
Keep an eye on the ITGix blogs for the next chapter.
