Wake a human as a last resort: an AI first responder for infrastructure incidents

The on-call math that doesn't add up

Run infrastructure long enough and you learn the shape of your pager: most alerts are real but routine. A service instance leaks memory and needs a restart. A queue backs up and drains itself. A disk crosses a threshold. The classic setup pages a human for all of it, and the human performs the same ritual every time — open the dashboards, read the logs, check the runbook, apply the known fix, watch, close.

That ritual is precisely specified, safety-classified work. Which makes it exactly the kind of work an AI agent can do — provided you're rigorous about the word safe.

What happens when an alert fires

The incident manager is a set of agents triggered by our alerting rules on infrastructure metrics. Each alert starts the same disciplined sequence a good engineer would follow:

1. Read the incident

   The agent gathers the alert context: which system, which metric, what changed, what else is firing.

2. Investigate

   It inspects the relevant metrics and reads the server logs around the event window.

3. Consult the knowledge base

   Our troubleshooting KB — years of incident history and runbooks — is searched for matching signatures and known fixes.

4. Decide: safe or sensitive?

   Simple, non-critical operations (restart an application instance, clear a stuck worker) are on an approved list. Anything touching critical infrastructure is out of bounds by definition.

5. Act — or hand over

   Safe fix: the agent executes it. Sensitive: it writes a full summary — findings, log excerpts, hypothesis, suggested next steps — and pages the on-call engineer.

6. Verify, then close

   After a fix, the agent waits, re-checks the metrics, and only then closes the incident — posting a note to the internal comms channel. Not solved? It investigates further or escalates, case by case.

The guardrails are the feature

• An explicit allowlist of actions. The agent can only execute operations we've pre-classified as safe and non-critical. Everything else is read-only investigation.
• Escalation is a first-class outcome. Handing an incident to a human isn't a failure mode — for sensitive systems it's the designed behavior.
• Every step is logged. What the agent read, concluded, executed, and verified is in the incident record. The audit trail matches the standard of the ISO 27001 / SOC 2 environment it runs in.
• Verified closure. No incident closes on "fix applied." It closes on "metrics recovered, re-checked after a wait."

What we learned

• Classify before you automate. The project's hardest work wasn't agents — it was the honest inventory of which operations are safe to delegate. That classification is the system's real safety boundary.
• The knowledge base is the model's ceiling. Agent quality tracked our runbook quality almost exactly. Every incident that needed a human became a KB entry; the agents get better because the documentation does.
• The summary is a product, not a byproduct. We now write agent summaries to the standard of a hand-off between engineers — because that's literally what they are.
• Wait-and-verify beats fire-and-forget. The re-check loop catches the fixes that only looked like fixes, before they close an incident that isn't over.

Where this pattern applies

Any environment with mature alerting and written runbooks can adopt this pattern incrementally: start with agents that only investigate and summarize, then allowlist the first safe action, then expand as trust accumulates. On the Microsoft stack, the same architecture maps to Azure Monitor alerts, Log Analytics, and Teams for the notification channel — with Entra ID scoping what agents can touch.

AI agents · Incident response · SRE · Observability · Governance