What we learned putting an LLM in charge of production servers

Letting a language model run commands on real infrastructure sounds reckless — and done naively, it is. But with the right scaffolding, an agent that triages logs and proposes fixes in plain English is genuinely useful. Here’s what we learned shipping OpsAgent.

Start read-only

The first version of OpsAgent could not change anything. It could read logs, list processes, and summarize metrics — nothing more. That constraint forced us to get the hard part right first: turning vague requests (“why is checkout slow?”) into precise, safe queries across a server fleet.

Only once the read path was trustworthy did we add a write path — and even then, behind a wall of checks.

Make every action reversible

The rule we never break: if an action can’t be undone, the agent doesn’t take it. Instead it drafts the change and asks a human.

• Every command runs as a dry-run first, and the diff is shown to the operator.
• Destructive operations require an explicit confirmation token.
• Each action is logged with the prompt, the plan, and the result — a full audit trail.

An agent you can’t audit is an agent you can’t trust. The log is the product.

Evals are not optional

We maintain a suite of scenario evals — real incidents replayed against the agent — that run on every change. A model upgrade that improves one task often regresses another; without evals you simply don’t see it.

The payoff is real: on-call engineers reach a root cause faster, and the boring 2 a.m. work of “grep the logs across twelve boxes” mostly disappears. But the safety scaffolding is what makes it shippable — not the model.