Runbooks fail for a boring reason: writing them is tedious and nobody does it while the knowledge is fresh. The result is documentation that describes a system from two years ago, which is worse than none — it gets followed.
AI removes most of the tedium, because the raw material already exists in your shell history, your scripts, and the Slack thread from the last incident. What it cannot do is know whether the procedure is correct, and that is exactly the part that matters at 3 AM.
Start From Real Material, Not a Blank Prompt
"Write me a runbook for restarting the API" produces plausible fiction. Feeding actual commands produces something grounded:
# What you actually ran, deduplicated, in order
$ history 200 | sed 's/^ *[0-9]* *//' | awk '!seen[$0]++'
# Everything you ran touching one service
$ history 2000 | grep -E 'myapp|systemctl|psql' | sed 's/^ *[0-9]* *//'
# The scripts that already encode the procedure
$ cat /opt/scripts/deploy.sh
# What the service actually is
$ systemctl cat myapp.service
History is the best source — it records what genuinely worked, including the diagnostic steps you took along the way — which is the part people forget to document.
Scrub before use — history contains passwords typed on command lines, tokens, and internal hostnames. Review the extract before pasting it anywhere, and prefer a local model.
Include the failures — the commands that did not work are valuable — they become the troubleshooting section.
The Structure That Makes a Runbook Usable
Ask for a specific structure. Left to itself a model produces prose; what an on-call engineer needs is a checklist with an escape hatch at every step:
$ ollama run llama3.2 "Write an operational runbook from this material.
$(cat /tmp/material.txt)
Use exactly this structure:
# <Title>
## When to use this
Symptoms and alerts that lead here. When NOT to use it.
## Preconditions
Access, credentials, approvals needed. What must be true before starting.
## Impact
What users experience while this runs. Is downtime involved?
## Steps
Numbered. Each step: the exact command, expected output, and what to do if it differs.
## Verification
How to prove it worked — specific commands with expected results.
## Rollback
How to undo every step, in reverse order.
## Escalation
When to stop and who to wake.
## Troubleshooting
Known failure modes and their fixes.
Rules: use only commands present in the material above. If a step is needed but
you do not have the command for it, write TODO: <what is missing> instead of
inventing one. Mark every destructive step with DESTRUCTIVE."
The TODO instruction is the important one — it converts the model's tendency to fill gaps with plausible commands into an explicit list of things you must supply. Those TODOs are your review checklist.
Expected output per step — "run this" is not a runbook. "Run this; you should see active (running)" lets a tired person tell success from failure.
Impact before steps — the reader needs to know whether this causes downtime before they start, not halfway through.
Verification Gates
A generated runbook is a draft. Three gates before it becomes something people follow:
1. Resolve every TODO — each one is a gap the model correctly refused to invent. Fill it from reality, not from memory.
2. Verify every command exists — run each one with --help, or check it against the system. Invented flags are the most common defect — see our command verification guide.
3. Walk it in staging — follow the runbook literally, as though you knew nothing, on a non-production system. Steps that assume unstated context reveal themselves immediately.
The third gate is the one that gets skipped and the one that matters most. A runbook that has never been executed end-to-end by someone following it exactly is an untested script.
Rollback Is Where AI Is Weakest
Models write confident rollback sections that do not work, because reversing an operation requires knowing what state existed before it — which is not in the material you provided.
Some things do not roll back — a completed database migration, a deleted file without a backup, a rotated credential already distributed. The honest rollback section says "not reversible — restore from backup", and a model will rarely say that unprompted.
Rollback needs its own preconditions — "restore the previous config" requires that a copy was taken in step 1. If the forward steps do not create it, the rollback is fiction.
Test rollback separately — walk the rollback in staging after walking the forward path. This is the single most valuable thing you can verify.
Write the backup step into the procedure — make step 1 an explicit copy — cp /etc/app/config.yml /etc/app/config.yml.$(date +%s) — so the rollback has something to restore.
Keeping Runbooks From Rotting
The same tooling that generates them can keep them honest:
# Keep runbooks in git next to the code they describe
docs/runbooks/
restart-api.md
rotate-db-credentials.md
recover-from-disk-full.md
# Flag drift: does the runbook still match reality?
$ ollama run llama3.2 "Compare this runbook against the current unit file and
script. List anything in the runbook that no longer matches, and anything in the
system the runbook does not mention.
RUNBOOK:
$(cat docs/runbooks/restart-api.md)
CURRENT UNIT:
$(systemctl cat myapp.service)
CURRENT SCRIPT:
$(cat /opt/scripts/deploy.sh)"
Version them with the system — a runbook in the same repo as the service gets reviewed when the service changes. One in a wiki does not.
Review after every incident — the moment you used a runbook is the moment you know what was wrong with it. Fix it while you remember.
Date and owner in the header — a runbook with no last-verified date gives the reader no way to judge how much to trust it.
Automate the drift check — a monthly systemd timer that runs the comparison above and emails the diff keeps documentation honest with no discipline required.
What AI Should Not Write
Approval and authority steps — who may authorize a production change is an organizational fact, not something to generate.
Escalation contacts — names, rotas, and phone numbers must come from your on-call system, not from a model.
Compliance and legal steps — notification obligations and retention rules are not inferable and are costly to get wrong.
Anything it invented — every TODO left unresolved is a landmine. A runbook with unfilled gaps should not be merged.
Judgement calls — "decide whether to fail over" cannot be automated into a step. Name the decision and who makes it.
Frequently Asked Questions
Can AI write a useful runbook?
It can produce a strong draft if you feed it real material — shell history, existing scripts, unit files, incident notes. Given only a description it produces plausible fiction. The generation saves time; the verification is what makes the result trustworthy.
How do I stop a model inventing commands in a runbook?
Instruct it to use only commands present in the material you supplied, and to write TODO plus a description of what is missing rather than filling gaps. Those TODOs become your review checklist, and any left unresolved should block the runbook from being merged.
What is the most important section of a runbook?
Rollback, and it is where AI is weakest — reversing an operation requires knowing the prior state, which is usually not in the source material. Make the first forward step create a backup so the rollback has something to restore, and walk the rollback in staging separately.
Should runbooks live in a wiki or in git?
In git, alongside the code or configuration they describe. That way they surface in code review when the system changes, which is the only reliable mechanism for keeping them current. Wiki pages drift because nothing forces a reader to update them.
How can I tell whether an existing runbook is still accurate?
Have a model compare it against the current unit files and scripts and report both mismatches and omissions. Running that comparison on a schedule and emailing the diff catches drift without relying on anyone remembering to check.