The incident is over, the service is back, and someone has to write the postmortem. That job is mostly archaeology: reconstructing what happened, in what order, across several machines, from logs nobody was reading carefully at the time. It is slow, and it is exactly the kind of structured summarisation an LLM is good at.
It is also where a model can do quiet damage. A postmortem is a record people act on — action items get funded, and "root cause" gets repeated in meetings for months. A model that turns "A happened, then B" into "A caused B" produces a confident document that sends the follow-up work in the wrong direction. This guide is about getting the speed without that failure: timeline first, citations everywhere, causation reviewed by a human.
Extract Exactly the Incident Window
Unlike live triage, you know the boundaries now. Pad them by fifteen minutes on each side — the precursor is usually just before the first alert — and pull only what matters:
# Find the right boot if anything rebooted during the incident
$ journalctl --list-boots | tail -3
-2 8d1c... Mon 2026-09-28 06:12:40 UTC Wed 2026-09-30 14:21:07 UTC
-1 4f7a... Wed 2026-09-30 14:23:55 UTC Wed 2026-09-30 14:58:12 UTC
0 b29e... Wed 2026-09-30 14:59:30 UTC Fri 2026-10-02 09:40:01 UTC
# The window, the units involved, warnings and worse, precise UTC timestamps
$ journalctl --utc -o short-iso-precise --no-pager \
--since "2026-09-30 13:45:00 UTC" --until "2026-09-30 15:15:00 UTC" \
-u nginx -u myapp -u postgresql -p warning > /tmp/pm-web1.log
# The kernel's view: OOM kills, disk errors, link flaps
$ journalctl --utc -k -b -1 -o short-iso-precise --no-pager \
--since "2026-09-30 13:45:00 UTC" > /tmp/pm-web1-kernel.log
-b -1 for the previous boot — if the host crashed or was rebooted mid-incident, the interesting part is in the boot before the current one. --list-boots shows which index covers your window.
short-iso-precise — gives microsecond timestamps with a timezone offset. Default short output drops the year and the zone, which is exactly what you need when merging hosts.
-p warning is a starting filter — it keeps warning, err, crit, alert and emerg. Run a second pass without it for the five minutes around the first symptom — the informational line saying a config was reloaded is often the most important one.
Persistent journal required — if /var/log/journal does not exist, the journal is in memory and was lost on reboot. Check this before the next incident, not during the postmortem.
Merge Multiple Hosts Into One Ordered Stream
Incidents rarely stay on one machine. The load balancer, the app servers, and the database each saw a piece. JSON output plus jq trims every entry to the fields worth reading and gives you a numeric timestamp to sort on:
$ for h in lb1 web1 web2 db1; do
ssh "$h" "journalctl --utc -o json --no-pager \
--since '2026-09-30 13:45:00 UTC' --until '2026-09-30 15:15:00 UTC' \
-p warning" \
| jq -c '{ts: (.__REALTIME_TIMESTAMP|tonumber), host: ._HOSTNAME,
unit: (._SYSTEMD_UNIT // .SYSLOG_IDENTIFIER), pri: .PRIORITY,
msg: .MESSAGE}' > "/tmp/pm-$h.jsonl"
done
# One stream, ordered by time, each line numbered for citation
$ cat /tmp/pm-*.jsonl | jq -s -c 'sort_by(.ts)[]' \
| jq -r '[(.ts/1000000|floor|todate), .host, .unit, .msg] | @tsv' \
| nl -ba -nrz -w4 > /tmp/pm-merged.txt
$ head -3 /tmp/pm-merged.txt
0001 2026-09-30T13:58:02Z db1 postgresql.service checkpoints are occurring too frequently
0002 2026-09-30T14:02:41Z web1 myapp.service pool timeout acquiring connection
0003 2026-09-30T14:02:41Z web2 myapp.service pool timeout acquiring connection
__REALTIME_TIMESTAMP — is microseconds since the epoch, as a string — hence tonumber. Sorting on it keeps sub-second ordering even though the printed time is rounded to the second.
Binary messages — MESSAGE is emitted as an array of bytes when it contains non-UTF-8 data. If jq prints numbers where text should be, that is why; such lines are rare and usually safe to drop.
Central collection is easier — if you already ship journals with systemd-journal-remote, journalctl -D /var/log/journal/remote on the collector reads all hosts at once and interleaves them by time.
The line numbers are the point — every claim in the postmortem will cite one. Without stable IDs, there is no way to check what the model based a sentence on.
Clock Skew Will Lie to You
The journal stamps each entry with the receiving host's clock. If db1 is three seconds fast, its errors appear to precede the app errors that actually caused them, and the merged timeline tells a false story — which the model will then explain fluently.
# On each host: how far from true time is this clock, and is it synced?
$ chronyc tracking | grep -E 'System time|Leap status'
System time : 0.000214 seconds fast of NTP time
Leap status : Normal
$ timedatectl | grep -E 'Time zone|synchronized'
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
Record the offsets in the postmortem — if two hosts disagree by more than the gaps between the events you are ordering, say so explicitly. Ordering claims inside that margin are not supported by the data.
Always --utc — a mix of local-time and UTC logs is the classic way to produce a timeline that is off by exactly one or several hours. Normalise before merging, never after.
Application timestamps differ — the time inside a log message is when the app wrote it; the journal timestamp is when journald received it. Buffered loggers can be seconds apart. _SOURCE_REALTIME_TIMESTAMP, where present, is the sender's own time.
Build the Timeline First, Conclusions Second
Do not ask for a postmortem in one shot. Ask for a factual timeline, review it, and only then ask for analysis. Separating the two stops the model from bending the facts to fit a story it has already chosen:
$ wc -c /tmp/pm-merged.txt # ~4 characters per token, roughly
$ ollama run llama3.2 "Below are log lines from an incident, merged from several
hosts and sorted by UTC time. Each line starts with an ID.
$(cat /tmp/pm-merged.txt)
Produce a TIMELINE only. Rules:
- One entry per significant event, in time order: time, host, what happened.
- Every entry must cite the line IDs it is based on, like [0002, 0003].
- Collapse repeated lines into one entry with a count and the first/last ID.
- Describe what the logs show. Do NOT state causes or use words like
because, caused, led to, or due to.
- If there is a gap of more than five minutes with no lines, note it as a GAP."
Banning causal words is deliberate — it keeps the timeline a list of observations. Causation is a separate, later question with its own evidence standard.
Gaps are information — five silent minutes on a host that normally logs constantly may mean it was hung, or that logging itself failed. A model left alone will smooth over the gap.
Add what the logs cannot show — deploys, config pushes, alerts, and the human actions from the incident channel belong in the timeline too. Paste them in with their own IDs, such as H01, so they can be cited the same way.
Spot-check citations — pick five entries at random and grep ^0142 /tmp/pm-merged.txt. A model that cites lines which do not say what it claims has failed the task, however polished the output.
Then the Postmortem Template
With a reviewed timeline in hand, the document itself is a structured rewrite. Give the model a fixed blameless template so the output is consistent from one incident to the next:
$ ollama run llama3.2 "Write a blameless postmortem from this reviewed timeline.
$(cat /tmp/pm-timeline-reviewed.md)
Sections, exactly:
## Summary (3 sentences, no jargon)
## Impact (who was affected, how, for how long — cite lines)
## Timeline (copy the reviewed timeline, do not alter it)
## Root cause (HYPOTHESIS: ... ; evidence: [IDs]; what would confirm it)
## Contributing factors
## What went well
## Action items (table: action | owner: TBD | priority | evidence)
Rules: cite line IDs for every factual claim. Describe systems and decisions,
never individuals. Where the logs do not establish something, write
NOT ESTABLISHED FROM LOGS instead of guessing."
| Section | What the model is good at | What a human must own |
|---|---|---|
| Summary / Impact | Condensing the timeline into plain language | Customer impact numbers from metrics and support |
| Timeline | Ordering, collapsing repeats, citing | Adding human actions and decisions |
| Root cause | Proposing hypotheses consistent with the logs | Deciding which one is true, and testing it |
| Contributing factors | Spotting warnings that preceded the failure | Judging which ones actually mattered |
| Action items | Suggesting candidate fixes | Owners, priorities, and whether to do them |
Owner: TBD on purpose — assigning work is an organisational decision made in the review meeting, not something a model should fill in from a hostname.
Blameless phrasing — "the deploy pipeline allowed a config without a pool limit" is actionable; "the engineer forgot" is not. The instruction to describe systems, never individuals, does most of this work.
Correlation Is Not Causation — the Review Pass
The most common defect in AI-drafted postmortems is a root cause section that names the first loud error rather than the cause. Postgres checkpoint warnings at 13:58 followed by app pool timeouts at 14:02 looks like a cause; it may equally be two symptoms of a third thing, such as a batch job that never logged anything to the journal at all.
Ask for alternatives — a follow-up prompt — "give three other explanations consistent with the same lines, and what evidence would distinguish them" — exposes how thin a confident root cause really is.
Demand a confirming test — a root cause you cannot reproduce or confirm with metrics stays labelled HYPOTHESIS. That label is honest and costs nothing.
Check what is missing — cron jobs, other teams' services, and network devices often do not log to these journals. If the window shows a sudden change with no visible trigger, the trigger is probably somewhere you did not collect from.
Keep it local — incident logs contain hostnames, IPs, usernames, and sometimes tokens. Run the model on your own hardware — see running LLMs locally or the air-gapped setup for stricter environments.
Mind the context window — Ollama truncates input beyond the model's context, silently. Raise it with PARAMETER num_ctx in a Modelfile, or pre-filter harder — the reduction techniques in our log summarisation guide apply directly.
For the live part of an incident — before there is anything to write up — see AI-assisted incident response. If an action item changes a service unit, debugging systemd with AI covers verifying that fix before it ships.
Frequently Asked Questions
Can an LLM write a postmortem from logs?
It can write a strong draft if you give it a bounded, merged, time-sorted extract with line IDs, and ask for a timeline before any analysis. It should not decide the root cause on its own; it proposes hypotheses, and the people who ran the incident confirm or reject them.
How do I extract logs for a specific incident window with journalctl?
Use journalctl --utc -o short-iso-precise --since and --until with explicit UTC timestamps, add -u for each relevant unit and -p warning to cut noise. If the machine rebooted, use journalctl --list-boots to find the right boot and -b -1 for the previous one. Pad the window by about fifteen minutes on each side.
How do I merge journald logs from several servers?
Export each host with journalctl -o json, use jq to keep __REALTIME_TIMESTAMP, _HOSTNAME, the unit, priority and MESSAGE, then sort all entries by the timestamp with jq -s 'sort_by(.ts)'. Check clock offsets with chronyc tracking first, because skew between hosts can reorder events.
How do I stop the model inventing a root cause?
Require a line ID citation for every factual claim, ban causal words in the timeline stage, label root cause as a hypothesis with a confirming test, and tell it to write NOT ESTABLISHED FROM LOGS where evidence is missing. Then spot-check citations against the file.
Is it safe to send production logs to an AI service?
Treat it as a data disclosure. Incident logs routinely contain internal hostnames, IP addresses, usernames and occasionally credentials. A local model via Ollama keeps the data on your hardware; if you must use a hosted service, scrub the extract first and check your organisation's policy.