The instinct is to pipe a log file into a model and ask what went wrong. It does not work: a busy server produces more log data in an hour than any context window holds, and truncating to fit means the model reasons about an arbitrary slice.
The technique that does work inverts the order — reduce with classic Unix tools first, then ask the model about the reduction. You get better answers, on more data, for free.
Reduce Before You Summarize
Aggregation is where the information density is. A million log lines usually contain a few dozen distinct patterns, and those patterns fit comfortably in a prompt:
# Distinct messages with counts — the single most useful reduction
$ journalctl -u myapp --since "24 hours ago" --no-pager \
| sed -E 's/[0-9a-f]{8}-[0-9a-f-]{27}/UUID/g; s/\b[0-9]+\b/N/g' \
| awk '{$1=$2=$3=""; print}' \
| sort | uniq -c | sort -rn | head -40
The sed step is what makes this work: replacing UUIDs, IDs, and numbers with placeholders collapses thousands of near-identical lines into one counted pattern. Without it, every line is unique and the count tells you nothing.
Normalize, then count — timestamps, request IDs, IP addresses, and durations all need masking before aggregation. This is the step people skip.
Strip the timestamp prefix — the awk clause drops the first three fields (date, time, host) so identical messages group together.
Read the top 40, not the top 5 — the interesting error is rarely the most frequent one — noisy warnings dominate the head of the list.
A Reduction Pipeline You Can Reuse
#!/usr/bin/env bash
# logsum.sh — reduce a unit's recent logs to something a model can reason about
set -euo pipefail
UNIT="${1:?usage: logsum.sh <unit> [since]}"
SINCE="${2:-2 hours ago}"
norm() {
sed -E \
-e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/UUID/g' \
-e 's/([0-9]{1,3}\.){3}[0-9]{1,3}/IP/g' \
-e 's/\b[0-9]+(\.[0-9]+)?(ms|s|MB|GB|KB)?\b/N/g' \
-e 's/0x[0-9a-f]+/HEX/g'
}
echo "### VOLUME BY PRIORITY"
for p in err warning notice; do
printf '%-8s %s\n' "$p" "$(journalctl -u "$UNIT" --since "$SINCE" -p "$p" --no-pager -q | wc -l)"
done
echo
echo "### TOP ERROR PATTERNS"
journalctl -u "$UNIT" --since "$SINCE" -p err --no-pager -q \
| norm | awk '{$1=$2=$3=""; print}' | sort | uniq -c | sort -rn | head -25
echo
echo "### TIMELINE (errors per hour)"
journalctl -u "$UNIT" --since "$SINCE" -p err --no-pager -q -o short-iso \
| cut -c1-13 | sort | uniq -c
echo
echo "### FIRST AND LAST ERROR"
journalctl -u "$UNIT" --since "$SINCE" -p err --no-pager -q | head -3
echo "..."
journalctl -u "$UNIT" --since "$SINCE" -p err --no-pager -q | tail -3
That output is typically a few hundred lines regardless of whether the source was ten thousand or ten million — and it preserves what matters: what happened, how often, when it started, and the exact first occurrence.
Then Ask the Model
$ ./logsum.sh myapp "6 hours ago" > /tmp/summary.txt
$ ollama run llama3.2 "You are an SRE analyzing a reduced log summary.
$(cat /tmp/summary.txt)
Answer as:
1. WHAT BROKE - the most likely primary failure
2. EVIDENCE - which specific lines and counts support that
3. CAUSAL ORDER - which errors are the cause and which are downstream effects
4. NEXT COMMANDS - what to run next to confirm
5. UNCERTAIN - what the summary does not tell you
Base every claim on the data above. Do not speculate beyond it."
Causal order is the real value — a cascade produces hundreds of connection-refused errors downstream of one process crash. Sorting cause from effect is exactly the kind of pattern reading models do well.
Ask what the data does not show — it prevents confident conclusions from a summary that happens to exclude the relevant service.
Keep it local — logs contain hostnames, IPs, user identifiers, and sometimes tokens. A local model means none of that leaves the machine — see our Ollama guide.
Structured Logs Are Easier
If your application emits JSON, skip the regex normalization — jq does the aggregation precisely:
# Count by error type and message
$ jq -r 'select(.level=="error") | "\(.error_type) | \(.msg)"' app.log \
| sort | uniq -c | sort -rn | head -20
# Errors per endpoint
$ jq -r 'select(.status>=500) | .path' access.log | sort | uniq -c | sort -rn
# Slowest requests, deduplicated by route
$ jq -r 'select(.duration_ms>1000) | "\(.duration_ms) \(.route)"' app.log \
| sort -rn | head -20
# journald's own JSON output
$ journalctl -u myapp --since "1 hour ago" -o json --no-pager \
| jq -r '"\(.PRIORITY) \(.MESSAGE)"' | sort | uniq -c | sort -rn | head
Our JSON/YAML converter helps when inspecting a single event, and the regex tester is useful for building the normalization patterns above without guesswork.
Automating Daily Triage
The practical payoff is an unattended job that reads overnight logs and tells you whether anything needs attention:
#!/usr/bin/env bash
# nightly-triage.sh — summarize, classify, alert only if warranted
set -euo pipefail
SUMMARY=$(/opt/scripts/logsum.sh myapp "24 hours ago")
VERDICT=$(ollama run llama3.2 "Classify this log summary.
$SUMMARY
Reply with exactly one line: either
OK: <one sentence>
or
ATTENTION: <one sentence naming the problem>")
echo "$VERDICT"
if [[ "$VERDICT" == ATTENTION:* ]]; then
printf '%s\n\n%s\n' "$VERDICT" "$SUMMARY" \
| mail -s "[triage] myapp needs attention" [email protected]
fi
Constrain the output format — a single line with a fixed prefix is testable in bash. Free-form output cannot drive a conditional.
Alert on ATTENTION only — a job that emails every night gets filtered to a folder within a week and stops being read.
Keep the raw summary in the alert — the human receiving it needs the evidence, not just the verdict.
Run it under a systemd timer — with Persistent=true so a missed window still runs — see our timer generator.
This is triage, not monitoring — it complements real alerting on metrics; it does not replace it. A model summary will not page you within 60 seconds of an outage.
Mistakes That Waste Time
Feeding raw logs — truncation means the model sees an arbitrary slice, usually the least interesting part. Always reduce first.
Not normalizing before counting — without masking IDs and numbers, every line is unique and uniq -c returns a count of 1 for everything.
Asking an open question — "what is wrong with these logs" invites a generic essay. Ask for specific fields with cited evidence.
Trusting a root cause without checking — the model reads correlation. Confirm with the follow-up commands it suggests before acting.
Sending logs to a cloud API — logs are among the most sensitive data on a server. If you must, scrub aggressively first — but local is simpler and cheaper.
Ignoring the time window — "since yesterday" hides whether errors are constant or started 20 minutes ago. The per-hour timeline answers that.
Frequently Asked Questions
Can I just paste a log file into an LLM?
Not usefully for anything large. Server logs exceed any context window, so the model sees a truncated and arbitrary slice. Reduce first with grep, sed normalization, sort, and uniq -c, then ask about the aggregate — which fits easily and carries more information.
Why do my log counts show 1 for every line?
You are counting before normalizing. Timestamps, request IDs, UUIDs, IP addresses, and durations make every line unique. Mask them with sed first, then sort | uniq -c collapses thousands of near-identical lines into counted patterns.
What is AI actually good at in log analysis?
Separating cause from effect in a cascade, recognizing which of many error patterns is the primary failure, and suggesting the next diagnostic command. It reads patterns well; it cannot confirm a root cause, so verify before acting.
Should I use a local model or a cloud API for logs?
Local. Logs routinely contain hostnames, internal IPs, usernames, and occasionally tokens, and sending them to a third party is a data disclosure. A local model also removes per-token cost, which matters when you run triage on a schedule.
Can this replace my monitoring and alerting?
No. It is triage over historical logs, useful for a daily digest and for post-incident review. Real-time paging still needs metric-based alerts; a scheduled summary will not tell you about an outage within seconds.