An OOM-killer report is forty lines of hex masks, page counts and a process table, dumped into the kernel log at the exact moment something important died. A kernel panic is worse: a register dump and a call trace, often from a machine that rebooted before anyone looked. Both are dense, structured, and well documented — which makes them a good fit for a language model.
They are also easy to misread in ways that send you after the wrong process. A model will happily tell you the process named in the first line caused the problem, or read virtual size as memory use. The useful workflow is: collect the full report, let the model decode it, then confirm the cause yourself against /proc and the cgroup counters.
Collect the Evidence Before It Rotates
The kernel ring buffer does not survive a reboot. If the machine panicked or was power-cycled, the evidence is only where something persisted it:
# Kernel messages from the previous boot (needs a persistent journal)
$ journalctl --list-boots
$ journalctl -k -b -1 --no-pager -o short-iso > /tmp/kern-prev.txt
# Current boot, human-readable timestamps (can drift after suspend)
$ dmesg -T --level=emerg,alert,crit,err,warn
# Syslog copies: Debian/Ubuntu and RHEL-family respectively
$ grep -iE 'oom|panic|oops|BUG:' /var/log/kern.log
$ grep -iE 'oom|panic|oops|BUG:' /var/log/messages
# No /var/log/journal? Make the journal persistent for next time
$ sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald
A real panic usually kills the box before journald can flush, so the last lines on disk stop just short of the interesting part. Two mechanisms catch what the journal cannot:
# pstore: the kernel writes the oops to firmware/EFI storage on the way down
$ ls /sys/fs/pstore/
$ ls /var/lib/systemd/pstore/ # systemd-pstore archives entries here at boot
# kdump: a reserved crash kernel captures a full vmcore
$ cat /proc/cmdline | grep -o 'crashkernel=[^ ]*'
$ kdumpctl status # RHEL/Fedora
$ kdump-config show # Debian/Ubuntu (kdump-tools)
$ ls /var/crash/
# kdump also saves vmcore-dmesg.txt (RHEL) or a dmesg file (Ubuntu): analyse that.
# The vmcore itself needs the crash utility and matching debug symbols
$ crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore
crash> log # kernel log at time of death
crash> bt # backtrace of the panicking task
Test kdump before you need it — on a test machine, echo c | sudo tee /proc/sysrq-trigger forces a crash. If no vmcore appears in /var/crash afterwards, your kdump setup does not work and you have learned that cheaply.
Feed text, not binaries — a model cannot read a vmcore. Give it vmcore-dmesg.txt, the output of crash's log and bt commands, or the journal extract.
Anatomy of an OOM-Killer Report
Here is a trimmed global OOM from a 6.1 kernel. The shape is stable across versions even when individual columns change:
postgres invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
CPU: 2 PID: 4182 Comm: postgres Not tainted 6.1.0-18-amd64 #1 Debian 6.1.76-1
Call Trace:
...
Mem-Info:
active_anon:1843210 inactive_anon:12034 isolated_anon:0
active_file:1021 inactive_file:877 isolated_file:0
slab_reclaimable:10233 slab_unreclaimable:21877
mapped:1304 shmem:2311 pagetables:4210
free:21455 free_pcp:213 free_cma:0
Free swap = 0kB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 812] 0 812 12345 2301 126976 0 -250 systemd-journal
[ 4182] 105 4182 563210 41230 1003520 0 0 postgres
[ 5521] 1000 5521 3405112 1789003 15212544 0 0 java
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/app.service,task=java,pid=5521,uid=1000
Out of memory: Killed process 5521 (java) total-vm:13620448kB, anon-rss:7155964kB, file-rss:48kB, shmem-rss:0kB, UID:1000 pgtables:14856kB oom_score_adj:0
| Field | What it means | Common misreading |
|---|---|---|
X invoked oom-killer | the task whose allocation failed — the one unlucky enough to ask last | treating it as the culprit |
gfp_mask | allocation type; GFP_HIGHUSER_MOVABLE is an ordinary userspace page, GFP_KERNEL a kernel allocation | ignoring it when it points at a kernel or driver path |
order | size as 2^order pages; 0 is one 4 KiB page. High orders suggest fragmentation, not exhaustion | assuming every OOM means RAM is full |
oom_score_adj | bias from -1000 (never kill) to 1000 (kill first) | forgetting that -1000 hides the real hog |
total_vm / rss | task table values are in pages, not kB; rss is resident, total_vm is address space | multiplying wrong, or reading total_vm as usage |
constraint= | CONSTRAINT_NONE plus global_oom is whole-system; CONSTRAINT_MEMCG is a cgroup limit | calling a container limit a host memory shortage |
In this example postgres triggered the OOM, but java held 1,789,003 pages — about 6.8 GiB — and was chosen because its badness score (resident memory plus swap plus page tables, adjusted by oom_score_adj) was highest. Note also Free swap = 0kB: the box ran out of swap first.
Global OOM, cgroup OOM, and systemd-oomd
Three different mechanisms kill processes for memory, and they leave different fingerprints. Getting this wrong is the difference between "buy more RAM" and "raise one service's limit":
# 1. Global OOM: the whole machine ran out
Out of memory: Killed process 5521 (java) ...
oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,...
# 2. cgroup v2 limit (memory.max): only this cgroup ran out
Memory cgroup out of memory: Killed process 9910 (node) ...
oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/system.slice/api.service,...
memory: usage 524288kB, limit 524288kB, failcnt 0
# 3. systemd-oomd: userspace, pressure-based - NOT in dmesg at all
$ journalctl -u systemd-oomd -b
systemd-oomd[702]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/build.scope
due to memory pressure for /user.slice/user-1000.slice/[email protected] being 63.12% > 50.00% for > 20s with reclaim activity
$ oomctl # current monitored cgroups and limits
A cgroup OOM is a configuration event — the host may have gigabytes free. The fix is the service's MemoryMax=, the container's --memory, or the pod limit — not the hardware. Containers report it as exit code 137 and OOMKilled.
systemd-oomd kills whole cgroups — it acts on memory pressure (PSI) or swap usage before the kernel would, and it kills every process in the chosen cgroup. It is enabled by default on Fedora and recent Ubuntu, which surprises people who only check dmesg.
Look in both places — a service that died with status=9/KILL and nothing in the kernel log was probably oomd. Our systemd debugging guide covers reading the unit side of that story.
What to Feed the Model, and What to Scrub
Extract the complete report, from the invoked oom-killer line to the Killed process line, plus a little context either side. Then remove what identifies your infrastructure:
# Every OOM report in the previous boot, complete
$ journalctl -k -b -1 -o short-iso | sed -n '/invoked oom-killer/,/Killed process/p' > /tmp/oom.txt
# Scrub hostnames, IPs, MACs and hardware identifiers
$ sed -E \
-e "s/$(hostname -s)/HOST/g" \
-e 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/IP/g' \
-e 's/([0-9a-f]{2}:){5}[0-9a-f]{2}/MAC/gI' \
-e 's/Hardware name: .*/Hardware name: REDACTED/' \
/tmp/oom.txt > /tmp/oom-clean.txt
$ ollama run llama3.2 "Explain this Linux OOM-killer report.
$(cat /tmp/oom-clean.txt)
Context: 16 GiB RAM, 2 GiB swap, kernel 6.1, cgroup v2, app.service runs java.
Answer in this order:
1. TYPE - global OOM, cgroup (memcg) OOM, or something else? Quote the line that proves it.
2. TRIGGER - which task invoked the killer, and is it likely the cause?
3. VICTIM - what was killed and how much memory the kill freed (anon-rss).
4. LARGEST CONSUMERS - top 3 from the task table by rss, converted from pages to MiB.
5. MEMORY NOT IN THE TABLE - shmem, slab_unreclaimable, pagetables, in MiB.
6. HYPOTHESES - ranked, each with the command I should run to confirm or rule it out.
State the page size you assumed. Do not guess at anything not in the report."
Give it the machine size — without total RAM and swap the model cannot judge whether 6.8 GiB was a lot or a little.
Process names leak too — comm names and cgroup paths can contain customer or project names. Read the extract before it leaves the box, and prefer a local model for production logs — see running LLMs locally with Ollama.
For panics, include the taint line — Tainted: P O means a proprietary or out-of-tree module was loaded, which changes the most likely suspect. Also include the RIP: line, the call trace, and the Kernel panic - not syncing: reason.
For a broader method of feeding logs to a model, see AI log analysis on Linux; for privacy-sensitive hosts, run the model locally.
Where Models Get OOM Reports Wrong
Blaming the trigger — the task that invoked the OOM killer is whichever one allocated last. A small, healthy process often trips it while a large one holds the memory. Models read the first line as the headline.
Assuming the victim is the hog — usually it is, but an oom_score_adj of -1000 makes a process unkillable, so the kernel takes the next biggest. If the largest rss in the table belongs to a protected task, the victim was collateral damage.
RSS versus VSZ — total_vm (VSZ) counts reserved address space. A JVM with a large heap reservation or a Go binary can show huge total_vm and modest rss. Only resident and swapped memory matter to the OOM killer.
Pages read as kilobytes — the task table is in pages (4 KiB on x86_64); the final Killed process line is in kB. Models mix the two and produce numbers off by a factor of four or a thousand.
Memory nobody owns — tmpfs files, shared memory segments and kernel slab are not charged to any process in the table. If the table adds up to far less than RAM, the culprit is not a process at all — killing things will not help.
Inventing a leak — one OOM report is a snapshot and cannot show growth, so a model claiming "memory leak" from a single report is guessing. Log the suspect's rss every minute to find out.
Verify Against /proc and memory.events
Every hypothesis the model offers should end in a command. These are the ones that settle most OOM questions:
# Who is big right now, by resident memory (KiB)
$ ps -eo pid,user,rss,vsz,comm --sort=-rss | head
$ grep -E 'VmRSS|RssAnon|RssFile|RssShmem|VmSwap' /proc/5521/status
$ cat /proc/5521/oom_score /proc/5521/oom_score_adj
# Memory that belongs to no process
$ grep -E 'MemAvailable|Shmem:|SUnreclaim|SwapFree' /proc/meminfo
$ df -h -t tmpfs
# Global OOM kills since boot, and current memory pressure
$ grep oom_kill /proc/vmstat
$ cat /proc/pressure/memory
# cgroup v2: is this service hitting its own limit?
$ cat /sys/fs/cgroup/system.slice/app.service/memory.events
low 0
high 0
max 1843
oom 3
oom_kill 3
$ cat /sys/fs/cgroup/system.slice/app.service/memory.max
$ cat /sys/fs/cgroup/system.slice/app.service/memory.peak
$ systemctl show app.service -p MemoryMax -p MemoryCurrent
memory.events is the ground truth for cgroups — a rising oom_kill count confirms the limit, not the host, is the problem. A high max count with no kills means the cgroup is constantly reclaiming against its ceiling — a slowdown before an outage.
memory.peak needs a recent kernel — it appeared in 5.19. On older kernels, sample memory.current over time instead.
Write the fix as a limit — whatever the cause, set MemoryMax= and MemoryHigh= on the service so the next runaway kills itself rather than its neighbours. The systemd generator can produce the unit.
If the OOM was part of a larger outage, fold these findings into the timeline described in AI-assisted incident response.
Frequently Asked Questions
Is the process that invoked the OOM killer the one that caused it?
Not necessarily. The invoking task is simply the one whose memory allocation failed at that moment. The process holding the most memory is usually a different one, and it is the one with the highest rss in the task table that the kernel normally kills.
How do I tell a cgroup OOM from a system-wide OOM?
A cgroup OOM logs "Memory cgroup out of memory" and constraint=CONSTRAINT_MEMCG with an oom_memcg path, followed by the cgroup's usage and limit. A global OOM logs "Out of memory" with constraint=CONSTRAINT_NONE and global_oom. Confirm with the oom_kill counter in the cgroup's memory.events.
Why is there nothing in dmesg when my service was killed for memory?
It was probably systemd-oomd, which kills cgroups from userspace based on memory pressure and logs to the journal instead of the kernel ring buffer. Check journalctl -u systemd-oomd and oomctl.
Are the numbers in the OOM task table in kilobytes?
No. The Tasks state table reports total_vm and rss in pages, which are 4 KiB on x86_64, while pgtables_bytes is in bytes and the final Killed process line is in kB. Multiply rss by four to get KiB before comparing.
Can AI analyse a kernel panic vmcore?
Not directly. Use the crash utility with matching debug symbols to extract the log and backtrace, or use the vmcore-dmesg.txt file kdump writes alongside the dump, and give that text to the model. Treat its diagnosis as a hypothesis to check against the trace.