systemd failures are a good fit for an LLM assistant, because the hard part is usually translation rather than reasoning. status=203/EXEC and Failed at step NAMESPACE are precise, documented conditions that simply do not read like the problems they describe.
The catch is that a model asked about systemd will confidently invent directives that do not exist. This is the workflow that gets the benefit while catching that.
Collect Context First
The most common mistake is pasting only the error line. A model given the unit file, the status, and the log almost always identifies the problem; given one line, it guesses. Gather everything in one go:
svcdump() {
local unit="$1"
echo "=== UNIT FILE ==="
systemctl cat "$unit" 2>&1
echo
echo "=== STATUS ==="
systemctl status "$unit" --no-pager -l 2>&1
echo
echo "=== RECENT LOG ==="
journalctl -u "$unit" -n 80 --no-pager 2>&1
echo
echo "=== UNIT VERIFY ==="
systemd-analyze verify "$unit" 2>&1
echo
echo "=== DEPENDENCIES ==="
systemctl list-dependencies "$unit" --no-pager 2>&1 | head -20
}
$ svcdump myapp.service > /tmp/myapp-debug.txt
systemctl cat, not cat — it shows the unit plus every drop-in override in load order. A drop-in you forgot about is a common cause of behaviour that contradicts the unit file you are reading.
Include systemd-analyze verify — it catches typos and unknown directives before the model does, and it is authoritative.
Scrub before sending anywhere non-local — unit files contain paths, usernames, and sometimes credentials. This is a strong argument for a local model.
Ask With the Right Framing
$ ollama run llama3.2 "You are a systemd expert debugging a failing unit.
$(cat /tmp/myapp-debug.txt)
Respond as:
1. ROOT CAUSE - the single most likely cause, citing the specific log line
2. FIX - the exact change, as a unit-file diff
3. VERIFY - the command that proves the fix worked
4. UNCERTAIN - any directive or claim you are not fully confident about
Only use real systemd directives. If you are unsure a directive exists, put it in UNCERTAIN."
Requiring a cited log line is the highest-value instruction here. It forces the answer to be grounded in your actual output instead of a generic systemd troubleshooting essay, and it makes a wrong answer obvious immediately.
What This Is Genuinely Good At
Decoding exit codes — 203/EXEC, 200/CHDIR, 217/USER, 226/NAMESPACE — each maps to a specific condition, and a model recalls the mapping instantly.
Sandboxing failures — the hardening directives (ProtectSystem, ReadWritePaths, PrivateTmp) fail in ways that look like permission bugs. Explaining which directive blocked which path is a strong use case.
Ordering and dependency problems — the difference between After=, Requires=, and Wants=, and why a service starting before the network is up needs network-online.target.
Type= mismatches — a forking daemon declared as Type=simple looks like it starts then immediately dies. Models spot this reliably.
Timer syntax — reading and writing OnCalendar= expressions — verifiable instantly, which makes it low risk.
Verify Before You Apply
Every suggested change gets checked against systemd itself, not against the model's confidence:
# 1. Does the directive exist? (empty output = it does not)
$ systemd-analyze verify /etc/systemd/system/myapp.service
# 2. Confirm the directive is real and does what was claimed
$ man systemd.exec | grep -n 'ReadWritePaths'
$ man systemd.service | grep -n 'Type='
$ man systemd.unit | grep -n 'After='
# 3. Check a calendar expression against real dates
$ systemd-analyze calendar 'Mon *-*-* 03:00:00'
# 4. Show the effective configuration after a change
$ systemctl show myapp.service -p ReadWritePaths -p ProtectSystem -p Type
# 5. Apply and watch
$ sudo systemctl daemon-reload
$ sudo systemctl restart myapp && journalctl -u myapp -f
systemd-analyze verify is the gate — it reports unknown directives and syntax errors. An invented directive is silently ignored by systemd at runtime, so without this check a "fix" can appear to work while changing nothing.
systemctl show proves the effect — it prints what systemd actually parsed, which is the only proof the change took effect.
What Models Get Wrong Here
Invented directives — plausible names that do not exist. Because systemd ignores unknown directives rather than erroring at runtime, this produces a fix that changes nothing while looking correct.
Wrong section — putting an [Install] key in [Service], or a service directive in [Unit]. Verify catches most of these.
Deprecated syntax — advice from older systemd versions, such as older network target names or superseded resource-control options.
Overreaching fixes — suggesting you remove ProtectSystem=strict when the correct fix is adding one path to ReadWritePaths. Prefer the narrowest change that works.
Ignoring SELinux and AppArmor — a permission denial may not be systemd at all. Check ausearch -m avc -ts recent on RHEL-family systems before rewriting the unit.
Missing the drop-in — if you did not include systemctl cat output, the model reasons about a file that is being overridden.
Common Failures and Their Real Causes
status=203/EXEC — the binary in ExecStart= does not exist, is not executable, or has a bad interpreter line. Test with sudo -u <user> /full/path --version.
status=200/CHDIR — WorkingDirectory= does not exist or is not accessible to the service user.
status=217/USER — the User= account does not exist. Create it, or fix the typo.
Failed at step NAMESPACE — a sandboxing directive cannot be satisfied — commonly ProtectHome or ProtectSystem conflicting with a path the service needs.
Service starts then immediately exits — usually a Type= mismatch, or the process daemonizes when systemd expected it to stay in the foreground.
Works manually, fails as a service — environment. systemd provides a minimal environment with no shell profile — the same class of problem as with cron, covered in our cron guide.
Start request repeated too quickly — the service is crash-looping. The real error is above the rate-limit message; raise RestartSec= only after fixing the crash.
For the environment class of failure, the parallel with cron jobs is exact: absolute paths and explicitly declared environment variables solve most of it.
Frequently Asked Questions
What context should I give an LLM to debug a systemd unit?
The unit file via systemctl cat (so drop-in overrides are included), systemctl status output, the last 50–100 journalctl lines for that unit, and systemd-analyze verify output. With all of these a model usually identifies the cause; with only the error line it tends to guess.
Why do AI-suggested systemd fixes sometimes change nothing?
The suggested directive does not exist. systemd ignores unknown directives rather than failing loudly, so an invented option looks applied but has no effect. Always run systemd-analyze verify, and confirm with systemctl show -p that the value was actually parsed.
What does status=203/EXEC mean?
systemd could not execute the ExecStart binary. Usually the path is wrong, the file is not executable, or a script has a bad shebang. Test it directly as the service user with the full absolute path before changing anything else.
Why does my service work when I run it manually but fail under systemd?
systemd provides a minimal environment and does not read your shell profile, so PATH and other variables are absent. Use absolute paths in ExecStart and declare what the service needs with Environment= or EnvironmentFile=.
Should I disable sandboxing directives when they cause failures?
No — narrow them instead. If ProtectSystem=strict blocks a needed write, add that specific path to ReadWritePaths rather than removing the protection. Models often suggest the broad fix; prefer the smallest change that works.