Ask a model for "an Ansible playbook that installs nginx and opens the firewall" and you get valid YAML in seconds. It usually runs, too. The problems show up later: the second run reports eight changes when nothing changed, --check silently skips half the tasks, a file lands with mode 1204, or a module parameter that never existed fails only when the play reaches production.
The cause is predictable. Models learned Ansible from a decade of blog posts and Stack Overflow answers, most of them written for Ansible 2.x before collections existed, and they reach for shell because it always "works". This guide lists the failure modes you will actually see, then builds a verification ladder, from ansible-lint to a real idempotence test, that catches them before a host does. It is the Ansible counterpart to our checklist for reviewing AI-generated shell scripts.
The Failure Modes, Ranked by How Often They Bite
| Pattern in generated code | Why it is a problem | What to use instead |
|---|---|---|
shell: apt-get install -y nginx | Always reports changed, skipped in check mode, ignores package state | ansible.builtin.apt or ansible.builtin.package |
command: systemctl restart nginx | Restarts on every run, even when nothing changed | A handler notified by the task that changed the config |
mode: 644 | YAML reads 644 as a decimal integer, which is octal 1204 | mode: "0644" |
| Short module names (apt, copy) | Ambiguous once collections are installed; ansible-lint flags fqcn | ansible.builtin.apt, ansible.builtin.copy |
| Invented parameters | Unknown options fail at runtime, not at syntax check | Check every option with ansible-doc |
include: | Removed in ansible-core 2.16 | include_tasks or import_tasks |
apt_key | apt-key is deprecated on current Debian and Ubuntu | deb822_repository with signed_by, or a keyring file plus apt_repository |
state: latest | Every run can upgrade packages; no two hosts are guaranteed identical | state: present, with upgrades as a separate, deliberate play |
ignore_errors: true | Hides real failures so the play reports success | failed_when with the specific condition you expect |
| Secrets in plain YAML | Passwords and tokens end up in git and in -v output | ansible-vault plus no_log: true on the task |
| shell with pipes and no pipefail | The task succeeds when the first command in the pipe fails | A module, or set -o pipefail with executable: /bin/bash |
The common root: shell as a universal module — modules describe a desired state and report changed only when they act. shell and command just run something, so Ansible cannot know whether anything changed. Most of the other problems in the table follow from that.
Version drift is the second root — generated playbooks mix syntax from several Ansible eras. sudo: yes, include:, and bare with_items everywhere are strong hints that the model is pattern-matching on old material. Tell it your version up front.
Hallucinated parameters look plausible — a classic is ssh_key: on ansible.builtin.user. The module has generate_ssh_key and ssh_key_file, but installing an authorized key is a job for ansible.posix.authorized_key.
Before and After: A Typical Generated Playbook
This is representative of what a general-purpose model produces for a simple web server request. It runs cleanly the first time:
- hosts: all
become: yes
tasks:
- name: Install nginx
shell: apt-get update && apt-get install -y nginx
- name: Configure nginx
shell: echo "server_tokens off;" >> /etc/nginx/conf.d/security.conf
- name: Restart nginx
command: systemctl restart nginx
- name: Open firewall
shell: ufw allow 80
ignore_errors: yes
Run it twice and the second run appends a second server_tokens line, which makes nginx -t fail with a duplicate directive error, restarts nginx regardless, and reports every task as changed. Here is the same intent written with modules:
---
- name: Configure web servers
hosts: web
become: true
tasks:
- name: Install nginx
ansible.builtin.apt:
name: nginx
state: present
update_cache: true
cache_valid_time: 3600
- name: Deploy security settings
ansible.builtin.copy:
dest: /etc/nginx/conf.d/security.conf
content: "server_tokens off;\n"
owner: root
group: root
mode: "0644"
notify: Reload nginx
- name: Allow HTTP
community.general.ufw:
rule: allow
port: "80"
proto: tcp
- name: Ensure nginx is enabled and running
ansible.builtin.service:
name: nginx
state: started
enabled: true
handlers:
- name: Reload nginx
ansible.builtin.service:
name: nginx
state: reloaded
copy with content is idempotent — Ansible compares checksums and writes only when the file differs, so the second run reports ok instead of changed and the handler does not fire.
Validate configs before they go live — for a full nginx.conf managed with ansible.builtin.template, add validate: nginx -t -c %s. nginx resolves relative include paths against the directory of the file it is testing, so use absolute include paths in the template. For sshd, validate: /usr/sbin/sshd -t -f %s does the same job.
Collections are dependencies — community.general.ufw needs the community.general collection. Put it in collections/requirements.yml and install with ansible-galaxy collection install -r; do not assume the control node has it.
The Verification Ladder
Each rung catches a different class of bug, and each is cheaper than the next. Do not skip to running the play against real hosts.
# 1. Does it parse? (does NOT validate module arguments)
$ ansible-playbook site.yml --syntax-check
# 2. Does every module and option exist in YOUR installed version?
$ ansible-doc ansible.builtin.user | grep -A2 -i 'ssh_key'
$ ansible-galaxy collection list community.general
# 3. Lint against a strict profile
$ ansible-lint --profile production site.yml
site.yml:5: fqcn[action-core]: Use FQCN for builtin module actions (shell).
site.yml:5: command-instead-of-module: apt-get used in place of apt module
site.yml:5: no-changed-when: Commands should not change things if nothing needs doing.
site.yml:10: ignore-errors: Use failed_when and specify error conditions instead of using ignore_errors.
site.yml:2: yaml[truthy]: Truthy value should be one of [false, true]
# (abridged)
# 4. Dry run with diffs against one staging host
$ ansible-playbook -i inventory/staging site.yml --check --diff --limit web-stg-01
# 5. Real run, then run again: the second recap must show changed=0
$ ansible-playbook -i inventory/staging site.yml --limit web-stg-01
$ ansible-playbook -i inventory/staging site.yml --limit web-stg-01 | tail -3
PLAY RECAP *********************************************************
web-stg-01 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Syntax check is not validation — an invented parameter passes --syntax-check. It only fails when the module runs, with Unsupported parameters for (ansible.builtin.user) module: ssh_key. Check mode or a staging run is the first place you see it.
Check mode hides shell tasks — command and shell are skipped under --check unless they set creates or removes. A generated playbook full of shell can show a clean dry run that tells you nothing.
Profiles ratchet — ansible-lint ships profiles from min through basic, moderate, safety and shared to production. For generated code, start at production: the point is to be strict with code nobody wrote by hand.
The second run is the real test — idempotence is the property generated playbooks most often lack, and only a repeated run proves it. Anything still changed on run two is a bug or needs changed_when.
Ground the Model in Your Environment
Most of the failure modes above are avoidable if the model knows what it is targeting. Give it your exact versions, the collections you have, and the real documentation for the modules involved, then constrain the output:
$ ansible --version | head -1
ansible [core 2.16.3]
$ ansible-doc -s ansible.builtin.apt ansible.builtin.copy ansible.builtin.service > /tmp/docs.txt
$ ollama run qwen2.5-coder "Write an Ansible playbook.
TARGET: ansible-core 2.16, managed hosts are Ubuntu 24.04.
INSTALLED COLLECTIONS: ansible.posix, community.general.
TASK: install nginx, disable server_tokens, allow tcp/80 in ufw.
RULES:
- Use fully qualified module names (ansible.builtin.*).
- Use only parameters that appear in the module docs below. If you need
one that is not listed, say so instead of inventing it.
- No shell or command tasks. If one is unavoidable, explain why and add
creates/removes or changed_when.
- Every task must be idempotent: a second run must report changed=0.
- Restarts and reloads go in handlers, triggered by notify.
- Quote file modes as strings, e.g. mode: \"0644\".
- Use true/false, not yes/no. No ignore_errors. No state: latest.
- Secrets come from variables I will store in ansible-vault; never inline them.
MODULE DOCS:
$(cat /tmp/docs.txt)"
Docs in the prompt beat docs in the weights — ansible-doc -s prints a compact parameter list for exactly the version you have installed. It turns "remember the options" into "read the options", which models are far better at.
Ask for the reasoning separately — after the playbook, ask the model to list each task, why it is idempotent, and what it does in check mode. Gaps in that explanation point at tasks to read closely.
Inventories stay local — inventories carry hostnames, IPs and group structure. A locally hosted model lets you include them without sending your network layout to a third party.
Secrets, become and Blast Radius
Generated playbooks tend to be generous with privilege and careless with secrets, because the examples they learned from were demos. Review these by hand on every playbook, whatever the linter says.
# encrypt one value and paste the output into group_vars
$ ansible-vault encrypt_string 'S3cr3t-from-your-vault' --name 'db_password'
db_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
6231336436...
# keep the value out of logs, even at -vvv
- name: Create database user
community.postgresql.postgresql_user:
name: app
password: "{{ db_password }}"
become: true
become_user: postgres
no_log: true
Scope become to tasks — a play-level become: true is convenient, but it runs every task as root. Where a task only needs a service account, use become_user, as above, and keep root for tasks that need it.
Always limit the first run — generated plays often target hosts: all. Run with --limit on one host, and change the play to a named group before it is merged.
Watch for downloaded code — a get_url or shell: curl ... | bash without a checksum runs whatever that URL serves today. Pin a version and a checksum.
Hardening tasks need a human — SSH, sudoers and firewall tasks can lock you out of every host at once. Compare them against a known baseline, such as our server hardening checklist, before they run in parallel across a fleet.
Make Idempotence a CI Gate With Molecule
For roles you intend to keep, Molecule automates the whole ladder in a throwaway container: create, converge, run again to check idempotence, verify, destroy. Its idempotence step fails if the second converge reports any change, which is exactly the check generated roles most often fail.
$ pip install molecule "molecule-plugins[docker]" ansible-lint
$ cd roles/nginx
$ molecule init scenario default --driver-name docker
$ molecule test
# .github/workflows/ansible.yml
name: ansible
on: [pull_request]
jobs:
lint-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install ansible-core ansible-lint molecule "molecule-plugins[docker]"
- run: ansible-galaxy collection install -r collections/requirements.yml
- run: ansible-lint --profile production
- run: cd roles/nginx && molecule test
Containers are not full hosts — systemd, firewalls and kernel parameters behave differently in a container. Use an image built to run systemd, or test those roles on a VM, and keep the staging-host run for them.
Treat the model as a junior contributor — its pull request goes through the same lint, test and review as anyone else's. The pipeline does not need to know who wrote the YAML.
Write the runbook too — the playbook is half of the change. Our guide to AI-generated runbooks covers documenting how to run and roll it back.
Frequently Asked Questions
Can I trust an Ansible playbook written by ChatGPT or another LLM?
Treat it as a draft. Generated playbooks usually parse and often run, but they commonly overuse shell and command, invent module parameters, use syntax from older Ansible versions, and are not idempotent. Run ansible-lint, a check-mode dry run, and a repeated real run on a staging host before using one in production.
Why does ansible-playbook --syntax-check pass on a broken playbook?
Syntax check only parses YAML and play structure. It does not validate module arguments, so an invented parameter fails only when the task runs, with an Unsupported parameters error. Use ansible-doc to confirm each option exists, and a check-mode run to catch the rest.
How do I test whether a playbook is idempotent?
Run it twice against the same host. The PLAY RECAP of the second run should show changed=0. Any task that still reports changed is either not idempotent or needs a changed_when condition. Molecule automates this with its idempotence step.
Why is mode: 644 wrong in an Ansible task?
YAML parses an unquoted 644 as a decimal integer. Ansible passes it on as a number, which corresponds to octal 1204, not rw-r--r--. Always quote modes as strings with a leading zero, such as mode: "0644", or use symbolic modes like u=rw,g=r,o=r.
Which ansible-lint profile should I use for AI-generated code?
Start with the production profile, the strictest one. It flags missing fully qualified module names, command used instead of a module, missing changed_when, ignore_errors and unquoted modes, which are the most common problems in generated playbooks.