Refactoring Ansible playbooks: from imperative shell scripts to declarative idempotence

Anyone managing Linux server fleets knows that automating routine operations is essential. However, in the rush to get automation working quickly, it is easy to fall into the trap of writing shell scripts disguised as YAML.

While reviewing a legacy playbook used to update a monitoring agent and apply OS patches, I noticed several structural anti-patterns that made execution fragile, non-idempotent, and prone to silent failures.

Here is the breakdown of why imperative command tasks fail in production, and how to refactor them into a declarative, resilient, and idempotent Ansible workflow.

1. The legacy playbook#

The initial automation goal was simple:

1. Update all installed system packages.
2. Download the latest .deb package from an HTTP endpoint.
3. Install the package on the target host using dpkg.
4. Output execution logs to the console.

The original playbook was structured like this:

# ------------------------------------------------------------------------------
# Legacy Playbook: update_srv.yml
# ------------------------------------------------------------------------------

- name: Update server packages
  hosts: app_hosts
  become: yes

  tasks:
    - name: Upgrading packages
      package:
        name: "*"
        state: latest

    - name: Downloading and installing custom package
      command: wget "https://repo.yourdomain.com/repo/custom-agent.deb?nocache=$(date +%s)" -O /tmp/custom-agent.deb && sudo dpkg -i /tmp/custom-agent.deb
      register: os_release_output

    - name: Display command output
      debug:
        msg: "{{ os_release_output.stdout_lines }}"

Did it work? Yes. But it worked the wrong way.

2. Anatomy of the problems#

Dissecting the execution reveals four critical anti-patterns:

A. Loss of idempotency (dpkg -i vs libapt)#

Ansible is designed to be declarative. Invoking raw dpkg -i via the command module forces the task to return changed: true on every run, even when the identical package version is already installed. Raw dpkg does not inspect metadata against system state before unpacking files, creating unnecessary disk I/O and CPU overhead.

B. Dependency resolution failures#

The dpkg utility operates at a low level: if an updated package introduces a new system dependency (such as an updated libssl), dpkg aborts with broken dependency errors, leaving the system in an unconfigured state. Conversely, Ansible's apt module communicates with libapt-pkg to automatically resolve and install missing dependencies from configured repositories.

C. Unmanaged process lifecycle (orphan daemons)#

Updating a binary on disk does not restart the running process in memory. Without triggering a systemd daemon reload, the active service continues executing old instructions from RAM.

D. Shell interpolation & redundancies#

Using shell interpolation like $(date +%s) delegates execution to the remote host's shell. If server clocks drift or non-standard shells are active, commands fail unexpectedly. Furthermore, prepending sudo inside a play that already specifies become: yes is redundant and pollutes audit logs.

3. The solution: declarative refactoring#

To resolve these issues, the playbook was refactored using native Ansible modules (apt, get_url, file, and handlers):

---
# ------------------------------------------------------------------------------
# Refactored Playbook: update_srv.yml (Declarative & Idempotent)
# ------------------------------------------------------------------------------

- name: Infrastructure and Custom Agent Upgrade
  hosts: app_hosts
  become: yes
  gather_facts: yes
  vars:
    agent_repo_url: "https://repo.yourdomain.com/repo/custom-agent.deb"
    tmp_path: "/tmp/custom-agent.deb"

  tasks:
    - name: "[System] Update APT cache and upgrade packages"
      apt:
        upgrade: dist
        update_cache: yes
        cache_valid_time: 3600
      register: apt_upgrade_result

    - name: "[Agent] Download binary with TLS and timeout controls"
      get_url:
        url: "{{ agent_repo_url }}?nocache={{ ansible_date_time.epoch }}"
        dest: "{{ tmp_path }}"
        mode: '0644'
        force: yes
        timeout: 30
      register: download_status

    - name: "[Agent] Install via Native APT Module (Dependency Management)"
      apt:
        deb: "{{ tmp_path }}"
        state: present
      register: agent_install_result
      notify: "Restart Agent Service"

    - name: "[Cleanup] Remove temporary artifacts"
      file:
        path: "{{ tmp_path }}"
        state: absent

    - name: "[Debug] Upgrade Status"
      debug:
        msg: 
          - "System Upgrade Changed: {{ apt_upgrade_result.changed }}"
          - "Agent Status: {{ agent_install_result.stdout_lines | default('Already up-to-date') }}"

  handlers:
    - name: Restart Agent Service
      service:
        name: custom-agent
        state: restarted
        enabled: yes
      ignore_errors: yes

4. What changed under the hood?#

1. True Idempotency with apt: deb={{ tmp_path }}: The apt module checks the .deb package header against /var/lib/dpkg/status. If the version matches the installed state, Ansible marks ok: [host] without reinstalling.
2. Native Ansible Facts: Replaced fragile shell calls with {{ ansible_date_time.epoch }} collected directly during fact gathering.
3. Secure Transfers with get_url: Handles HTTP/HTTPS timeouts, sets file permissions (mode: 0644), and supports SHA-256 checksum validation.
4. Event-Driven Handlers: The custom-agent service restarts only when the installation task reports changed: true.
5. Filesystem Hygiene: A dedicated file task purges the temporary download from /tmp upon completion.

Low-level troubleshooting tips#

When executing playbooks across larger fleets, two common low-level issues can arise:

• APT Lock Contention: If unattended-upgrades or another process is running on the host, /var/lib/dpkg/lock-frontend will be locked. To inspect the active process:

lsof /var/lib/dpkg/lock-frontend

• Syscall Tracing with Strace: If execution hangs during node provisioning and you need to trace kernel syscalls:

strace -f -e trace=process,openat ansible-playbook update_srv.yml -vvv

Best practices for idempotent automation#

Automation is more than chaining bash commands inside YAML documents. Leveraging native Ansible modules ensures state enforcement, automatic error handling, true idempotency, and operational predictability across any fleet size.

See also

DevOps Under the hood of Ansible: from static inventories to SSH multiplexing and privilege escalation

DevOps Automating user access management at scale: from Bash and cPanel WHM to Ansible orchestration

DevOps From batch SSH to UI automation: managing multi-vps fleets with Ansible Semaphore

This post is licensed under CC BY-NC.