Under the hood of Ansible: from static inventories to SSH multiplexing and privilege escalation

Managing infrastructure as code seems straightforward in introductory tutorials: list a few IP addresses in a text file, execute a command, and automation takes care of the rest.

However, once you scale out to real-world infrastructure across heterogeneous networks, low-level operating system and networking behaviors surface quickly.

Here is an architectural deep dive into Ansible's inventory engine, OpenSSH multiplexing mechanics in the kernel, privilege escalation (become) edge cases, and practical connection troubleshooting.

1. Inventory architecture: beyond plain text files#

The gateway to Ansible execution is the Inventory. While YAML is common, INI remains popular for its brevity and rapid inspection:

[webservers]
web01.yourdomain.com ansible_host=192.168.10.11 ansible_user=deploy
web02.yourdomain.com

[dbservers]
db01.yourdomain.com

[production:children]
webservers
dbservers

What happens during initialization?#

When you run ansible-playbook, the engine executes three discrete phases before transmitting payloads:

1. Parsing: Reads the inventory source and resolves nested group relationships (children).
2. Host Pattern Matching: Evaluates target patterns specified in the playbook against active memory structures.
3. Variable Merging: Merges inline definitions with variables defined in group_vars/ and host_vars/.

In dynamic cloud environments, static files quickly become stale. The standard solution is a Dynamic Inventory (an executable script or plugin). Ansible invokes the executable with --list, parses the JSON output from STDOUT directly into memory, and evaluates targets dynamically.

2. Connection mechanics: SSH multiplexing and syscalls#

Because Ansible is agentless, communication overhead relies entirely on the SSH connection plugin.

Without ControlPersist / ControlMaster configured on your control node, every single task in a playbook initiates a fresh TCP session, executing a 3-way handshake and re-negotiating Diffie-Hellman keys. At the kernel level, this triggers hundreds of redundant socket() and connect() syscalls.

[Control Node] ──(Persistent Socket in ~/.ansible/cp/)──► [Target Host]
       │
       ├─► 1. Transfers compressed Python module via SFTP to /tmp/ansible_remote_tmp
       ├─► 2. Runs chmod +x on the temporary script
       └─► 3. Executes via execve() invoking /usr/bin/python3

With SSH multiplexing enabled:

• Ansible reuses persistent control sockets in ~/.ansible/cp/.
• Payloads (compressed Python modules) are transferred via SFTP directly to the directory defined in ansible_remote_tmp.
• The remote node marks the payload executable (chmod +x) and invokes execve() against the Python interpreter to execute the task.

3. Systematic connection troubleshooting#

When newly provisioned nodes fail to respond, follow a structured triage workflow:

# 1. Test SSH handshake and Python execution with maximum verbosity
ansible webservers -m ping -vvv

# 2. Inspect merged host variables loaded into memory
ansible-inventory --host web01.yourdomain.com

Common root causes:#

• Permission Denied (Publickey): Having the public key in place is not enough. The SSH daemon silently rejects connections if filesystem permissions are overly permissive. Ensure 700 on ~/.ssh and 600 on authorized_keys.
• Missing Python Interpreter: On modern Linux distributions, /usr/bin/python is absent by default, replaced by /usr/bin/python3. Explicitly configure the interpreter path:

ansible_python_interpreter=/usr/bin/python3

• Tunnel MTU/MSS Mismatches: When managing hosts over VPNs or GRE tunnels, large packets carrying serialized Python module code can suffer silent packet drops. Inspect interface MTU settings with ip link.

4. Edge cases: host key checking and escalation (become)#

Two common operational issues in large server fleets:

A. Interactive fingerprint prompts#

When executing playbooks against newly spawned instances, tasks can stall waiting for interactive SSH host key confirmation.

• Workaround: Setting host_key_checking = False in ansible.cfg.
• Security Best Practice: Rather than disabling host key verification globally (which exposes sessions to Man-in-the-Middle attacks), pre-populate known_hosts on the control node:

ssh-keyscan -H 192.168.10.11 >> ~/.ssh/known_hosts

B. Sudo escalation failures (requiretty)#

Connecting as an unprivileged automation user and escalating with become: yes executes a command similar to:

sudo -H -S -p "[sudo password prompt] " -u root /bin/sh -c '...'

If Defaults requiretty is enabled inside /etc/sudoers on the target host, execution fails immediately. By default, Ansible does not allocate a pseudo-terminal (TTY) to minimize I/O overhead. Resolving this requires adjusting /etc/sudoers to allow passwordless commands without an attached TTY for the automation user.

Best practices for reliable Ansible operations#

Reliable automation depends on aligning SSH trust chains, network MTU integrity, explicit Python interpreter paths, and proper sudoers privilege escalation rules. Mastering these operating system behaviors ensures predictable execution across any fleet size.

See also

DevOps Refactoring Ansible playbooks: from imperative shell scripts to declarative idempotence

DevOps Automating user access management at scale: from Bash and cPanel WHM to Ansible orchestration

DevOps From batch SSH to UI automation: managing multi-vps fleets with Ansible Semaphore

This post is licensed under CC BY-NC.