D-state and phantom load average

Recently, I faced a critical incident involving the mounting of external backup volumes (NFS) on a production instance. The symptom was classic, but the root cause required a descent to the syscall level to avoid a complete cascading failure on the server.

The system showed a load average exceeding 50, even though global CPU usage was below 10%. While analyzing utilities like top and htop, I noticed several maintenance and backup processes (cpbackup and rsync) stuck in Uninterruptible Sleep (D state). The NFS mount point, intended for external backup storage, did not respond to a simple ls command, indefinitely hanging the terminal session of anyone trying to access it.

1. What is "d-state" and phantom load average?#

In Linux, processes marked with the D (Uninterruptible Sleep) state are in a deep sleep that ignores any attempt to signal the operating system. Unlike the S (Interruptible Sleep) state, which responds to immediate interruptions, a process in D-state is stuck in kernel space waiting for a system call (syscall) to return directly from the physical device driver or I/O bus layer.

Since processes in the uninterruptible state are technically ready to execute as soon as the hardware resource responds, the Linux scheduler includes them in the count of active tasks in the execution queue (runqueue). This generates the Phantom Load Average phenomenon: the load metric spikes to alarming values without any actual CPU or memory consumption.

2. The `intr` mount option is deprecated#

A common mistake in legacy systems administration runbooks is to suggest using the intr (interruptible) flag when mounting NFS to allow the kernel to interrupt hung operations.

The NFS dilemma: hard vs. soft mounts#

To prevent processes from being stuck permanently in D-state, you must correctly configure the mount timeout parameters:

hard Mount (Default): If the destination NFS server goes down, the NFS client retry RPC write and read operations infinitely. This robustly preserves file integrity and consistency, but causes processes to hang permanently in the D state if the remote storage does not respond.
soft Mount: If the remote server goes down, the client attempts to reestablish communication and, after reaching the timeout limit (timeo) and retry limit (retrans), the kernel returns an I/O error (EIO) directly to the application. This prevents processes from hanging forever, but it can lead to silent file corruptions if the client software does not know how to handle the write failure.

Fine-tuning NFS timeout parameters#

timeo: Defines the timeout limit for responses in tenths of a second (e.g., timeo=30 represents 3 seconds).
retrans: Defines the number of transmission attempts before the client reports an error or unavailability warning.

Recommended mount commands for modern kernels#

# SOFT OPTION: Recommended for batch backups and logs (non-critical, prevents permanent I/O hangs)
mount -t nfs -o soft,timeo=30,retrans=3,rsize=32768,wsize=32768,nosuid IP_STORAGE:/export/backup /mnt/meu_backup

# CONTROLLED HARD OPTION: Recommended for critical database or transactional files (guarantees data integrity with a longer timeout)
mount -t nfs -o hard,timeo=600,retrans=2,rsize=32768,wsize=32768,nosuid IP_STORAGE:/export/backup /mnt/meu_backup

3. Backup protocol before unmounting with `umount -l`#

When an NFS mount point freezes, the immediate reaction is usually to force unmounting (umount -f) or perform a lazy unmount (umount -l).

Before forcing the unmount, perform the following containment and backup steps:

Identify stuck processes accessing the mount:

   # List open files and descriptors associated with the NFS mount point
   lsof +f -- /mnt/backup_ovh

Backup critical files accessible in memory or local cache:

   # Try to pack critical files from the directory before the I/O reset
   tar czf /root/pre-nfs-fix-backup.tar.gz /mnt/backup_ovh/critical-files/ 2>/dev/null

   # Or run an incremental synchronization with a rigid network timeout
   rsync -av --timeout=10 /mnt/backup_ovh/ /root/nfs-backup/ 2>/dev/null

4. Investigating the kernel and remote NFS server#

If system calls are hanging, begin by inspecting the kernel mount files and validate communication with the stat syscall:

# Inspect mounts as recorded by the kernel
cat /proc/mounts

# Trace the stat() syscall and check for the timeout error (ETIMEDOUT)
strace -e trace=stat ls /mnt/backup_ovh

Remote NFS server health check#

Ensure the destination remote storage is healthy and accepting connections:

# 1. Test basic ICMP connectivity and latency
ping -c 3 IP_STORAGE

# 2. Consult the remote server's portmapper
rpcinfo -p IP_STORAGE

# 3. List active exports and authorized folders on the NFS server
showmount -e IP_STORAGE

# 4. Validate TCP communication ports using Netcat
nc -zv IP_STORAGE 2049   # Native NFS port
nc -zv IP_STORAGE 111    # Portmapper port (RPC bind)

5. Network diagnostics, MTU, and fragmentation#

High RPC retransmission rates indicate that large read and write packets (rsize/wsize) may be exceeding the MTU capacity of the network route, resulting in IP packet fragmentations that are dropped by intermediate firewalls.

IP packet fragmentation test#

To validate the maximum packet size that passes without fragmentation through the active network route:

# Send a 1472-byte ICMP packet (1500-byte payload with header) forcing the Don't Fragment (DF) bit
ping -M do -s 1472 IP_STORAGE

Route tracing and active sockets#

# Analyze route and possible latency bottlenecks to storage
traceroute IP_STORAGE

# Check the state of TCP sockets associated with NFS communication
ss -tnp | grep -E "2049|111"

6. Verifying NFS protocol versions#

Ensure that client and server are talking on the same protocol version (NFSv3, NFSv4, NFSv4.1) to avoid unexpected behaviors in file locks.

# List compiled and supported NFS protocol versions in the Linux client
cat /proc/fs/nfsfs/versions

# Check the active version statistics on the NFS server
nfsstat -s | head -5

# Check under which version and options the active NFS is mounted
mount | grep nfs

7. Relationship with PHP-FPM and i/o stall#

In web environments, NFS inconsistency propagates rapidly to the application. If a PHP script performs log persistence or backup dumps via file_put_contents() or fwrite(), the write() syscall gets blocked in the kernel if the write buffers are full.

Without the soft timeout option, PHP-FPM workers get stuck waiting for the syscall to finish and enter D-state, quickly exhausting the max_children directive. The web server then starts returning 502 Bad Gateway errors to all legitimate users because the interpreter is out of free resources.

8. Checking other NFS mounts and SELinux status#

Audit if other network mounts on the server have been contaminated by cascading I/O stalls:

# List all active remote mounts on the server
mount | grep nfs

# Check the disk space response of each mounted point
df -h | grep nfs

Checking permissions with SELinux#

SELinux can block communication channels with remote storage if the required policies are not active:

# Check SELinux policing status
getenforce

# Search for NFS-related AVC denials in system logs
sudo ausearch -m avc -ts recent | grep -i "nfs"

# View security contexts of folders on the mount point
ls -Z /mnt/meu_backup/

9. Mount persistence and fstab configuration#

To ensure that mounting is automatic and does not hang system boot during temporary network unavailability, add the entry in /etc/fstab with the special _netdev flag. This instructs the operating system to wait for the network stack to initialize completely before attempting to attach the NFS volume:

# Inject the optimized and network-resilient persistent entry
echo "IP_STORAGE:/export/backup /mnt/meu_backup nfs soft,timeo=30,retrans=3,rsize=32768,wsize=32768,nosuid,_netdev 0 0" | sudo tee -a /etc/fstab

To validate syntax and mounting without restarting the server:

# Execute batch mounting from the edited fstab file
sudo mount -a

10. Post-fix verification and testing#

After restoring the mount with optimized options, validate the stability of the mount point and server metrics:

# 1. Validate physical reading of the remote directory
ls -la /mnt/meu_backup/

# 2. Run touch and immediate delete test in storage
touch /mnt/meu_backup/test-file && rm /mnt/meu_backup/test-file

# 3. Ensure load average has returned to normal parameters
uptime

# 4. Check if there are still processes stuck in D-state in the system
ps aux | awk '{if ($8 ~ /D/) print}'

# 5. Analyze RPC retransmission statistics of the NFS client
nfsstat -rc

Checklist: NFS freeze troubleshooting#

1. Diagnosis and triage#

[ ] Check phantom load average: uptime
[ ] List all active processes stuck in D-state: ps aux | awk '{if ($8=="D") print}'
[ ] Trace file syscall hangs: strace -e trace=stat ls /mnt/backup_ovh
[ ] Inspect NFS pending connection stats: nfsstat -rc

2. NFS server triage#

[ ] Test ICMP response (ping) of the storage server: ping IP_STORAGE
[ ] Validate active services via remote portmapper: rpcinfo -p IP_STORAGE
[ ] List valid NFS export points on the storage: showmount -e IP_STORAGE
[ ] Check TCP ports 2049 and 111 with Netcat: nc -zv IP_STORAGE 2049

3. Backup and safe unmounting procedure#

[ ] Identify processes with open files on the frozen point: lsof +f -- /mnt/backup_ovh
[ ] Run backup of files that still respond in local cache via tar or rsync with timeout.
[ ] Perform lazy and safe unmounting of references: umount -l /mnt/backup_ovh
[ ] If residual processes exist, apply gentle SIGKILL with PID verification.

4. Network and protocol diagnosis#

[ ] Test MTU and IP packet fragmentation: ping -M do -s 1472 IP_STORAGE
[ ] Trace network hops: traceroute IP_STORAGE
[ ] Check active NFS TCP connection stats: ss -tnp | grep -E "2049|111"
[ ] Validate supported protocol versions: cat /proc/fs/nfsfs/versions

5. Resilient correction and mount#

[ ] MANDATORY: Remove references to the obsolete/deprecated intr parameter from the mount script.
[ ] Set resilient soft mount with fine timeout control (timeo=30, retrans=3).
[ ] Adjust read/write buffer size (rsize=32768,wsize=32768) to mitigate fragmentation.
[ ] Test manual mount with new directives.

6. System persistence and security#

[ ] Add resilient entry in /etc/fstab containing the _netdev flag.
[ ] Test mount table file consistency: sudo mount -a
[ ] Verify if SELinux is in Enforcement (getenforce) and search for NFS blocks in ausearch.
[ ] Validate security context of mounted folders (ls -Z).

7. Post-fix verification#

[ ] Verify load average has decreased and stabilized.
[ ] Validate file creation and deletion in the NFS directory.
[ ] Confirm PHP-FPM interpreter does not have workers in D-state.
[ ] Monitor nfsstat to guarantee zero RPC retransmission rates.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

This post is licensed under CC BY-NC.