Anatomy of a silent deadlock in CloudLinux and cPanel: phantom processes, session locks, and TCP queues

Anyone managing shared Linux servers with cPanel and CloudLinux has likely encountered a puzzling scenario: a website loads indefinitely in the browser before failing with ERR_CONNECTION_TIMED_OUT, yet checking the account's CloudLinux limits (lveinfo) shows the user completely idle, with zero CPU faults and ample free memory.

This type of silent deadlock often misleads operators because standard resource monitors show no sign of exhaustion. The root cause is not insufficient server capacity, but internal execution conflicts: mismatched PHP handlers, exclusive session file locks, or saturated TCP backlog queues in the Linux kernel.

In this guide, we break down a real incident of this kind, exploring how to audit the process table, inspect Unix sockets, trace syscalls with strace, and recover the environment via the command line.

1. Timeout symptoms and process table inspection#

The issue surfaced when requests to a specific domain (domain.com) hung indefinitely. The first step on the server console was listing the processes running under that cPanel user (user):

ps aux | grep user

The output revealed the following process distribution:

user  823582  0.2  1.9 244988 230716 ?        R    00:26   1:42 spamd child
user  900307  0.0  0.0  49232 11068 ?        S    09:12   0:00 dovecot/imap
user  901292  0.0  0.0  49544 11524 ?        S    09:15   0:00 dovecot/imap
user  902486  0.6  1.6 211880 197416 ?       S    09:19   0:15 spamd child
user  913853  0.2  0.2  98260 24748 ?        S    09:57   0:00 php-fpm: pool user_user
user  913949  0.0  0.0   5908  3700 ?        S    09:57   0:00 /usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=node.domain.com --suffix=-bytes_log
user  914085  0.1  0.1  59640 12832 ?        R    09:57   0:00 /opt/cpanel/ea-php83/root/usr/bin/php-cgi
user  914126  0.0  0.0  23820  8600 ?        R    09:57   0:00 /opt/cpanel/ea-php83/root/usr/bin/php-cgi
user  914127  0.5  0.0  30892 10228 ?        R    09:57   0:00 /opt/cpanel/ea-php83/root/usr/bin/php-cgi
user  914138  0.0  0.0  23688  8448 ?        S    09:57   0:00 /opt/cpanel/ea-php83/root/usr/bin/php-cgi

What this output indicates#

• Processes in state R (Running): PIDs 914085, 914126, and 914127 running /opt/cpanel/ea-php83/root/usr/bin/php-cgi were marked R. This indicates they were sitting in the kernel scheduler queue, actively computing instructions or waiting for user-space I/O.
• Handler conflict: the user already had an active PHP-FPM worker pool (PID 913853). The simultaneous existence of standalone php-cgi processes proves that inbound requests were being forked as legacy CGI rather than passed via FastCGI over the FPM Unix socket.

2. Investigating CloudLinux LVE resource limits#

On CloudLinux servers, the default assumption during web slowdowns is that the user hit their Entry Processes (EP) or concurrent process limits (NPROC). When that occurs, the kmod-lve kernel module delays new connections or rejects fork() calls.

To check the user's resource usage over the past five minutes:

lveinfo --user user --period=5m

The output showed:

+-----------+-----------+----+----+----+---+---+---+-----+-----+-----+-----+---+-----+-----+-----+------+------+------+-----+------+------+------+-----+-----+-----+-----+
|From       |To         |aCPU|mCPU|lCPU|aEP|mEP|lEP|aVMem|mVMem|lVMem|VMemF|EPf|aPMem|mPMem|lPMem|aNproc|mNproc|lNproc|PMemF|NprocF|aIO   |mIO   |lIO  |aIOPS|mIOPS|lIOPS|
+-----------+-----------+----+----+----+---+---+---+-----+-----+-----+-----+---+-----+-----+-----+------+------+------+-----+------+------+------+-----+-----+-----+-----+
|04-23 09:58|04-23 09:59|2   |2   |100 |0  |0  |20 |0B   |0B   |0B   |0    |0  |1.2MB|1.2MB|1.0GB|0     |0     |100   |0    |0     |148KB |1.0MB |1.0MB|1    |1    |1.0K |
+-----------+-----------+----+----+----+---+---+---+-----+-----+-----+-----+---+-----+-----+-----+------+------+------+-----+------+------+------+-----+-----+-----+-----+

Why a clean lveinfo output can be deceptive#

• CPU usage: the limit was 100% (1 CPU core), but peak usage hovered at 2% (mCPU=2).
• Entry processes: the ceiling was 20 concurrent connections; the observed value was zero (mEP=0) with zero faults (EPf=0).
• Memory footprint: memory ceiling was 1 GB; actual consumption was just 1.2 MB.

Because all fault counters (EPf, PMemF, NprocF) remained at zero, CloudLinux was not throttling the account. The deadlock lived entirely within the application runtime and web server stack.

3. Auditing the PHP-FPM unix socket#

To verify whether the PHP-FPM service was hanging or rejecting IPC connections, we inspected the local socket with ss:

ss -lnxp | grep "user"

The result returned:

u_str LISTEN 0      4096   /var/cpanel/php-fpm/user/sock 5587329   * 0   users:(("cpanel_php_fpm",pid=914882,fd=10),("cpanel_php_fpm",pid=914576,fd=5),("cpanel_php_fpm",pid=808640,fd=8))

• LISTEN state and queue depths: Recv-Q was at 0 while the maximum backlog was 4096. This confirmed the PHP-FPM daemon was completely responsive and waiting for work.
• Diagnosis: the FPM daemon was healthy, but Apache's request router was launching orphaned php-cgi workers due to an overridden handler directive.

4. Syscall tracing with strace#

Because the php-cgi workers remained in state R without responding and without significant CPU consumption, we needed to know what system calls they were waiting on.

We attached strace to the busiest process (PID 914127):

strace -ff -y -tt -T -s 512 -p 914127

This highlighted two common deadlock scenarios:

Scenario a: session file lock contention#

PHP's default session handler locks user session files on disk using flock() or fcntl(). If one browser tab starts a long network operation and the user opens a second tab, that subsequent request stalls waiting for the lock to release:

09:58:02.102345 flock(8, LOCK_EX) = ? ERESTARTSYS (To be restarted if system call is interrupted)
09:58:05.405912 flock(8, LOCK_EX <waiting...>)

To verify which file descriptor was blocked:

lsof -p 914127 | grep "sess_"

If the descriptor points to /var/cpanel/php/sessions/ea-php83/sess_[ID], the application is holding the session file open instead of calling session_write_close() right after reading session data.

Scenario b: timeouts in external network queries#

Another common stall happens when PHP scripts (such as WordPress plugins making outbound cURL calls to verify licenses or query APIs) hang on unresponsive DNS resolvers:

09:58:02.200110 connect(9, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
09:58:02.200450 poll([{fd=9, events=POLLOUT}], 1, 5000) = 0 (Timeout)

The process remains trapped in poll() waiting for external data until network timeouts expire.

5. Transport layer timeout: cURL error 28#

When testing the site externally with curl:

curl https://domain.com

The client terminated after more than two minutes:

curl: (28) Failed to connect to domain.com port 443 after 134437 ms: Couldn't connect to server

The curl error 28 indicates CURLE_OPERATION_TIMEDOUT. Waiting 134 seconds without receiving an explicit TCP reset (RST) shows that incoming SYN packets were being silently dropped by the server's firewall or swallowed by an overflowing kernel backlog queue.

Saturated Apache accept queues#

When Apache workers hang waiting for deadlocked PHP scripts, the kernel's accept queue fills up.

To check TCP socket states across the system:

# Sockets stuck in SYN handshake
ss -nt state syn-recv

# Backlog depth on HTTPS port 443
ss -tlnp | grep :443

If the Recv-Q column on port 443 equals the configured limit in Send-Q (for instance, 511/511), Apache has stopped calling accept() and the Linux kernel silently drops incoming connections.

6. Recovery playbook#

To clear stalled workers, reset firewall states, and align PHP handlers back to PHP-FPM, execute these commands in sequence:

Step 1: Terminate stuck PHP processes#

Kill any lingering CGI and FPM processes for the affected user:

pkill -u user -9 php-cgi
pkill -u user -9 cpanel_php_fpm

Using -9 (SIGKILL) ensures the kernel terminates processes immediately, freeing file locks and descriptors.

Step 2: Flush and rebuild the firewall (csf/iptables)#

If the server firewall rate-limited or temporarily dropped repeated connection retries:

csf -f && csf -r

This clears iptables chains and rebuilds CSF rules cleanly.

Step 3: Rebuild PHP-FPM user configuration and restart daemons#

To fix misaligned handler directives causing Apache to spawn php-cgi:

# Rebuild vhost and user pool configurations
/usr/local/cpanel/scripts/php-fpm-config --rebuild --user=user

# Restart the PHP-FPM manager and Apache
/scripts/restartsrv_cpanel_php_fpm
/scripts/restartsrv_httpd

Operational habits to prevent silent deadlocks#

1. Standardize PHP handlers: ensure all tenant accounts use a unified handler such as PHP-FPM or lsapi. Mixed legacy CGI directives cause unpredictable process forks that evade pool limits.
2. Release session locks promptly: any script performing heavy computation or external API calls should call session_write_close() as soon as initial session data is read.
3. Inspect kernel socket queues (Recv-Q): when websites time out despite near-zero CPU usage, check port 443 with ss -tlnp. A full receive queue reveals that the web server has stalled and cannot accept new connections from the kernel.

See also

Servers Low-level debugging in CloudLinux: syscalls, LVE limits, and the trap of unlimited defaults

Servers Advanced HTTP 500 troubleshooting in cPanel: from multi-tier log tracing to PHP runtime compatibility

Infrastructure Session termination anatomy: debugging LiveZilla, PHP, and Linux sockets

This post is licensed under CC BY-NC.