Mitigating MariaDB connection exhaustion under Layer 7 attacks on CloudLinux and cPanel

When hosting multiple websites on a shared cPanel server, performance degradation often follows familiar patterns. However, when a multi-tenant environment on CloudLinux faces Layer 7 application attacks, interactions between Apache, PHP-FPM, the Linux Kernel, and MariaDB trigger cascading failures that require surgical containment.

I recently handled a critical MariaDB resource exhaustion incident. The lockup blocked standard CLI tools and threatened overall host stability. In this article, I document low-level operating system mechanics, affected binaries, and practical containment steps applied without restarting the database.

1. The symptom: saturated thread pool and the sending data false positive#

The incident started with monitoring alerts indicating that MariaDB had exhausted available worker threads. Checking the processlist revealed this state:

+----------+--------------+-----------+--------------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+
| Id       | User         | Host      | db           | Command | Time  | State        | Info                                                                                                 |
+----------+--------------+-----------+--------------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+
| 59634010 | user_dbuser  | localhost | user_chatdb  | Query   | 124   | Sending data | SELECT COUNT(*) AS `total`,(SELECT COUNT(*) FROM (SELECT `lz_chat_archive`.`chat_id` FROM ...        |
| 59649250 | user_dbuser  | localhost | user_chatdb  | Query   | 3     | Sending data | SELECT COUNT(*) AS `total`,(SELECT COUNT(*) FROM (SELECT `lz_tickets`.`id` FROM `lz_tickets`) AS ... |
+----------+--------------+-----------+--------------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+

What the sending data state represents#

At first glance, "Sending data" appears to indicate network latency, as though outgoing TCP packets were buffering. However, inside MariaDB and MySQL engine code (specifically sql_select.cc), this status encompasses the entire query execution lifecycle, disk reads, and result transmission.

Because incoming queries contained nested subqueries (SELECT ... FROM (SELECT ...)), the InnoDB storage engine was forced to build temporary tables.

MariaDB first attempts building temporary tables in RAM via the MEMORY engine. When datasets exceed tmp_table_size or max_heap_table_size, the engine spills data onto disk (using Aria or InnoDB temporary tablespace). This created heavy disk I/O saturation, trapping database worker threads in read() and pread64() kernel system calls.

2. Server lockout: TLS handshake failure and error 1040#

My initial reaction was attempting to clear long-running queries via CLI:

mysql -e "SELECT concat('KILL ', id, ';') FROM information_schema.processlist WHERE user = 'user_dbuser' AND time > 100" | grep -v "concat" | mysql

The command failed immediately:

ERROR 2002 (HY000): Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections

Connection architecture and pre-TLS rejection#

The mariadbd daemon runs a main loop listening on port 3306 with accept() and epoll() system calls. When a connection request arrives, the daemon inspects the global max_connections counter before delegating the session to OpenSSL.

Because all connection slots were occupied by threads waiting on disk I/O, the daemon rejected incoming TCP requests with a raw error packet. The mysql CLI client, expecting a TLS Server Hello handshake message, failed to negotiate encryption and printed the error above.

3. Network tracing: Unix socket vs. TCP/IP#

To trace where traffic originated, I inspected the network stack:

ss -atpn | grep :3306
# Output:
# LISTEN     0      80         0.0.0.0:3306      0.0.0.0:*      users:(("mariadbd",pid=3312899,fd=20))
# TIME-WAIT  0      0        127.0.0.1:2077    127.0.0.1:33064

lsof -i :3306
# Output:
# COMMAND      PID  USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
# mariadbd 3312899 mysql   20u  IPv4 497374505      0t0  TCP *:mysql (LISTEN)

The ss utility showed only the listening descriptor and residual loopback connections (127.0.0.1). There was no direct external traffic hitting port 3306 from the internet.

Traffic was arriving on HTTP/HTTPS ports (80/443). Requests reached the web server, which forwarded them to PHP worker processes (lsphp on CloudLinux). PHP acted as a local proxy, opening connections against /var/lib/mysql/mysql.sock or loopback, masking traffic under the localhost identifier.

4. Countermeasures: isolation and containment strategies#

Raising global max_connections or restarting MariaDB during peak hours would disrupt other hosted tenants and inflate memory usage. I opted for isolated containment.

A. Restricting the user within MariaDB#

MySQL and MariaDB reserve an extra connection slot for administrative accounts with SUPER privileges. Connecting directly through the local Unix socket:

mysql -u root --socket=/var/lib/mysql/mysql.sock -e "ALTER USER 'user_dbuser'@'localhost' WITH MAX_USER_CONNECTIONS 30; FLUSH PRIVILEGES;"

This updated the mysql.user grant table. When PHP scripts attempted opening a 31st concurrent connection, mariadbd gracefully rejected attempts with error 1226 (42000): User has exceeded the 'max_user_connections', protecting global capacity.

B. Confining web workers via LVE cgroups#

To stop incoming PHP requests before they even opened database sockets, I updated Kernel cgroups via CloudLinux LVE:

lvectl set user --maxEntryProcs=15 --nproc=50
lvectl apply all
pkill -u user

The maxEntryProcs parameter caps concurrent web requests admitted into the user container. Lowering it to 15 caused surplus web requests to receive an immediate HTTP 508 Resource Limit Is Reached response, shielding host CPU. The pkill command cleaned lingering orphaned workers.

5. Identifying the attacker: POSIX log parsing#

With database load contained, I needed to locate and block the attacking IP on the firewall (CSF). When analyzing Apache domlogs with tail, a common globbing issue occurred:

tail -5000 /usr/local/apache/domlogs/user/* | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10
# Output:
# tail: option used in invalid context -- 5

The wildcard expanded across numerous files and subdirectories. GNU tail lost positional context with the -5000 flag.

The clean solution used cat with standard error redirection to suppress directory warnings:

cat /usr/local/apache/domlogs/user/* 2>/dev/null | awk '{print $1, $7}' | sort | uniq -c | sort -nr | head -n 20

This immediately isolated the offending IP address and target endpoint (/chat/server.php). I blocked the IP on CSF:

csf -d 198.51.100.45 "Layer 7 flood against chat endpoint"

6. Alternative hypotheses and edge cases#

In production, I avoid relying on a single working assumption. During triage, two alternative hypotheses were examined:

1. Persistent database connections (PConnect):

If PHP enabled persistent connections (PDO::ATTR_PERSISTENT), workers could have answered HTTP requests while keeping database file descriptors idle in Sleep state. I ruled this out because active threads showed ongoing execution with Sending data.

1. Missing table indexes:

If tables like lz_chat_archive grow large without indexes on chat_id, count queries trigger full table scans. To verify schema status:

EXPLAIN SELECT COUNT(*) FROM (SELECT `lz_chat_archive`.`chat_id` FROM `lz_chat_archive`) AS tmp;

When type reports ALL with high row counts, schema indexing must be remediated alongside connection limits.

7. Incident response toolkit#

Here is the compiled command reference in logical triage sequence:

Local Unix socket rescue connection#

mysql -u root --socket=/var/lib/mysql/mysql.sock -e "SHOW PROCESSLIST;"

Emergency GDB live memory injection#

If root connections lock out entirely, raise max_connections directly in process memory without downtime:

gdb -p $(pgrep -x mariadbd) -ex "set max_connections=2000" -ex "detach" -ex "quit"

Terminate loopback socket threads#

tcpkill -i lo port 3306

Restrict CloudLinux tenant limits#

lvectl set user --maxEntryProcs=15 --nproc=50
lvectl apply all
pkill -u user

Parse Apache domlogs for top attacking IPs#

cat /usr/local/apache/domlogs/user/* 2>/dev/null | awk '{print $1, $7}' | sort | uniq -c | sort -nr | head -n 20

Key takeaways from incident containment#

Managing production emergencies requires looking across architectural tiers. Database timeouts rarely stem only from database configuration; they frequently reflect application-layer floods overwhelming web workers.

Isolating accounts in the Linux kernel via LVE, bypassing networking using local Unix sockets, and dropping offending IPs at the firewall resolves multi-tenant crises while preserving uptime for unaffected websites.

See also

Databases Mastering MySQL connection limits on CloudLinux and cPanel: a real incident post-mortem

Databases Surviving MariaDB connection exhaustion on CloudLinux with zero downtime

Infrastructure MySQL handshake desynchronization in multi-tenant environments: cPanel, CloudLinux, and CageFS

This post is licensed under CC BY-NC.