Mastering MySQL connection limits on CloudLinux and cPanel: a real incident post-mortem

When managing web servers with cPanel hosting multiple websites or clients, you eventually face the classic scenario: a single application starts spawning database connections uncontrollably and threatens to bring down the entire server.

I ran into this exact situation recently. A live chat application (LiveZilla) running inside a client account started monopolizing mysqld resources. The server slowed down and other sites began timing out waiting for database queries. The objective was straightforward: enforce a strict limit of 20 concurrent connections for that specific database user.

What seemed like a five-second command in the terminal turned into an in-depth debugging session across native MySQL host resolution, syntax quirks in the CloudLinux MySQL Governor wrapper, and Kernel-level process constraints in LVE.

Below is the complete post-mortem of what I ran, where syntax failed, the errors encountered, and the low-level operating system reasons behind each behavior.

Phase 1: native enforcement and the privilege resolution error in MySQL#

My initial instinct was to apply the constraint directly in the database engine. When limits are enforced within MySQL, the mysqld daemon rejects the handshake before allocating heavy memory buffers for the session.

Upon receiving an accept() syscall, MySQL checks active counters inside its internal function check_for_max_user_connections (located in sql/sql_connect.cc). If the configured threshold is reached, it drops the socket with an ER_TOO_MANY_USER_CONNECTIONS (ERROR 1203 (42000)) error as detailed in the official MySQL User Resource Limits documentation, preventing CPU mutex contention.

I attempted to apply the 20-connection cap using the standard query:

mysql -e "ALTER USER 'user_chatlivezilla'@'localhost' WITH MAX_USER_CONNECTIONS 20;"

The command failed immediately:

ERROR 1396 (HY000) at line 1: Operation ALTER USER failed for 'user_chatlivezilla'@'localhost'

Why error 1396 occurred#

Error code 1396 (HY000) (ER_CANNOT_USER) happens inside the MySQL privilege parser (specifically the mysql_edit_user function in C), as documented in the MySQL Server Error Reference. In MySQL, the primary key identifying any account is the pair User@Host.

If I attempt to modify 'user_chatlivezilla'@'localhost', but the account was registered with wildcard 'user_chatlivezilla'@'%' (common across cPanel setups) or tied to the server hostname, MySQL fails to locate the record and bails out with this generic error code.

To verify the exact registered account structure before running modifications:

mysql -e "SELECT User, Host, plugin FROM mysql.user WHERE User LIKE 'user_chat%';"

Rather than focusing solely on SQL grant definitions at that point, I decided to test the server's resource isolation layer first.

Phase 2: MySQL Governor and the dbctl utility behavior#

Because the native SQL route encountered a host mismatch, I switched to the CloudLinux resource management tier: MySQL Governor. The Governor monitors thread consumption and enforces restrictions based on LVE quotas.

My first attempts with the dbctl CLI ran into parameter syntax issues:

dbctl --set user --max_user_connections=20
# Output: Incorrect syntax: unknown command "--set"

dbctl -set user --max_user_connections=20
# Output: Incorrect syntax: unknown command "-set"

dbctl set user --max_user_connections=20
# Output: /usr/share/lve/dbgovernor/utils/dbctl_orig: unrecognized option '--max_user_connections=20'

The wrapper binary underneath#

The error message revealed that dbctl is a shell script wrapper invoking the compiled binary at /usr/share/lve/dbgovernor/utils/dbctl_orig.

According to the official CloudLinux MySQL Governor documentation, the dbctl set <user> [OPTIONS] subcommand strictly expects double-dash flags for resource allocation (--cpu, --read, --write, --slow).

When I tested positional syntax:

dbctl set user max_user_connections 20

Running this command produced no output and exited with status 0. However, this was a classic Unix placebo effect: the dbctl_orig binary simply discarded the unrecognized trailing positional arguments after the username. No connection restriction was actually saved.

How Governor handles connection caps via XML configuration#

MySQL Governor does not manage per-user connection ceilings through manual command-line flags. Its restriction model is reactive: whenever an account breaches CPU or I/O thresholds across short, mid, or long-term windows (1s, 5s, 1m, 5m), the db_governor daemon places the account into a restricted state (Levels 1 through 4).

This behavior is configured inside /etc/container/mysql-governor.xml within the <restrict> directive:

<restrict level1="60s" level2="15m" level3="1h" level4="1d" timeout="1h" log="/var/log/dbgovernor-restrict.log" user_max_connections="30"></restrict>

When throttling triggers, Governor temporarily updates the user in MySQL to the ceiling defined in user_max_connections="30". This explains why in Phase 4 all restricted accounts displayed exactly 30 connections.

Essential tools for monitoring and operating Governor in production:

• dbctl list-restricted: lists currently penalized accounts, breached metrics, and remaining cooldown.
• tail -f /var/log/dbgovernor-restrict.log: tracks live throttle events and resource triggers.
• dbtop: interactive top-like console displaying real-time database resource consumption with throttled users colored red.
• dbctl unrestrict <user>: lifts the restriction immediately without waiting for the timeout interval.

Phase 3: Kernel barrier with LVE and the NPROC vs EP conflict#

To build defense in depth and stop PHP scripts before they could even request TCP sockets against MySQL, I decided to constrain the cPanel account at the Kernel level using lvectl. The goal was to limit the total number of processes (NPROC) for that system user.

I executed:

lvectl set-user user --nproc=20

The Kernel rejected the configuration immediately:

error: You're trying to set invalid LVE limits. NPROC limit must be greater than EP + 15 limit, because number of processes and threads within LVE includes also Apache processes/threads, SSH sessions and etc, which enter into LVE. Limits of user with LVE ID "1001" are affected.

Understanding the Kernel safety rule#

This error message from lvectl reveals how the kmod-lve kernel module manages process limits, documented in the CloudLinux LVE Limits guide:

• EP (Entry Processes): Controls entry points into the container (HTTP requests handled by Apache or LiteSpeed).
• NPROC: Absolute process ceiling, combining child workers (PHP-FPM processes, cron jobs, SSH sessions, background threads).

LVE enforces in code that $NPROC > EP + 15$. This safeguard prevents aggressive limits from starving vital background processes and bricking the user account. Since the inherited plan had a high EP value, lowering NPROC to 20 violated the rule.

I adjusted both values to satisfy the formula:

lvectl set-user user --maxEntryProcs=5 --nproc=20

By capping web entry processes at 5, the application could not spin up dozens of concurrent web workers, creating an effective bottleneck before queries ever touched MySQL.

Diagnostic checks and LVE quota compliance#

To confirm that other cPanel accounts or hosting packages do not carry ratio violations that could lead to Apache or LiteSpeed worker starvation, run the official CloudLinux diagnostic tool:

# Validate LVE limits compliance across all users and packages
cldiag --check-lve-limits

# Inspect historical entry processes and process bursts
lveinfo --period=1d --by-usage=ep,nproc

If cldiag reports non-compliant packages, update the base package via lvectl set-pkg <package> --nproc=100 --save to ensure newly provisioned accounts maintain proper Kernel proportions.

Phase 4: the mystery of 59 persistent connections#

With Governor and LVE limits applied, I checked the database to inspect active host mappings:

mysql -e "SELECT User, Host, max_user_connections FROM mysql.user WHERE User LIKE 'user%';"

The table returned:

+--------------+--------------------+----------------------+
| User         | Host               | max_user_connections |
+--------------+--------------------+----------------------+
| user         | 192.0.2.100        |                   30 |
| user         | localhost          |                   30 |
| user         | server.domain.com  |                   30 |
| user_usrlive | server.domain.com  |                   30 |
| user_usrlive | 192.0.2.100        |                   30 |
| user_usrlive | localhost          |                   30 |
| user         | server.domain2.com |                   30 |
+--------------+--------------------+----------------------+

The user user_chatlivezilla did not appear in that filtered list. Yet checking active threads in real time showed the problem persisting:

watch -n 5 "mysqladmin proc stat | grep user_chatlivezilla | wc -l"

The counter was holding at 59 connections.

Why were 59 threads still active?#

mysqladmin inspects live thread structures maintained by the MySQL engine, while SELECT queries static grant tables. There were three distinct reasons for the pileup:

1. Existing connections are not killed automatically: Setting MAX_USER_CONNECTIONS or Governor thresholds does not terminate established sessions. MySQL only evaluates limits during handshake (sys_connect). The 59 connections were already active.
2. Persistent connections in chat applications: Live chat systems frequently enable persistent connections (PDO::ATTR_PERSISTENT or mysql_pconnect). When PHP-FPM processes linger without sending a TCP FIN packet, the database socket remains idle in sleep mode.
3. Wildcard host fallback: The application may have authenticated via a % wildcard host not caught in the initial query.

PHP-FPM worker lifecycle and socket leakage#

Chat applications and polling widgets frequently establish persistent database connections with PDO::ATTR_PERSISTENT => true. Under PHP-FPM, persistent database connections remain tied to the worker process lifecycle.

If the pool configuration at /opt/cpanel/ea-phpXX/root/etc/php-fpm.d/user.conf leaves pm.max_requests = 0 (the default setting where workers are never recycled), PHP processes retain open database sockets in Sleep state indefinitely, even after delivering HTTP responses.

To eliminate socket leakage at the application layer:

pm = ondemand
pm.max_children = 15
pm.max_requests = 500
pm.process_idle_timeout = 10s

With pm.max_requests = 500, every worker recycles after serving 500 requests, cleanly closing old database sockets via proper TCP FIN handshakes and preventing hidden connection buildup.

Phase 5: forced termination without server downtime#

To make the new limits take effect, I needed to reload the Governor service and purge stale threads without restarting MySQL, which would disrupt all other production websites.

First, I attempted to reload the service via dbctl:

dbctl restart
# Output: Incorrect syntax: unknown command "restart"

Service lifecycle is managed via Systemd:

systemctl restart db_governor

Next, I verified socket counts held by the offending system user:

ss -tanp | grep 3306 | grep -c $(id -u user)

With socket volume confirmed, I executed a surgical purge by generating KILL CONNECTION commands dynamically for idle threads:

mysql -e "SELECT concat('KILL CONNECTION ', id, ';') FROM information_schema.processlist WHERE user='user_chatlivezilla' AND command='Sleep' AND time > 10;" | grep -v "concat" | mysql

Differentiating termination commands is essential in production:

• KILL CONNECTION <id>: closes the network socket associated with the thread, freeing file descriptors and session memory buffers.
• KILL QUERY <id>: aborts only the currently executing SQL statement, leaving the TCP connection open.

Within seconds, all 59 stale threads terminated. When the application attempted to reconnect, LVE restricted incoming processes to 5 and Governor blocked attempts exceeding 20 connections, instantly stabilizing server Load Average.

Lessons learned from this incident and edge cases#

Taming database resource spikes in multi-tenant environments requires knowing how layers interact. If threads continued piling up after manual termination, two edge cases were ready for review:

• Users with SUPER privileges: There is an important difference between database engines: in MariaDB (documented in the MariaDB Knowledge Base), accounts holding SUPER or CONNECTION ADMIN privileges are completely exempt from max_user_connections. In Oracle MySQL 8.0, max_user_connections strictly applies to all accounts; holding administrative privileges only grants access to the max_connections + 1 emergency reserve slot. Always verify account grants with SHOW GRANTS FOR 'user'@'host';.
• TCP backlog and prolonged idle timeouts: When applications abandon connections in sleep state, lowering wait_timeout in /etc/my.cnf (e.g., from 28800 down to 60 seconds) flushes idle sockets faster. I covered this approach in detail in my guide on handling MariaDB connection exhaustion under CloudLinux without downtime.

Managing database stability on cPanel goes beyond editing my.cnf. It comes down to a deliberate balance between internal MySQL controls, Governor thread monitoring, and Kernel-level process isolation with LVE.

Frequently asked questions about connection limits on CloudLinux and cPanel#

Does MySQL Governor replace native MySQL MAX_USER_CONNECTIONS?#

No, they complement each other. Native MAX_USER_CONNECTIONS is static and rejects connections during handshake inside the database engine for specific accounts. MySQL Governor dynamically tracks CPU and I/O consumption; when an account exceeds thresholds, it automatically sets the throttled connection cap configured in /etc/container/mysql-governor.xml (typically 30) as a temporary containment measure.

How can I immediately lift a restriction applied by MySQL Governor?#

To lift penalties without waiting for the cooldown timeout to expire, run dbctl unrestrict <username>. To inspect which accounts are currently throttled and why, use dbctl list-restricted or monitor real-time activity with dbtop.

Why does the LVE kernel module strictly require NPROC > EP + 15?#

Because Entry Processes (EP) only track active incoming web requests handled by Apache or LiteSpeed. Every web request can spawn additional processes, including PHP-FPM workers, helper scripts, cron jobs, and SSH sessions. The Kernel enforces this 15-process buffer to prevent system process starvation and account lockouts.

How can I stop persistent PHP database connections from accumulating in Sleep state?#

Configure the account's PHP-FPM pool with periodic worker recycling (pm.max_requests = 500) and a short idle timeout (pm.process_idle_timeout = 10s). Additionally, lower wait_timeout and interactive_timeout in /etc/my.cnf from the default 28800 seconds down to 60 seconds.

See also

Databases Mitigating MariaDB connection exhaustion under Layer 7 attacks on CloudLinux and cPanel

Databases Surviving MariaDB connection exhaustion on CloudLinux with zero downtime

Infrastructure MySQL handshake desynchronization in multi-tenant environments: cPanel, CloudLinux, and CageFS

This post is licensed under CC BY-NC.