Anatomy of a phantom subsystem: debugging cPanel v134 metadata

The mystery of invisible databases#

As system administrators, it is common to rely on abstraction layers (like cPanel/WHM) to manage complexities. However, what happens when the engine's "truth" (MySQL/MariaDB) diverges from the interface's "truth"? Recently, I faced a scenario where 18 databases physically existed, but only 16 were rendered in the panel.

In this post, I perform a technical deep dive into how cPanel v134 manages metadata, why APIs can fail in resource-constrained environments (1GB RAM), and how to perform data surgery via CLI.

1. The anatomy of truth: three abstraction layers#

To understand the error, it is necessary to first understand how cPanel v134 (especially in CloudLinux 9.x architectures) sees a database. It doesn't run a real-time SHOW DATABASES for the user; it uses a three-pillar system:

1. Engine Layer (Reality): The MariaDB/MySQL binary and physical files in /var/lib/mysql. This is where the mysql -u root -e 'show databases;' command gets its answer through the getdents64 syscall in the data directory.
2. Metadata Layer (Source of Truth): A JSON file located at /var/cpanel/databases/user.json. This file is cPanel's "registry." If the database isn't here, it belongs to no one.
3. Indexing Layer (Performance): A global SQLite database at /var/cpanel/databases/dbindex.db. This index is fed by the cpanel-dbindex daemon and is what the WHM/cPanel UI actually queries to render tables quickly.

2. When the index falls into disregard#

In the analyzed case, the databases user_prod_host_110 and user_prod_host_1102 were present in the Engine Layer and partially in the Metadata Layer, but were ignored by the Indexing Layer.

Why the API failed (unknown app error)?#

While trying to run whmapi1 set_autodetect_database_assignments, the system returned: reason: Unknown app ("set_autodetect_database_assignments")

Technical Traceability: This error does not mean the function doesn't exist, but rather that the cPanel API dispatcher (cpsrvd) failed to load the manifest for the Cpanel::Internal::API::Mysql Perl module.

• Likely Cause: On instances with 1GB of RAM, the kernel may invoke the OOM Killer or cause a soft-fault during the mmap() of Perl's shared libraries, resulting in a partial loading of the function tree in memory.

3. Low-level diagnosis#

Before correcting, it is necessary to validate index integrity. Using sqlite3 revealed that the databases table was missing from dbindex.db.

# Checking if the global index is corrupted
sqlite3 /var/cpanel/databases/dbindex.db ".tables"

If the return is empty, the cpanel-dbindex daemon failed to initialize the schema. This happens when SQLite attempts a journal_mode=WAL and encounters I/O contention or a lack of space/inodes in the /var directory.

4. The solution: metadata surgery via Python#

When UAPI and WHMAPI1 fail, the solution is to intervene directly in the Metadata Layer. The user JSON was structurally correct, but visibility keys were missing in the main "dbs" node.

The reconciliation script#

This script synchronizes SQL reality with cPanel's "registry":

import json
import subprocess

# User metadata path
user_json = "/var/cpanel/databases/user.json"
server_ip = "51.222.205.161" # DB node IP

# 1. Get the actual SQL engine list
cmd = "mysql -u root -e 'show databases;' | grep ^user_"
mysql_dbs = subprocess.check_output(cmd, shell=True).decode().split()

with open(user_json, "r") as f:
data = json.load(f)

# 2. Visibility Injection (Master Node)
for db in mysql_dbs:
if db not in data["MYSQL"]["dbs"]:
    data["MYSQL"]["dbs"][db] = server_ip

# 3. Privilege Injection (dbusers Node)
if "user" in data["MYSQL"]["dbusers"]:
for db in mysql_dbs:
    if db not in data["MYSQL"]["dbusers"]["user"]["dbs"]:
        data["MYSQL"]["dbusers"]["user"]["dbs"][db] = server_ip

with open(user_json, "w") as f:
json.dump(data, f)

5. Forcing the indexing rebuild#

Injecting into JSON is not enough; it is necessary to "wake up" the kernel and cPanel daemons.

L3 synchronization commands:#

1. Inode Mapping: setupdbmap translates JSON changes into system permissions.

/usr/local/cpanel/bin/setupdbmap --user=user --db=mysql --update

1. Indexing Daemon Reset: In v134, cpanel-dbindex maintains an in-memory cache. Restarting forces a new JSON read.

systemctl restart cpanel-dbindex

1. User Cache Bypass (CageFS): In CloudLinux, the user operates in a virtualized filesystem. Without remounting, they will continue to see the old version of the JSON.

cagefsctl --update user
cagefsctl --remount user

See also

Infrastructure The sshd_config illusion: how to actually ensure SSH authentication security on cPanel and CloudLinux

Infrastructure Definitive Troubleshooting Guide for Let's Encrypt and SSL in DirectAdmin (CLI, CAA, and WWW Subdomain)

Infrastructure Support diary: bypassing cPanel pkgacct and aligning customer communication

This post is licensed under CC BY-NC.