The anatomy of a troubleshooting: ghost quotas, Dovecot, and the illusion of physical disk space on cPanel

Anyone managing cPanel hosting environments frequently encounters discrepancies that initially look like baffling control panel glitches: a tenant receives alerts stating their account exceeded its storage allocation (reporting over 60 GB consumed), yet checking disk usage via terminal with du -sh shows physical storage hovering at only 25 or 30 GB.

To complicate matters, attempting to force quota recalibration using native server utilities causes the terminal to enter an infinite loop of exit code errors.

Here is the operational breakdown of this investigation: from unblocking exit code 67 loops triggered by orphaned mailbox directories to understanding Dovecot's Zlib compression architecture, clarifying why the Virtual File System (physical storage) and the application layer (logical quotas) measure two entirely different values.

Phase 1: The error 67 loop and orphaned filesystem states#

The troubleshooting began when forcing a quota recalculation for a user account:

/scripts/generate_maildirsize --confirm user

The script stalled, launching continuous background processes and outputting repetitive error lines:

The “/usr/bin/doveadm quota recalc -u orphaned@example.com” command (process 2311259) reported error number 67 when it ended.: doveadm(orphaned@example.com): Error: cmd quota recalc: User doesn't exist

Under the hood:#

The /scripts/generate_maildirsize utility is a Perl wrapper executing the following sequence:

1. Perl invokes the getdents() system call to scan /home/user/mail/example.com/.
2. For each subdirectory found on disk, it invokes Dovecot: doveadm quota recalc -u mailbox@example.com.
3. The doveadm C binary queries its authentication passdb, mapped by cPanel in /home/user/etc/example.com/passwd.

Error 67 (which maps to EX_NOUSER in POSIX sysexits.h) indicates a state desynchronization: the physical directory exists in the filesystem, but the mailbox record was deleted from the virtual authentication database.

Verifying the mismatch:

# 1. Querying the virtual password database:
grep "orphaned" /home/user/etc/example.com/passwd
# Empty output: the user does not exist logically.

# 2. Querying the filesystem:
ls -lad /home/user/mail/example.com/orphaned
# Output: drwxr-x--x 5 user user 118 Jun 5 16:26 /home/user/mail/example.com/orphaned

This represents an orphaned directory artifact: a previous operation removed the email account from the panel, but left its filesystem directory behind. Because Perl iterates directories while Dovecot validates database accounts, the script entered an endless loop.

Breaking the loop:#

To align the filesystem with cPanel's account database:

# 1. Refresh user domain mappings
/scripts/updateuserdomains

# 2. Repair mail permissions
/scripts/mailperm user

# 3. Move the orphaned directory to a backup location (or delete if obsolete)
mv /home/user/mail/example.com/orphaned /root/orphaned-backup/

# 4. Clear stale maildirsize cache files to force a clean rebuild
rm -f /home/user/mail/example.com/*/maildirsize

# 5. Run the quota generator again
/scripts/generate_maildirsize --confirm user

With the orphaned directory removed, the script completed within seconds without errors.

Phase 2: The 14 GB discrepancy between terminal and panel#

With the loop resolved, the central paradox emerged: cPanel continued reporting account usage exceeding 60 GB. However, auditing disk usage directly via terminal revealed:

du -sh /home/user
# Output: 59G

du -sh /home/user/mail/example.com/* | sort -hr | head -n 5
# 11G     /home/user/mail/example.com/admin
# 9.9G    /home/user/mail/example.com/general
# 9.6G    /home/user/mail/example.com/billing

The physical folder for general consumed exactly 9.9 GB on disk.

Yet, querying cPanel's UAPI (which powers the web interface graphs):

uapi --user=user Email list_pops_with_disk | awk '/email:/ {email=$2} /diskused:/ {printf "%-40s %s MB\n", email, $2}' | sort -k2 -nr

The output was staggering:

general@example.com                    24995.54 MB  (~24.41 GB)

A discrepancy of nearly 15 GB on a single mailbox! How can physical disk space differ so drastically from reported usage on the same partition?

Testing potential failure modes#

Before identifying the underlying mechanism, several common corruption hypotheses were tested:

Hypothesis 1: Permission locks on maildirsize or stale cPanel datastores#

If root generated maildirsize with improper permissions, cPanel's user-level daemon might fail to read disk usage and fall back on stale cache.

• Test: Ran chown user:user /home/user/mail/example.com//maildirsize and purged datastore caches via rm -rf /home/user/.cpanel/datastore/.
• Result: The 24.4 GB figure remained unchanged.

Hypothesis 2: Inconsistencies in Dovecot IMAP index caches#

If previous message deletions failed to update dovecot.index.cache, Dovecot could be counting ghost messages.

• Action: Forced a full index resync and quota recalculation:

find /home/user/mail/example.com/general -name "dovecot.index*" -type f -delete
find /home/user/mail/example.com/general -name "dovecot.list.index.log" -type f -delete
doveadm force-resync -u general@example.com '*'
doveadm quota recalc -u general@example.com

Inspecting the newly generated raw quota file:

head -n 2 /home/user/mail/example.com/general/maildirsize

The result was definitive:

0S
26209724197 311650

Dovecot consistently recorded 26,209,724,197 bytes (24.41 GB) across 311,650 individual messages. The daemon was confident: the mailbox contained 24.4 GB of data.

Phase 3: The architectural reality: zlib and logical vs. physical quotas#

The answer lies in the Maildir++ specification combined with Dovecot's Zlib compression plugin (mail_plugins = $mail_plugins zlib).

1. What du -sh measures (kernel / VFS layer):#

The du utility calls lstat() on every directory entry, reading the st_blocks field returned by filesystem inodes (ext4/XFS). This field reflects the actual physical blocks allocated on the SSD after transparent Zlib compression. Because Zlib compresses email headers and message bodies efficiently, physical disk usage compacted down to 9.9 GB.

2. What Dovecot measures (application layer):#

To sustain high I/O throughput, Dovecot follows the Maildir++ standard. If Dovecot had to decompress all 311,000 email files using read() every time a quota was queried, server CPU and I/O would collapse.

Instead, when an incoming 10 MB message is received and compressed, Dovecot writes the filename using the following convention:

1717618293.M491294P2839.server,S=10485760:2,S

The tag S=10485760 records the original uncompressed (logical) size in bytes. Although Zlib compression reduces the file's physical footprint on disk to just 2 MB, Dovecot's quota engine (src/plugins/quota/quota-maildir.c) simply parses the S= tag from the filename.

Why cPanel enforces logical quotas#

cPanel bills storage limits based on uncompressed logical size. The rationale is governance: if accounts were metered only on compressed bytes, users could store unbounded volumes of compressed files, creating severe CPU overhead whenever webmail searches, IMAP synchronizations, or decompression routines run.

Therefore, there was no bug or data corruption. The mailbox genuinely held 24.4 GB of uncompressed email data. Zlib compression effectively saved physical SSD storage on the server, but hosting package governance evaluated uncompressed message volume.

Advanced tracing with strace#

To watch doveadm extract these metadata tags at the syscall level:

strace -f -e trace=open,read,write,stat dovecot quota recalc -u general@example.com 2>&1 | grep "maildirsize"

The strace output reveals the binary opening maildirsize, iterating directory filenames to extract S= values, and writing the consolidated byte total into the file header.

To enable verbose logging in /etc/dovecot/dovecot.conf:

mail_debug = yes
auth_debug = yes

Diagnostic activity can then be monitored in real time inside /var/log/maillog.

Practical takeaways for quota discrepancies#

When addressing storage discrepancies between du -sh and cPanel:

1. Clear orphaned directories first: If quota scripts loop on exit code 67, look for directories under /home/user/mail/domain/ that no longer exist in /etc/domain/passwd.
2. Explain the physical vs. logical distinction: Clarify that server-side compression conserves physical disk blocks, but package limits govern uncompressed message payload sizes.
3. Pinpoint high-volume folders: Break down folder usage to show where messages accumulate (typically .Sent and .Trash):

   du -sh /home/user/mail/domain/account/.* 2>/dev/null | grep -v "\.$" | sort -hr

1. Offer actionable next steps: Users can either purge older archived messages or upgrade their package limit in WHM to accommodate their growing mail archive.

Understanding how the kernel, filesystem, and mail daemons calculate storage turns an apparent mystery into a straightforward, transparent technical explanation.

See also

Infrastructure Anatomy of a phantom email: filesystem and Dovecot IMAP desynchronization

Email From VestaCP to cPanel: fixing the Roundcube IMAP mailbox corruption error

Email Anatomy of a cascading failure: from spam IP blocks to Roundcube debugging in cPanel

This post is licensed under CC BY-NC.