Complete Guide to Exim Spool: Frozen Queues, Kernel Locks, and CloudLinux Troubleshooting

Anyone managing Linux mail servers running cPanel and Exim has likely encountered sudden queue surges filled with hundreds or thousands of messages flagged as frozen. When this occurs, disk queues suffer from elevated I/O wait, and delivery of legitimate messages slows down noticeably.

A frozen message does not signal an application crash: it serves as Exim's internal safety switch to prevent the server from wasting CPU cycles and network sockets repeatedly trying to dispatch messages that encountered unrecoverable delivery errors or strict policy violations.

In this guide, we break down how Exim structures spool directories, what triggers frozen by ACL events, why large attachments cause silent stalls, and how to inspect and clear the spool via the command line.

1. Spool Directory Layout, Frozen Lifecycle, and Kernel Interactions#

To understand why messages freeze, it helps to examine how Exim persists data inside /var/spool/exim/input/.

How Exim splits messages on disk#

Every spooled message is split into at least two distinct files:

1. Header file (-H): contains envelope metadata (actual sender, recipient list, timestamps, TLS parameters, and status flags). The logical frozen flag is recorded here.
2. Data file (-D): stores the raw message body and encoded attachments (typically in Base64).

This separation allows Exim to read only -H files during queue sweeps (exim -bp), saving memory and disk throughput by avoiding large message payloads.

File locks and the frozen flag#

When Exim attempts delivery:

• It opens the file with openat() and requests an exclusive lock using flock(fd, LOCK_EX | LOCK_NB). If another worker process is already handling that message, the call returns EWOULDBLOCK and Exim moves to the next queue entry, preventing duplicate deliveries.
• When a policy rule triggers or the remote server returns a permanent error, Exim writes -frozen into the -H header file. From that point on, regular queue sweeps skip the item entirely.

The Lifecycle and Disk Impact of a Frozen Email#

An email enters a frozen state when the server fails to deliver the original payload and subsequently fails to notify the sender:

1. Initial attempt and failure: Exim attempts delivery (for example, an automated cron report generated by root destined for an external mailbox). The remote MX rejects the transaction with a permanent error (5xx) or exceeds temporary retry limits (4xx).
2. Bounce generation: In compliance with RFC SMTP standards, the MTA generates a Non-Delivery Report (NDR) with a null return-path: <>.
3. Bounce delivery failure: Exim attempts to deliver this bounce back to the originating identity ([email protected]). If no valid alias exists in /etc/aliases, the mailbox is over quota, or local DNS resolution fails, the bounce cannot be delivered.
4. Freezing: Unable to deliver the message or return the bounce, Exim executes its internal deliver_freeze() routine. The header file receives a frozen flag on disk, and the daemon stops scheduling automated delivery sweeps (exim -q) for this item.

When thousands of frozen messages accumulate over weeks, /var/spool/exim/input/ exhausts directory inode limits, dramatically increasing disk I/O wait times and slowing down queue inspections.

2. Queue Auditing, Message Inspection, and Terminal Forensics#

Summarizing queued domains#

To get a consolidated view of domains accumulating queued volume:

exim -bp | exiqsumm | sort -n | tail -n 20

exim -bp inspects queued items, exiqsumm groups counts by destination domain, and tail highlights the 20 largest backlogs.

Isolating message identifiers#

When a particular domain (domain.com) shows unusual backlog:

exiqgrep -r "@domain\.com$" -i

The -i flag prints only the message ID (e.g. 1wF7Aa-00000009iej-1s3H), making it easy to pipe into downstream batch operations.

Reading metadata and message logs#

With a target ID, retrieve the historical timeline and stored headers:

# View complete message delivery history
exim -Mvl 1wF7Aa-00000009iej-1s3H

# Inspect raw headers recorded in the -H file
exim -Mvh 1wF7Aa-00000009iej-1s3H

Case study 1: authentication failures and dictionary attacks: backscatter campaigns and frozen by ACL#

A recurring entry in frozen queues is:

2026-04-21 06:13:30 Received from [email protected] H=(noreply.spamdome.com) [34.82.177.3] P=esmtps X=TLS1.3:TLS_AES_256_GCM_SHA384:256 S=3207 [email protected] T="\351\205\215\351\200\201\347\212\266\346\263\201\343\201\256\343\201\224\346\241\210\345\206\205"
2026-04-21 06:13:30 frozen by ACL

What this log entry means#

1. Forged sender envelope: the sender uses a crafted return address to provoke an automated bounce back to an external victim.
2. Encoded subject: the octal sequence \351\205\215... in T= represents UTF-8 characters (in this case, Japanese text impersonating parcel delivery notifications).
3. ACL action: the connection passed initial handshake checks, but scored high on antispam rules during the data phase (acl_smtp_data). When Exim ACLs are configured to freeze rather than deny, the message is spooled and frozen for administrative review.

Operational tip: if your server encounters large volumes of this traffic, update cPanel antispam ACL actions from Freeze to Deny (550) to prevent spam from consuming disk inodes.

Case study 2: oversized messages and storage exhaustion: large attachments and storage constraints#

When single messages retain 15 MB or more in queue for hours, two technical constraints are commonly responsible:

Remote size limits (message_size_limit)#

Exim defines message_size_limit inside /etc/exim.conf. If your server permits up to 50 MB, but the destination mail exchanger enforces 10 MB, delivery fails:

• Destination announces 250-SIZE 10485760.
• Transmitting a larger payload triggers 552 Message size exceeds fixed maximum message size. If automated bounce generation fails, the item remains frozen.

Base64 encoding overhead and mailbox quotas#

Binary files attached to email are encoded into 7-bit ASCII via Base64, adding a fixed overhead of roughly 33%:

• A 16 MB attachment expands to over 21 MB on disk and over the wire.
• If the recipient mailbox has limited quota, the local delivery agent (dovecot_lda) fails with EDQUOT (Quota exceeded). Exim freezes the message to avoid data loss.

Inspecting Queue Items Before Deletion#

Before removing spooled items in bulk, inspect representative messages to verify whether they originated from legitimate applications or security compromises:

# View complete message headers
exim -Mvh 1wG0DU-00000004C5t-42tQ

# View email body content
exim -Mvb 1wG0DU-00000004C5t-42tQ

# View individual transaction retry logs
exim -Mvl 1wG0DU-00000004C5t-42tQ

In the headers (exim -Mvh), identify the PHP origin header:

X-PHP-Originating-Script: 1001:sendmail.php

The integer preceding the colon indicates the Linux user ID (UID). This identifier pinpoints the exact directory under /home/username/public_html harboring the script.

3. Low-Level Syscall Tracing with strace and I/O Bottlenecks#

When logs do not explain a frozen queue item, trace the system calls during a forced delivery attempt:

strace -f -s 256 -e trace=network,openat,write,flock exim -M 1wF7Aa-00000009iej-1s3H

• -f: tracks child processes spawned to handle SMTP delivery.
• -s 256: prints command arguments and responses without truncation.
• -e trace=network,openat,write,flock: focuses on network operations, file opens, and locks.

Common bottlenecks revealed by strace#

1. Path MTU Discovery (PMTUD) Black Hole: if Exim completes the TCP handshake and sends DATA, but hangs indefinitely in write() calls with large Base64 chunks without receiving a reply via read(), an intermediate router is silently dropping packets with the Don't Fragment (DF) bit set.
2. Inode exhaustion: if df -h shows available space but df -i reports 100% inode utilization on /var, calls to openat() fail with ENOSPC. Exim cannot write spool headers and freezes queued messages in cascade.

Tracing Single Message Failures with strace#

When an individual message stalls repeatedly and you must pinpoint socket drops or DNS timeouts:

strace -f -tt -s 512 -e trace=openat,write,connect,unlink exim -M 1wG0DU-00000004C5t-42tQ

The -f flag monitors child threads, while -e trace=connect displays the exact destination IP address and port attempted before the failure.

4. Surgical Queue Remediation: Thawing, Retrying, and Mass Purging#

Purging frozen spam for a specific domain#

To remove frozen spam messages associated with a spam campaign:

exiqgrep -r "@domain\.com$" -i | xargs -I {} exim -Mrm {}

exim -Mrm deletes the -H and -D files from disk and removes associated entries from retry.db.

Thawing and forcing delivery for legitimate mail#

Once the underlying issue (such as a temporary DNS outage) is resolved:

# 1. Thaw the messages
exiqgrep -r "@domain\.com$" -i | xargs -I {} exim -Mt {}

# 2. Force immediate delivery attempt
exiqgrep -r "@domain\.com$" -i | xargs -I {} exim -M {}

Mass-Purging Frozen Messages#

Once verified, clean out accumulated frozen items without disrupting Exim:

exiqgrep -z -i | xargs exim -Mrm

Understanding this pipeline:

• exiqgrep -z -i: Filters exclusively frozen items (-z) and outputs raw Message-IDs (-i), one per line.
• | xargs: Batches IDs efficiently to subsequent commands, preventing process creation overhead.
• exim -Mrm: Removes the corresponding -H and -D spool files from disk storage.

5. Configuring Remote MySQL Access under CloudLinux and CageFS#

Enabling external MySQL/MariaDB connections in cPanel environments running CloudLinux requires adjustments across network bindings, CageFS mount points, and resource governors.

1. Enabling remote network listening#

By default, MySQL binds only to 127.0.0.1. To accept connections from outside, update /etc/my.cnf:

[mysqld]
bind-address = 0.0.0.0
skip-name-resolve = 1

• bind-address = 0.0.0.0: instructs MySQL to listen across all active network interfaces.
• skip-name-resolve = 1: prevents MySQL from doing reverse DNS lookups on incoming client IPs. Without this, external DNS timeouts will stall connections and exhaust max_connections.

Restart the database daemon and verify the listening port:

/scripts/restartsrv_mysql
ss -tulpn | grep 3306

2. CageFS mount points and socket mappings#

CloudLinux runs each user inside an isolated virtual filesystem jail (CageFS). When local scripts or cron jobs connect to MySQL, they need access to the database socket inside their individual jails.

Check that MySQL sockets are registered in /etc/cagefs/cagefs.mp:

/var/lib/mysql/mysql.sock
/var/run/mysqld/mysqld.sock

If database packages or shared client libraries were recently updated on the host, update user skeletons:

cagefsctl --addrpm MariaDB-client
cagefsctl --force-update

The --force-update flag rebuilds /usr/share/cagefs-skeleton/, ensuring all user jails load matching shared libraries.

3. Monitoring resource ceilings with MySQL governor#

CloudLinux includes MySQL Governor to monitor per-user CPU and disk I/O in real time.

If an external application runs unindexed queries that read full tables, MySQL Governor throttles that user to keep the rest of the server responsive.

To check whether an account is currently restricted:

dbctl list | grep username

Throttled accounts will experience slow query execution, which can look like network connectivity failures to remote clients.

6. Common Pitfalls and Network Troubleshooting (CSF/LFD)#

Firewall connection throttling in CSF (PORTFLOOD)#

If port 3306 is open in CSF but remote clients report dropped connections, review PORTFLOOD in /etc/csf/csf.conf:

PORTFLOOD = "3306;tcp;5;10"

If active, this triggers a temporary ban whenever an IP opens more than 5 connections within 10 seconds.

To safeguard legitimate client applications, allow the remote IP explicitly:

csf -a REMOTE_IP "External database access"
csf -r

Authentication plugin mismatch (caching_sha2_password)#

If older database clients or legacy PHP applications connect to MySQL 8 and encounter Authentication plugin 'caching_sha2_password' cannot be loaded, update the user authentication plugin:

ALTER USER 'username'@'REMOTE_IP' IDENTIFIED WITH mysql_native_password BY 'YourStrongPassword';
FLUSH PRIVILEGES;

File descriptor limits in the kernel (limitnofile)#

High concurrency from remote connections can cause Too many open files errors.

Check the active process limit:

cat /proc/$(pgrep mysqld)/limits | grep "Max open files"

Increase the limit by creating /etc/systemd/system/mysqld.service.d/override.conf:

[Service]
LimitNOFILE=65535

Reload systemd and restart MySQL:

systemctl daemon-reload
/scripts/restartsrv_mysql

7. Diagnostic Troubleshooting Matrix and Operational Checklist#

Quick Troubleshooting Matrix for Mail Spool Issues#

| Symptom | Probable cause | Verification command | Resolution | |

To confirm both mail and database subsystems are healthy:

1. Clean Exim queue: run exim -bpc and verify that the total message count remains low.
2. Port 3306 connectivity: test reachability from an external machine using nc -zv SERVER_IP 3306.
3. CageFS integrity: run cagefsctl --validate to verify all mount points are operational.
4. Log health: monitor /var/log/exim_mainlog and /var/log/messages to verify that emails route cleanly and no daemon errors repeat.

See also

Infrastructure MySQL handshake desynchronization in multi-tenant environments: cPanel, CloudLinux, and CageFS

Infrastructure How to bypass the global email limit in cPanel and CloudLinux: resolving error XID psafa9

Infrastructure Crontab kernel mechanics, Linux logging differences, and Exim mail spool maintenance

This post is licensed under CC BY-NC.