The Forwarding Trap and Backscatter Loops in Exim: Avoiding SPFBL and UCEPROTECT-L3 Blocks

The Forwarding Trap and the Backscatter Poison in Exim/cPanel Servers#

In this technical post-mortem, I investigate a reputation incident that appeared simple when viewed through the control panel GUI but revealed high architectural stakes once analyzed at the OS and MTA levels. A domain on a cPanel/WHM server - running Exim, Imunify360, and a Spam-Scanner integration - was listed on the SPFBL as a spam propagator.

The critical pivot for the entire analysis was that I initially failed to find the "classic" signs of direct abuse. There was no runaway PHP script in the directory /home/user/public_html firing thousands of emails via the sendmail wrapper, no obvious pattern of authorized SMTP blasting from a single compromised mailbox, and no leaked credentials. Instead, the evidence pointed toward internal routing logic and the inherent risks of external forwarding.

The definitive proof was found in the working directory pattern recorded in the Exim logs:

cwd=/var/spool/exim

In the Exim ecosystem, this value is a major fork in the road of any investigation. While a cwd=/home/user/public_html almost always points to a web-based injection (e.g., a vulnerable WordPress plugin or a webshell), the folder /var/spool/exim indicates that the MTA itself is the initiator, likely performing a retry, a bounce, a route, or a programmed forward. Instead of asking "which script is the source?", I shifted to "which legitimate routing flow is unintentionally laundering junk mail through my IP?".

1. Technical architecture & operational context#

The production environment audited consisted of multiple integrated layers:

• cPanel/WHM serving as the administrative panel;
• Exim serving as the primary Message Transfer Agent (MTA);
• Imunify360 providing web security and firewall protection;
• Spam-Scanner integrated via Pipe for content analysis;
• Local domain configured with forwarders targeting external boxes (Gmail, Outlook, Microsoft 365);
• SPFBL monitoring IP addresses reputation and managing blacklists;
• Dovecot acting as the SASL authentication agent for SMTP connections.

The local domain received legitimate and illegitimate emails and forwarded them immediately to an external recipient. This "forwarder" design is highly common in shared hosting environments, but in 2026 it introduces significant operational risks due to strict global antispam parameters.

2. The symptom and reputation under SPFBL#

The SPFBL flagged the server IP or domain as a spam source. The listing occurred despite the absence of common direct compromise indicators:

• No malicious scripts found within the website directory /home/user/public_html;
• No mass mailings originating from direct SMTP auth sessions;
• No mail queue dominated by local messages generated directly on the server.

This highlights why typical file-based security scanners miss the core issue. The SPFBL monitors the outward routing behavior and delivery reputation of the IP. When a server processes and forwards spam received from external sources, it is classified as the source of spam, regardless of origin.

3. Forwarding mechanics and the SRS trap#

The core of the problem lies in the Sender Rewriting Scheme (SRS). When an external sender emails a local address that has forwarding enabled:

[email protected] -> [email protected] -> [email protected]

If Exim forwards the mail using the original envelope details (MAIL FROM: [email protected]), the destination recipient checks the SPF record of external.com. Since our cPanel IP is not listed in the SPF record for the origin domain, the message will fail SPF checks and will be rejected or classified as spam.

To resolve this protocol limitation, Exim rewrites the envelope sender using SRS:

[email protected]

This rewriting aligns the return path with our server's domain. Although this preserves deliverability for legitimate mail, it introduces a severe reputation vulnerability: our server takes operational ownership of any forwarded spam. The recipient's mail system sees our IP actively sending the junk and penalizes our server.

4. The poison of backscatter#

Backscatter is the most damaging outcome of this configuration. It occurs when our server accepts a message, rewrites it via SRS, forwards it, and the destination recipient rejects the mail (returning a 550 Spam Rejected or 550 User Unknown message).

Upon receiving this rejection from the external server, our MTA attempts to deliver a Delivery Status Notification (DSN or Bounce) back to the sender. Since spammers routinely forge return addresses, our server ends up sending these bounces to innocent third-party victims.

The backscatter process operates as follows:

1. Spammer sends mail with a forged return address;
2. Our server accepts the message and applies SRS;
3. The message is forwarded to the destination (e.g., Gmail);
4. The destination server rejects the mail as spam;
5. Our Exim MTA generates a bounce and sends it to the forged victim address;
6. Our server IP gets listed on RBLs for transmitting unsolicited bounces (backscatter).

5. Auditing mail queue, quotas, and disk space#

Before modifying any mail server parameters, it is critical to audit the physical storage and disk quota limits of the Exim spool partition. If the spool or log partition becomes full, Exim will fail to write transactional records, leading to database corruption and silent delivery lockups.

Check disk partition usage for spool and logs directories:

# Check disk usage on key partitions
df -h /var/spool/exim
df -h /var/log

Next, evaluate the total size and behavior of the Exim mail queue to verify if there is an excessive buildup of frozen messages or bounce messages:

# Check the number of messages in the queue
exim -bpc

# List the active queue contents
exim -bp

If the queue contains a disproportionate number of items, or if the directory /var/spool/exim is near maximum capacity, legitimate emails will experience severe delivery delays, causing reputation drops.

6. Forensic queue and flow triage with exigrep#

The exigrep tool is an essential utility for analyzing Exim environments. Because the primary log file /var/log/exim_mainlog logs transactions asynchronously, a standard grep will show disjointed log lines because mail receipts, deliveries, and errors are interleaved.

To group all log entries belonging to a single transaction for a given domain, run:

zgrep "domain.com" /var/log/exim_mainlog* | exigrep -l "domain.com"

The -l flag tells the utility to output only the initial line of each transaction. exigrep tracks the Exim message identifier and displays the full atomic set of logs related to that transaction, presenting a complete delivery and routing history.

7. Tracking UID, GID, and cwd evidence#

A vital investigative step is tracking the Current Working Directory (cwd) parameter where messages are injected into the queue to determine if the source is web-based or MTA-routed.

To query for Exim log entries associated with a user account and their respective cwd, run:

grep "user" /var/log/exim_mainlog | grep "cwd="

• cwd=/home/user/public_html: Indicates a web application script (e.g., a vulnerable WordPress plugin or webshell) injected the message;
• cwd=/var/spool/exim: Indicates the message was initiated internally by the MTA (mail routing, bounce generation, or active forwarders).

Differentiating between these paths prevents administrators from wasting time auditing application files when the root cause is a routing policy issue.

8. Dovecot authentication forensics#

While forwarding issues are highly likely, a comprehensive audit must rule out compromised SMTP credentials. On cPanel servers, Dovecot manages the SASL authentication layer for Exim.

To audit and count SMTP authentication sends by user IP address, run:

grep "dovecot_login:user" /var/log/exim_mainlog | awk '{print $NF}' | sort | uniq -c

If the command lists a high volume of logins originating from suspicious geo-IP blocks, the associated user password must be changed immediately and SMTP credentials hardened.

9. Ghost forwarders detection and routing files#

A common problem in Exim/cPanel is the "Ghost Forwarder," where the cPanel interface lists no active forwarders, yet Exim continues to redirect messages externally. This typically occurs due to corrupted database files or manual entries on the filesystem.

The accurate configuration state is verified by checking the raw aliases files on the filesystem:

# Check the domain aliases file
cat /etc/valiases/domain.com

# Check the custom email filters file
cat /etc/vfilters/domain.com

These configuration files contain routing tables evaluated dynamically by Exim. If there are manually edited forwarders pointing to external targets, they must be cleaned directly and cPanel index databases synced.

10. Endpoint-side compromise and strace#

If a user's local mail client (such as Outlook or Thunderbird) is infected with malware, it may transmit spam using valid login credentials. In this situation, the logs show legitimate user connections.

To inspect running Exim processes and monitor what is being written to network sockets in real time without exposing extensive message contents, execute the following command:

strace -p <EXIM_PROCESS_PID> -s 80 -e write

This commands traces the write() system calls, displaying the first 80 characters of the payload. It allows administrators to verify spam templates and invalid headers at the exact moment they enter the system.

11. Delivery stack validation (connectivity and ports)#

To confirm that Exim is listening on the correct network ports and that the delivery infrastructure is responsive, audit the active sockets:

# Verify active Exim listening ports
ss -lntp | grep -E "25|465|587"

If the sockets are correct, review the firewall policies. For CSF environments, ensure the necessary SMTP ports are allowed in the configuration file /etc/csf/csf.conf:

# Configuration example in /etc/csf/csf.conf
TCP_IN = "25,465,587,993,995"
TCP_OUT = "25,465,587,993,995"

Lastly, execute network tests on the SMTP ports from an external or local system to ensure services respond properly:

# Test connection on port 25 using netcat
nc -zv mail.domain.com 25

# Test SMTP connection interactively
telnet mail.domain.com 25

# Test secure SMTP TLS handshake
openssl s_client -connect mail.domain.com:25 -starttls smtp

12. Verifying DKIM, DMARC, and forwarding signatures#

During forwarding operations, messages must maintain the original DKIM signature, or the forwarding MTA must sign the message under its own domain to ensure DMARC alignment. If DKIM or DMARC records are misconfigured, destination servers will block the mail.

Verify that the local public DKIM key matches the published DNS record:

# Validate local DKIM configuration
opendkim-testkey -d domain.com -s default -vvv

Next, query the active DMARC record for the domain:

# Check DMARC DNS settings
dig TXT _dmarc.domain.com

To ensure OpenDKIM is configured to sign outgoing mail on the system, confirm the signature settings are active in the configuration file /etc/opendkim.conf:

grep -i "Signing" /etc/opendkim.conf

13. Backup plan and safety rollback#

Modifying critical MTA parameters and firewall rules requires a reliable backup plan to allow restoration of the server to its baseline if unforeseen delivery failures occur.

Preventive backup procedure#

Before modifying any configuration files, run the following backup commands:

# 1. Back up the main Exim configuration file
cp /etc/exim.conf /root/exim.conf.bak.$(date +%Y%m%d)

# 2. Back up forwarding aliases recursively
cp -r /etc/valiases/ /root/valiases-backup/

# 3. Back up the active iptables firewall rules
iptables-save > /root/iptables-backup-$(date +%Y%m%d).rules

Rollback and reversion procedure#

If the changes disrupt mail operations, revert configurations using this playbook:

# 1. Restore the original Exim configuration file
cp /root/exim.conf.bak.YYYYMMDD /etc/exim.conf

# 2. Restore the original mail aliases
cp -r /root/valiases-backup/* /etc/valiases/

# 3. Restore the original firewall policies
iptables-restore < /root/iptables-backup-YYYYMMDD.rules

# 4. Restart the Exim MTA service
systemctl restart exim

# 5. Verify basic Exim configuration syntax
exim -bV

14. Advanced Exim log diagnostics#

Forensic diagnostic processes depend heavily on monitoring the active Exim log files. On cPanel servers, the main log is located at /var/log/exim_mainlog.

Use the following commands to isolate specific mail server errors:

# 1. Review the latest 100 entries in the main log
tail -100 /var/log/exim_mainlog

# 2. Filter for rejected delivery attempts
grep -i "reject\|denied" /var/log/exim_mainlog | tail -20

# 3. Filter for bounces or frozen messages in the queue
grep -i "bounce\|frozen" /var/log/exim_mainlog | tail -20

# 4. Filter for Dovecot SMTP login sessions
grep -i "dovecot_login\|dovecot_plain" /var/log/exim_mainlog | tail -20

15. Exim configuration syntax validation#

Syntax errors in the configuration file /etc/exim.conf can prevent Exim from starting up, causing immediate mail delivery failure. Always validate configuration syntax before reloading the daemon.

Validate the configuration using native Exim options:

# Validate the overall configuration file syntax
exim -bV

# Display active runtime configuration options
exim -bP | head -20

# Run a verbose configuration check to flag errors or warnings
exim -d -v 2>&1 | grep -i "error\|warn"

If these tools report any syntax errors or warning flags, correct the issues before applying changes to the service.

16. Exim mitigations and push vs. pull alternatives#

To permanently resolve reputation issues associated with forwarding, migrate from a Push model (active forwarding) to a Pull model (remote fetching).

• Push Model (Forwarding): Our server receives a message, applies SRS, and pushes it to Gmail. If the content is spam, our IP is flagged; if it is rejected, we process backscatter.
• Pull Model (POP3/IMAP Fetching): The recipient configures Gmail to log into the local server using POP3/IMAP and fetch messages.

In the Pull model, our server functions purely as a storage repository. Incoming spam is classified locally and retained in the Spam folder. The MTA never attempts outbound delivery to a third-party server, protecting the IP space.

If the user must maintain active forwarders for business workflows, implement the following Exim mitigations in WHM:

1. Ensure Sender Rewriting Scheme (SRS) is enabled globally to maintain SPF alignment for legitimate emails;
2. Enable the option "Do not forward mail to external recipients if it is detected as spam" in the WHM Exim Configuration Manager. This forces local spam analysis before forwarding, blocking junk mail before it enters the outbound queue.

17. Proactive reputation monitoring#

To prevent discovering IP blocks only after client complaints, set up automated checking routines for Real-time Blackhole Lists (RBLs) and reputation APIs.

Execute this script loop to query public blacklists:

# Query the server IP address against public RBLs
IP="YOUR_IP_HERE"
for rbl in zen.spamhaus.org bl.spamcop.net; do
    result=$(dig +short $(echo $IP | awk -F. '{print $4"."$3"."$2"."$1}').$rbl)
    if [ -n "$result" ]; then
        echo "LISTED on RBL $rbl: $result"
    fi
done

Additionally, request status reports from the SPFBL database via API:

curl -s "http://spfbl.net/api/v1/check/YOUR_IP_HERE"

18. ASN-Wide Blocklists (UCEPROTECT-L3), .forward Loops, and Smart Host Routing#

Anyone running cPanel and Exim servers in large cloud datacenters (such as OVH, Hetzner, or DigitalOcean) has likely encountered bulk delivery rejections where the server IP itself has a clean track record, yet emails are refused due to provider-wide blacklists.

Worse still, a common internal configuration oversight can turn that external block into a local storm: hundreds of error notifications looping indefinitely between root and the mail daemon, clogging the Exim spool with frozen messages.

In this guide, we examine a real incident where a UCEPROTECT-Level3 block collided with a misconfigured /root/.forward file, breaking down the log traces, ruling out compromise vectors, and applying a definitive cleanup script.

Asn-level blacklisting: understanding UCEPROTECT level 3#

The problem typically surfaces as a delivery failure inside /var/log/exim_mainlog when attempting to dispatch legitimate messages:

2026-06-18 16:00:00 H=mx.remote.com [192.0.2.1]: SMTP error from remote mail server after RCPT TO:<[email protected]>: 550-5.7.1 Service unavailable; Client host [40.160.3.114] blocked using UCEPROTECT-Level3

And in the reputation lookup report:

Reason for listing - Your ISP OVH, FR/AS16276 is UCEPROTECT-Level3 listed because of a spamscore of 216.

How this blacklist operates#

Unlike standard DNSBLs that target individual IP addresses (Level 1) or /24 subnets (Level 2), UCEPROTECT-Level3 blacklists the provider's entire Autonomous System Number (ASN).

If other customers within that same hosting network generate enough spam trap hits to exceed the ASN threshold over a rolling 7-day window, all IPs belonging to that provider are blacklisted, regardless of whether your specific host has sent any spam.

The SMTP conversation during rejection#

When Exim dispatches outbound mail:

1. The kernel opens an outbound TCP connection via connect() to destination port 25.
2. The remote MTA delivers the SMTP banner (220) and Exim sends EHLO.
3. At the RCPT TO:<[email protected]> stage, the destination MTA runs a reverse DNS check: <inverted_IP>.dnsbl.uceprotect.net.
4. Because the ASN is listed, the recipient immediately returns 550 5.7.1 Service unavailable and terminates the connection.

Uncovering spooled message accumulation#

Running a queue summary check to inspect spooled mail:

exim -bp | exiqsumm

The output showed an unusual distribution:

Count  Volume  Oldest  Newest  Domain
-----  ------  ------  ------  ------
    1    48KB     50m     50m  domain_a.com
    5   330KB      8h      8h  domain_b.com
  291    13MB     24h      0m  hostname.domain.com
   11  1224KB     23m     23m  domain_c.com
---------------------------------------------------------------
  327    24MB     72h      0m  TOTAL

Interpreting this backlog#

Out of 327 queued items, 291 were addressed to the server's own hostname (hostname.domain.com), with the newest arrival logged zero minutes ago (Newest: 0m). A continuous leak was generating messages every few seconds.

Listing the transaction IDs in the spool:

exiqgrep -i -r "hostname.domain.com"

Inspecting message headers with exim -Mvh:

1wFdCz-0000000EPmH-2H54-H
mailnull 47 12
<>
1776885961 0
-ident mailnull
-received_protocol local
-frozen 1776885961
-localerror
XX
[email protected]

182P Received: from mailnull by hostname.domain.com with local (Exim 4.99.1)
        id 1wFdCz-0000000EPmH-2H54
        for [email protected];
        Wed, 22 Apr 2026 16:26:01 -0300
054  X-Failed-Recipients: [email protected]
029  Auto-Submitted: auto-replied
071F From: Mail Delivery System <[email protected]>
039T To: [email protected]
059  Subject: Mail delivery failed: returning message to sender

Decoding the failure structure#

• mailnull 47 12: generated by mailnull, the unprivileged system user Exim uses to handle bounce processing.
• Empty sender (<>): required by RFC 5321 for automated bounce notifications to prevent mail loops.
• -localerror and -frozen: Exim froze the message after successive delivery failures.
• X-Failed-Recipients: [email protected]: the original intended recipient was root.

Root cause: the forwarding loop in .forward#

Checking the forwarding configuration for user root:

cat /root/.forward

The file contained:

[email protected]

The double-bounce cycle in action#

Combining this forward directive with the ASN-level block created an escalating feedback loop:

1. System trigger: cron jobs or firewall alert services (such as CSF's lfd) produce diagnostic output addressed to root.
2. Forward expansion: Exim parses /root/.forward and rewrites the destination to [email protected].
3. Remote route selection: if the server hostname is not explicitly listed in /etc/localdomains, Exim treats hostname.domain.com as an external destination via lookuphost and queues it for remote_smtp.
4. External rejection by RBL: the remote gateway rejects the connection because the server IP belongs to a blacklisted ASN.
5. Bounce generation: Exim generates a failure notification (Mailer-Daemon) addressed back to the originating caller (cpanel or root). That message encounters /root/.forward again, creating an infinite circular chain.
6. Protective freezing: detecting multiple delivery failures for an automated notification, Exim flags the message as frozen in the spool.

Ruling out secondary compromise vectors#

Before purging the spool, rule out whether the server has been compromised:

Hypothesis 1: Php script abuse#

Check whether web scripts are injecting spam via PHP's mail() function:

grep -r "X-PHP-Originating-Script" /var/spool/exim/input/

If the result is empty, local PHP applications are not injecting spam.

Hypothesis 2: Compromised SMTP authentication#

Inspect whether an existing mailbox credential has been brute-forced:

grep "P=esmtpa" /var/log/exim_mainlog | awk -F'A=' '{print $2}' | sort | uniq -c | sort -n

If no individual user accounts exhibit abnormal volume spikes, credentials remain secure.

Hypothesis 3: Raw socket outbound spam on port 25#

Confirm no hidden binary is bypassing Exim by communicating directly over port 25:

lsof -i :25 -n -P | grep -v "exim"

If only Exim processes appear, no external processes are bypassing the local mail agent.

Remediation script and spool cleanup#

This script eliminates the recursive loop, purges frozen double bounces, and ensures system mail stays local:

#!/usr/bin/env bash
# ==============================================================================
# Exim Spool Cleanup and Forwarding Loop Remediation
# ==============================================================================
set -euo pipefail

echo "[+] 1. Removing problematic forwarding file..."
if [ -f /root/.forward ]; then
    rm -f /root/.forward
    echo "[✔] /root/.forward successfully removed."
fi

echo "[+] 2. Purging frozen bounce messages from spool..."
# Remove frozen bounces with empty sender (<>)
exiqgrep -i -f '^<>$' | xargs -r exim -Mrm

# Remove spooled messages targeted at the server hostname
exiqgrep -i -r "$(hostname)" | xargs -r exim -Mrm
echo "[✔] Spool purged."

echo "[+] 3. Updating system aliases in /etc/aliases..."
# Direct root notifications to /dev/null or a valid local mailbox
if ! grep -q "^root:" /etc/aliases; then
    echo "root: /dev/null" >> /etc/aliases
else
    sed -i 's/^#\?root:.*/root: \/dev\/null/g' /etc/aliases
fi

newaliases
echo "[✔] Aliases database rebuilt."

echo "[+] 4. Verifying hostname presence in /etc/localdomains..."
HOSTNAME_STR=$(hostname)
if ! grep -q "${HOSTNAME_STR}" /etc/localdomains; then
    echo "${HOSTNAME_STR}" >> /etc/localdomains
    echo "[✔] Hostname added to /etc/localdomains."
fi

echo "[+] Remediation completed successfully."

Strategic resolution: using a smarthost to bypass ASN blocks#

Once the local loop is resolved, the queue returns to normal levels. However, the external delivery limitation caused by UCEPROTECT-Level3 remains an issue for any recipient host strictly enforcing that list.

Because ASN-level lists penalize every tenant across a cloud provider's network, paying removal fees to controversial blacklist operators is neither sustainable nor recommended.

The resilient operational fix is to decouple mail delivery from your server's host IP by routing outbound mail through an authorized SMTP relay (Smarthost) such as Amazon SES, Mailgun, or SendGrid:

1. Open WHM and access the Exim Configuration Manager (Advanced Editor).
2. In the ROUTERSTART block, configure a router directive directing non-local traffic to your authenticated relay.
3. In TRANSPORTSTART, configure TLS transport and authentication credentials matching your relay provider.

By handing off outbound mail through a dedicated relay platform, tenant email delivery relies on specialized IP pools with managed sender reputation, insulating your mail operations from neighborhood ASN blocks.

19. Operational Troubleshooting Checklist and Risk Matrix#

Operational troubleshooting checklist: Exim & forwarding#

• [ ] Initial Triage: Check disk space on spool /var/spool/exim and review initial logs.
• [ ] Transaction Analysis: Group asynchronous Exim logs by Message-ID using exigrep.
• [ ] Injection Verification: Inspect the cwd parameter to distinguish application injections (public_html) from MTA routing.
• [ ] Alias Auditing: Verify configuration files /etc/valiases/domain.com and /etc/vfilters/domain.com.
• [ ] Configuration Backups: Create backups of exim.conf, valiases, and active firewall policies.
• [ ] Antispam Filtering: Enable local spam filtering rules for forwarders in WHM.
• [ ] Architecture Migration: Transition users from a Push (forwarding) to a Pull (POP3/IMAP) model.
• [ ] DKIM/DMARC Audit: Verify domain cryptographic signatures and DMARC alignments.
• [ ] Connectivity Testing: Audit open Exim ports and verify firewall access control lists.
• [ ] Reputation Monitoring: Monitor IP address status across RBL databases and the SPFBL API.

Delivery and operational risk assessment matrix#

See also

Infrastructure Anatomy of a phantom email: filesystem and Dovecot IMAP desynchronization

Servers Anatomy of a cascading failure: from spam IP blocks to Roundcube debugging in cPanel

Email Anatomy of a dangling pointer in Exim: debugging the "transport not found" error in cPanel with smarthost and SRS

This post is licensed under CC BY-NC.