Infra diary: taming antispam gateways, SPF failures, and Bash filters

The scenario: SPF hard fail after moving to the filter#

Anyone working in large-scale infrastructure knows that email flow is one of the most sensitive and frustrating gears to keep running in a state-of-the-art condition. There are days when you need to operate at multiple layers of the OSI model: from dealing with a legacy server that cannot handle a simple account rsync (where the only way out is to manually hunt and kill PIDs to avoid disk I/O lockups), to debugging complex routing logic and shell scripts that play tricks on you during auditing.

Recently, I faced a critical delivery incident involving external gateways and decided to document the technical trail (packet pathing), the forensic differentiation between SPF policies, and the evolution of a Bash automation tool for DNS zone auditing.

Initial diagnosis and multi-panel tips#

Before diving into the SPF rule, it's vital to ensure that local infrastructure is healthy. If you are dealing with a cPanel environment, the first action to isolate permission issues (UID/GID) in mailboxes is to run the native script:

# Correcting mail permissions at root level in cPanel
/scripts/mailperm --verbose

Note: This command fixes ownership and permissions for the target account's mail tree (mail/). When targeting a specific user, the identifier is provided as a positional parameter: /scripts/mailperm --verbose username.

In the scenario I resolved today, in a DirectAdmin environment, the problem was an edge rejection. I created a test account ([email protected]) and monitored logs in real-time to capture the exact error:

The observed flow was: SpamExperts delivered the message to the destination server (internally identified as DEST-10, IP 192.168.x.x), where the transaction was summarily rejected with the error:

rejected RCPT <[email protected]>: SPF: [GATEWAY_IP] is not allowed to send mail from domain.com

The antispam as the "false negative"#

The behavior was inconsistent: messages coming from @hotmail.com addresses through a SpamExperts gateway were summarily rejected due to SPF failure, while those from @gmail.com, passing through the exact same flow, were delivered.

Forensic analysis: why did Gmail pass and hotmail didn't?#

The answer lies in the technical differentiation between SPF TXT record endings (RFC 7208):

1. Hotmail uses Hard Fail (-all): Their policy instructs the receiver: "If the connection IP is not in this explicit whitelist, reject the connection immediately." Since SpamExperts served as a bridge, the connection IP seen by my server was the gateway's, not the original Hotmail source IP. Exim obeyed the RFC and rejected the transaction at RCPT.
2. Gmail uses Soft Fail (~all): The directive says: "If the IP is not in the list, the message is suspicious, but accept it for further analysis." My server accepted the connection and, since the message contained a valid DKIM signature (d=gmail.com) and the filter was a trusted router, the reputation system allowed delivery.

The definitive solution: trusted SMTP IP whitelisting#

The goal is not to relax SPF (which would open a gap for spoofing), but to inform the MTA (Exim) that SpamExperts is a trusted bridge. The Sender -> Filter -> Destination flow needs to be validated internally.

In the DirectAdmin ecosystem, the file responsible for releasing IP blocks and informing the server that those senders are trusted is:

/etc/virtual/whitelist_hosts_ip

In this file, about 73 entries of blocks and individual IPs were already tracked, but the filter provider (N-able/SpamExperts) had updated its global infrastructure, making the local list obsolete for the new delivery clusters.

I went to the official N-able documentation to find their updated public delivery IP list (e.g., X.X.X.0/24) and added them to the exception file. After saving, I restarted Exim:

systemctl restart exim
# Or service exim restart

The Bash battle: bulk MX record auditing#

During this incident, a need arose to audit hundreds of domains in a zones.txt file to identify which ones still used local or external MX records that did not point to SpamExperts.

Here, the script went through a necessary technical evolution to avoid "ghost" false positives:

Iteration 1: The expansion error#

I started with a simple one-liner: for i in 'cat zones.txt'; do echo 'consulting $i' && grep MX /var/named/$i.db | head -1; done. Failure: The use of single quotes prevented the expansion of the $i variable. The shell tried to literally read the file as $i.db.

Iteration 2: Alignment and empty variables#

I tried to use printf to generate a columnar report, but when a domain had its record filtered by grep -v, the script generated broken lines. I needed to add a gatekeeper: if [ -n "$mx" ].

Iteration 3: The DKIM false positive#

The script suddenly returned public keys from TXT records within the MX query. Example: p=MIIBIjAN.... Cause: The base64 string of the DKIM key contained the sequence "MX" attached to other characters. Since I used a generic grep, it captured the TXT line accidentally.

The "shielded" final script#

I used the -w parameter to search for the exact word (word-regexp) and awk to extract only the last column (the MX hostname):

#!/bin/bash
# Forensic Audit of MX Records
for i in $(cat zones.txt); do 
    # Filters SpamExperts and local default MX, picking only the final endpoint
    mx=$(grep -w "MX" /var/named/$i.db | egrep -v "spamexperts\.com|(^|[[:space:]])mail\.${i}[[:space:]]" | head -1 | awk '{print $NF}')

    if [ -n "$mx" ]; then 
        # Columnar formatting for a clean and aligned report
        printf "%-35s | %s\n" "$i" "$mx"
    fi
done

Production takeaways#

Managing infrastructure is a task of constant vigilance between the macro (global DNS/SPF) and the micro (bash scripts and regex). The fundamental lesson is that SPF Hard Fail (-all) requires a surgical mapping of all relay nodes. Keep your whitelist updated and your auditing tools shielded against statistical noise in cryptographic keys.

See also

Infrastructure Gmail rejecting DirectAdmin system emails: authenticating server hostnames with SPF and DKIM

Email Crontab kernel mechanics, Linux logging differences, and Exim mail spool maintenance

Infrastructure How to bypass the global email limit in cPanel and CloudLinux: resolving error XID psafa9

This post is licensed under CC BY-NC.