Infra diary: taming antispam gateways, SPF

Q: The antispam as the "false negative"?

The behavior was inconsistent: messages coming from @domain.com addresses through a SpamExperts gateway were summarily rejected due to SPF failure, while those from @domain.com, passing through the exact same flow, were delivered.

Infra diary: taming antispam gateways, SPF failures, and Bash filters#

Anyone working in large-scale infrastructure knows that email flow is one of the most sensitive and frustrating gears to keep running in a state-of-the-art condition. There are days when you need to operate at multiple layers of the OSI model: from dealing with a legacy server that cannot handle a simple account rsync (where the only way out is to manually hunt and kill PIDs to avoid disk I/O lockups), to debugging complex routing logic and shell scripts that play tricks on you during auditing.

Recently, I faced a critical delivery incident involving external gateways and decided to document the technical trail (packet pathing), the forensic differentiation between SPF policies, and the evolution of a Bash automation tool for DNS zone auditing.

Initial diagnosis and multi-panel tips#

Before diving into the SPF rule, it's vital to ensure that local infrastructure is healthy. If you are dealing with a cPanel environment, the first action to isolate permission issues (UID/GID) in mailboxes is to run the native script:

# Correcting mail permissions at root level in cPanel
/scripts/mailperm --verbose

Note: This command scans the server correcting ownership and permissions for critical folders like mail/ and etc/. If it were for a specific user, I would use --login=username.

In the scenario I resolved today, in a DirectAdmin environment, the problem was an edge rejection. I created a test account ([email protected]) and monitored logs in real-time to capture the exact error:

The observed flow was: SpamExperts delivered the message to our destination server (internally identified as DEST-10, IP 192.168.x.x), where the connection was summarily dropped with the error:

rejected RCPT <[email protected]>: SPF: [GATEWAY_IP] is not allowed to send mail from domain.com

The antispam as the "false negative"#

The behavior was inconsistent: messages coming from @domain.com addresses through a SpamExperts gateway were summarily rejected due to SPF failure, while those from @domain.com, passing through the exact same flow, were delivered.

Forensic analysis: why did Gmail pass and hotmail didn't?#

The answer lies in the technical differentiation between SPF TXT record endings (RFC 7208):

Hotmail uses Hard Fail (-all): Their policy instructs the receiver: "If the connection IP is not in this explicit whitelist, reject the connection immediately." Since SpamExperts served as a bridge, the connection IP seen by my server was the gateway's, not the original Hotmail source IP. Exim obeyed the RFC and dropped the packet.
Gmail uses Soft Fail (~all): The directive says: "If the IP is not in the list, the message is suspicious, but accept it for further analysis." My server accepted the connection and, since the message contained a valid DKIM signature (d=domain.com) and the filter was a trusted router, the reputation system allowed delivery.

The definitive solution: trusted SMTP IP whitelisting#

The goal is not to relax SPF (which would open a gap for spoofing), but to inform the MTA (Exim) that SpamExperts is a trusted bridge. The Sender -> Filter -> Destination flow needs to be validated internally.

In the DirectAdmin ecosystem, the file responsible for releasing IP blocks and informing the server that those senders are trusted is:

/etc/virtual/whitelist_hosts_ip

In this file, we already tracked about 73 entries of blocks and individual IPs, but the filter provider (N-able/SpamExperts) had updated its global infrastructure, making our local list obsolete for the new delivery clusters.

I went to the official N-able documentation to find their updated public delivery IP list (e.g., X.X.X.0/24) and added them to the exception file. After saving, I restarted Exim:

systemctl restart exim
# Or service exim restart

The Bash battle: bulk MX record auditing#

During this incident, a need arose to audit hundreds of domains in a zones.txt file to identify which ones still used local or external MX records that did not point to SpamExperts.

Here, the script went through a necessary technical evolution to avoid "ghost" false positives:

Iteration 1: The expansion error#

I started with a simple one-liner: for i in 'cat zones.txt'; do echo 'consulting $i' && grep MX /var/named/$i.db | head -1; done. Failure: The use of single quotes prevented the expansion of the $i variable. The shell tried to literally read the file as $i.db.

Iteration 2: Alignment and empty variables#

I tried to use printf to generate a columnar report, but when a domain had its record filtered by grep -v, the script generated broken lines. I needed to add a gatekeeper: if [ -n "$mx" ].

Iteration 3: The DKIM false positive#

The script suddenly returned public keys from TXT records within the MX query. Example: p=MIIBIjAN.... Cause: The base64 string of the DKIM key contained the sequence "MX" attached to other characters. Since I used a generic grep, it captured the TXT line accidentally.

The "shielded" final script#

I used the -w parameter to search for the exact word (word-regexp) and awk to extract only the last column (the MX hostname):

#!/bin/bash
# Forensic Audit of MX Records
for i in $(cat zones.txt); do 
# Filters SpamExperts and local default MX, picking only the final endpoint
mx=$(grep -w "MX" /var/named/$i.db | egrep -v "spamexperts.com|mail.|$i" | head -1 | awk '{print $NF}')

if [ -n "$mx" ]; then 
    # Columnar formatting for a clean and aligned report
    printf "%-35s | %s\n" "$i" "$mx"
fi
done

Production takeaways#

Managing infrastructure is a task of constant vigilance between the macro (global DNS/SPF) and the micro (bash scripts and regex). The fundamental lesson is that SPF Hard Fail (-all) requires a surgical mapping of all relay nodes. Keep your whitelist updated and your auditing tools shielded against statistical noise in cryptographic keys.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

This post is licensed under CC BY-NC.