Cloudflare Error 521: Netfilter, DNAT, and Apache

Dissecting Cloudflare error 521: netfilter, DNAT, and Apache configuration drift autopsy#

When placing an application behind an edge reverse proxy or CDN such as Cloudflare, traffic should reach the origin smoothly. Sometimes, however, a quick DNS routing check triggers a deep investigation across kernel packet filters and web server configs.

I ran into this scenario while working on two Linux hosts running CloudLinux, cPanel, Imunify360, and SPFBL protection. On paper, both nodes had identical baselines: server1 and server2.

When pointing a newly registered domain on Cloudflare to each server IP before provisioning the account in cPanel, the two machines gave completely different responses:

server2 (Node B): returned HTTP 301, the expected fallback redirect for unmapped hosts.
server1 (Node A): returned Error 521 (Web Server is Down).

What initially looked like a common firewall block turned out to be a layered sequence of events: netfilter chain precedence, layer 7 interception via Lua routines inside an internal proxy, and configuration drift during Apache VirtualHost resolution.

The transport layer illusion and the xt_recent module#

The first suspicion when encountering Error 521 (where Cloudflare cannot complete a TCP handshake with origin) is that a local firewall rule dropped connections due to rate limits on proxy IP ranges.

On server1, I dumped the active iptables chains to inspect existing filters:

iptables -nvL | grep -E 'Chain|SPFBL|imunify360'

The SPFBL chain contained drop rules driven by the kernel's xt_recent module:

-A SPFBL -p tcp -m recent --rcheck --seconds 600 --name SPFBL32 --mask 255.255.255.255 --rsource -j DROP

The xt_recent module maintains dynamic IP tables in kernel memory, exposed via /proc/net/xt_recent/. The --rcheck parameter checks whether an incoming IP exists in the table and whether the last hit occurred within 600 seconds. If true, the kernel executes DROP, discarding packets silently without a reset (RST). Cloudflare reaches its connection timeout and renders Error 521 to visitors.

Before moving forward, I verified two other common causes:

SYN backlog exhaustion: running dmesg -w showed no listen queue overruns (net.ipv4.tcp_max_syn_backlog or net.core.somaxconn) and no active SYN cookie alarms.
MTU fragmentation drops: checking iptables -nvL | grep "f" confirmed no explicit rules were dropping fragmented frames.

To test whether xt_recent caused the issue, I placed a RETURN rule at the top of the SPFBL chain. In netfilter, RETURN stops evaluation in a custom chain and sends the packet back to the parent chain (INPUT), bypassing downstream drop actions:

# Save active rules before editing netfilter
iptables-save > /root/iptables_backup_$(date +%F_%H%M).rules

CF_RANGES=(
"173.245.48.0/20" "103.21.244.0/22" "103.22.200.0/22" "103.31.4.0/22"
"141.101.64.0/18" "108.162.192.0/18" "190.93.240.0/20" "188.114.96.0/20"
"197.234.240.0/22" "198.41.128.0/17" "162.158.0.0/15" "104.16.0.0/13"
"104.24.0.0/14" "172.64.0.0/13" "131.0.72.0/22"
)

for range in "${CF_RANGES[@]}"; do
    iptables -I SPFBL 1 -s $range -j RETURN
done

# Clear recent module tracking buffers
echo "/" > /proc/net/xt_recent/SPFBL32
echo "/" > /proc/net/xt_recent/SPFBL28

Even with these rules verified in iptables -nvL SPFBL, Error 521 remained. The packet counters (pkts) stayed at zero. HTTPS traffic on port 443 was not even reaching the SPFBL chain.

Netfilter precedence: when NAT runs ahead of filter#

Checking the INPUT chain with iptables -S INPUT clarified the filter layout:

-P INPUT ACCEPT
-A INPUT -j INPUT_imunify360
-A INPUT -p tcp -m tcp --dport 25 --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j SPFBL

The SPFBL chain was tied exclusively to port 25 (SMTP). Port 443 traffic was jumping directly into INPUT_imunify360.

I added a dedicated allow chain for Cloudflare ranges at the very top of INPUT:

iptables -N CLOUDFLARE_WHITE
for range in "${CF_RANGES[@]}"; do
    iptables -A CLOUDFLARE_WHITE -s $range -j ACCEPT
done
iptables -I INPUT 1 -j CLOUDFLARE_WHITE

The packet counters in CLOUDFLARE_WHITE immediately began ticking up. The kernel accepted the packets in the filter table. Still, Cloudflare kept returning Error 521.

I tested the port directly with netcat from an external host:

nc -zv 177.X.X.X 443
# Output: Connection to 177.X.X.X 443 port [tcp/https] succeeded!

Because netcat only tests layer 4, completing the three-way handshake meant the TCP socket was open. The failure had to be happening during TLS termination or inside the web server.

If packets were accepted in filter and completed the handshake, where did they divert? Inspecting the PREROUTING hook in the NAT table revealed the answer:

iptables -t nat -nvL | grep -E "52223|52224"

The output showed active redirection:

257 15004 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set i360.ipv4.splashscreen.sync src tcp dpt:443 to::52223

Imunify360 runs WebShield as an inline proxy. Before any packet hits the filter table in the INPUT chain, it traverses the PREROUTING hook in the nat table. Dynamic DNAT rules redirect incoming port 443 traffic to internal port 52223, where an internal Nginx/OpenResty instance listens.

Layer 7 traceability: WebShield, Lua, and SNI hooks#

Because WebShield acts as a reverse proxy for bot challenges and inspection, it terminates TLS connections before forwarding requests to Apache.

Looking into WebShield's error log showed what was happening:

tail -f /var/log/imunify360-webshield/error.log

The entries pointed directly to the root cause:

2026/04/14 18:36:58 [warn] 1790576#1790576: *796 [lua] sslutils.lua:123: get_cert_by_host(): Could not get SNI for request (IP: 177.X.X.X), context: ssl_certificate_by_lua*, client: 66.X.X.X, server: 0.0.0.0:52231
2026/04/14 18:29:34 [warn] 1790576#1790576: *774 [lua] sslutils.lua:148: get_cert_by_host(): Certificate not found (host: 177.X.X.X), context: ssl_certificate_by_lua*

WebShield's Nginx instance uses ssl_certificate_by_lua_block to load certificates on the fly. The sslutils.lua -> get_cert_by_host() function queries a local database to match the Server Name Indication (SNI) sent in the client handshake.

Because the test domain was configured in Cloudflare DNS but not yet added to cPanel on this server, WebShield found no matching certificate. Instead of falling back or passing the connection through, it sent a FIN/RST before sending the TLS ServerHello. Under Cloudflare SSL Strict mode, this sudden disconnect was flagged as an origin outage, producing Error 521.

To test this directly, I stopped the WebShield service:

systemctl stop imunify360-webshield

On servers where the agent manages the service, you can disable it via config:

imunify360-agent config update '{"WEBSHIELD": {"enable": false}}'

To keep WebShield active for ordinary traffic while allowing edge proxy ranges through, I added Cloudflare subnets to Imunify360's allowlist:

for range in "${CF_RANGES[@]}"; do
    imunify360-agent whitelist ip add "$range" --comment "Cloudflare Bypass"
done

This writes the ranges into Imunify360's internal SQLite database (/var/lib/imunify360/imunify360.db), instructing WebShield to bypass deep inspection for these addresses.

Testing with curl immediately succeeded:

curl -sklI https://server1.domain.com | grep HTTP
HTTP/2 200

Origin was answering again. However, one question remained: why did server2 respond with HTTP 301 while server1 returned HTTP 200?

Apache configuration drift and vhost hijacking#

Neither server had an active cPanel account for the test domain. Yet one returned a redirect and the other served full website content.

To inspect Apache's VirtualHost resolution tree, I ran:

httpd -S | grep -A 1 "port 443 namevhost" | head -n 2

Apache evaluates incoming requests against VirtualHosts in distinct steps:

Matches the physical interface IP and listening port (177.X.X.X:443).
Scans for a VirtualHost block whose ServerName or ServerAlias matches the incoming request.
If no match exists, Apache falls back to serving the request with the first VirtualHost loaded into memory for that port.

On server2, alphabetical inclusion loaded cPanel's default fallback host first. That block had a mod_rewrite rule sending unmapped hosts to port 2083 with an HTTP 301.

On server1, configuration drift had occurred. Inspecting the response headers revealed a case of VirtualHost hijacking:

curl -sklI https://server1.domain.com | grep -i Location

The request matched 01-client.conf, which happened to be the first file sorted alphabetically in /etc/apache2/conf.d/vhosts/. Apache served that client's content and TLS certificate for every orphan request.

Verifying the TLS handshake with OpenSSL confirmed the mismatch:

openssl s_client -connect 177.X.X.X:443 -servername server1.domain.com

The certificate presented in the handshake belonged to a different hosted domain.

To prevent this drift across nodes, I deployed a top-priority catch-all configuration file (000_default_catchall.conf) into /etc/apache2/conf.d/vhosts/, ensuring unmapped domains hit a predictable fallback VirtualHost.

Tracing the packet path and practical lessons#

When Error 521 appears, the quickest assumption is to blame Cloudflare or assume Apache crashed. In practice, a TCP socket can complete its handshake at the transport layer and still fail milliseconds later inside an internal proxy handling TLS termination.

Three diagnostic habits proved helpful during this incident:

Always inspect the nat table before assuming the filter table tells the complete story in iptables.
Check logs of intermediate components (such as WebShield) rather than focusing solely on Apache's access_log.
Audit VirtualHost parsing order with httpd -S so unmapped domains never accidentally expose another hosted client's site or certificate.

Validating these layers in your server stack prevents silent disconnects and keeps newly pointed test domains from serving unexpected content.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

678 reads

Creative Commons Attribution-NonCommercial

This post is licensed under CC BY-NC.