Cloudflare Error 521 on CloudLinux: TCP to Apache

Dissecting a Cloudflare error 521 on CloudLinux: from TCP handshakes to Apache segfaults#

When routing websites through an edge network such as Cloudflare, connection errors often look like basic routing or firewall blocks. In some cases, however, the real failure sits much deeper within operating system internals.

I recently investigated a case on a cPanel and CloudLinux host where the primary firewall (CSF) was disabled. The goal was straightforward: allow Cloudflare proxy traffic without bottlenecks while restoring visitor IP addresses to Apache. Instead, the site produced 521 Web Server is Down errors, VirtualHost resolution issues, and worker segmentation faults logged directly by the kernel.

Below is the full troubleshooting walkthrough, tracing system calls, packet filtering tables, and the Apache C module responsible for the crash.

Handling hybrid traffic without CSF and iptables raw controls#

The initial goal was to ensure Cloudflare proxy traffic reached Apache while visitor IPs were restored properly alongside direct connections. With CSF offline, I worked directly with the raw and filter tables in iptables, in addition to configuring mod_remoteip inside EasyApache 4.

Layer 3 packet control with iptables raw#

Without the LFD daemon managing dynamic ipsets, I added rules to the top of the INPUT chain and the raw table to optimize throughput and remove connection tracking (conntrack) overhead for Cloudflare nodes:

# Allow rules in the INPUT chain (Filter table)
for ip in $(curl -s https://www.cloudflare.com/ips-v4); do
    iptables -I INPUT -p tcp -m multiport --dports 80,443 -s $ip -j ACCEPT
done

# Bypass conntrack in the raw table to reduce overhead
for ip in $(curl -s https://www.cloudflare.com/ips-v4); do
    iptables -t raw -I PREROUTING -s $ip -p tcp --dport 80 -j NOTRACK
    iptables -t raw -I PREROUTING -s $ip -p tcp --dport 443 -j NOTRACK
done

The kernel parses the raw table before mangle and nat. Using the NOTRACK target tells the Linux nf_conntrack subsystem to bypass state tracking for these packets. This prevents exhausting the state table at /proc/sys/net/netfilter/nf_conntrack_max during heavy bursts of edge traffic.

Application layer and mod_remoteip architecture#

In EasyApache 4, proxy header rewriting occurs during the post-read-request phase of the Apache request cycle. The module replaces the client IP pointer inside the connection struct with the value received in the CF-Connecting-IP header.

I added this global directive to /etc/apache2/conf.d/includes/pre_main_global.conf:

<IfModule mod_remoteip.c>
    RemoteIPHeader CF-Connecting-IP
    # Official Cloudflare ranges
    RemoteIPInternalProxy 103.21.244.0/22 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
    RemoteIPInternalProxy 127.0.0.1
</IfModule>

Using RemoteIPInternalProxy instead of RemoteIPTrustedProxy is critical here. The InternalProxy directive strips proxy IPs from the forwarding chain and treats them as trusted internal infrastructure. On CloudLinux, this is necessary so the kmodlve kernel module can enforce Entry Process (EP) limits per visitor IP, preventing all incoming requests from aggregating against the account quota under a single Cloudflare address.

Isolating Error 521 symptoms and TCP socket states#

Even with firewall ports open, requests through Cloudflare returned 521 Web Server is Down. Direct testing via local /etc/hosts resolved without problems.

I began isolating each step of the connection pipeline.

Testing sockets and headers with local curl#

I simulated edge requests locally to rule out datacenter or external routing filters:

curl -v -H "CF-Connecting-IP: 1.1.1.1" --resolve domain.com:80:127.0.0.1 http://domain.com

Response received:

* Connected to domain.com (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> CF-Connecting-IP: 1.1.1.1
< HTTP/1.1 200 OK
< Content-Type: text/html
<html><head><META HTTP-EQUIV="refresh" CONTENT="0;URL=/cgi-sys/defaultwebpage.cgi"></head><body></body></html>

The TCP socket opened, but Apache served the default cPanel placeholder (defaultwebpage.cgi). This happens when Apache fails to match the Host: domain.com header against the correct VirtualHost, usually caused by IP mismatches between public interfaces and loopback addresses.

Checking connection tables with netstat#

Next, I checked socket states for incoming Cloudflare IP ranges:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | grep -E "(103.21.244|108.162.192|190.93.240)"

The output was completely empty. External traffic from Cloudflare was not staying in the ESTABLISHED state. Connections were terminating immediately after the initial SYN.

I ruled out three common network issues:

Apache listening on port 80: checked via ss -tulpn | grep :80. Apache was listening on 0.0.0.0:80.
Residual firewall drops: checked iptables -L INPUT -n to verify no DROP or REJECT rules were active.
MTU size issues: tested using ping -M do -s 1472 103.21.244.1, without receiving Packet Too Big ICMP responses.

Finding the root cause: inspecting segfaults in the kernel buffer#

The lack of active connections in netstat combined with Error 521 suggested that Apache workers were crashing at the moment of accept().

Checking kernel logs with dmesg confirmed this:

dmesg | tail -n 50

The log revealed repeating memory faults:

[1821374.907238] httpd[3338551]: segfault at 199c ip 0000149c9f07fa71 sp 0000149c9c781720 error 4 in libbrotlienc.so.1.0.9[149c9f001000+a3000]
[1821374.907252] httpd[3338556]: segfault at 199c ip 0000149c9f07fa71 sp 0000149c9bd7c720 error 4 in libbrotlienc.so.1.0.9[149c9f001000+a3000]

Dissecting the page fault message#

The crash details in dmesg explain what occurred:

httpd[3338551]: the Apache worker process handling the request.
segfault at 199c: attempt to read unmapped memory at 0x199c. A low memory address usually means a base pointer in C was NULL, and the code attempted to access a struct field at offset 199c.
ip 0000149c9f07fa71: the Instruction Pointer (RIP in x86_64) pointing to the binary offset of the crash.
error 4: in x86 page fault error codes, bit 2 set (value 4) denotes a user-space read fault rather than a kernel panic.
in libbrotlienc.so.1.0.9: the Brotli compression library.

The sequence leading to Error 521 became clear. When Cloudflare connects to the origin, it includes Accept-Encoding: br. Apache forwarded response payloads into the mod_brotli output filter. A null pointer issue inside libbrotlienc.so caused the worker to receive a SIGSEGV from the kernel, terminating before completing the HTTP response.

From the perspective of Cloudflare, this dropped connection appeared as a TCP RST, prompting Error 521.

I uninstalled the unstable module and restarted Apache:

yum remove ea-apache24-mod_brotli -y
systemctl restart httpd

When keeping the module for core analysis is necessary, attaching strace via strace -f -p $(cat /var/run/apache2/httpd.pid) -e trace=memory,network or enabling core dumps with ulimit -c unlimited allows step-by-step debugging in gdb.

SNI mismatches and AutoSSL validation#

After resolving the segmentation faults, I checked TLS handshakes using OpenSSL:

openssl s_client -connect 127.0.0.1:443 -servername domain.com

The output showed a secondary certificate issue:

 0 s:CN=server.host.net
 i:C=US, O=Let's Encrypt, CN=R13

The server presented the host's default machine certificate rather than the domain certificate.

Under Cloudflare Full (Strict) SSL mode, the SNI name must match the origin certificate. When Apache falls back to the server hostname certificate, the edge rejects the connection.

I triggered AutoSSL to regenerate and attach the user certificate:

/usr/local/cpanel/bin/autossl_check --user usuario_cpanel

Once the VirtualHost presented the matching certificate, TLS handshakes succeeded.

Residual WAF rules and Imunify360 lists#

Even with CSF inactive and /etc/hosts.allow verified, CloudLinux hosts often run Imunify360 filters through ipset.

To verify no Cloudflare IP ranges were caught in greylists:

imunify360-agent graylist ip list | grep -E "103.21.244|108.162.192|172.64"

I then added the official ranges to the persistent whitelist:

for range in $(curl -s https://www.cloudflare.com/ips-v4); do
    imunify360-agent whitelist ip add "$range" --comment "Cloudflare edge"
done

Rebuilding LVE telemetry and log processing#

With Apache workers stabilized, mod_remoteip active, and certificates in place, traffic flowed normally. To refresh cPanel metrics without waiting for scheduled nightly cronjobs, I triggered manual updates.

Synchronizing LVE hardware metrics#

The lvestats daemon aggregates data from /proc/lve/list. Restarting the daemon forces immediate ingestion into its SQLite database:

systemctl restart lvestats

# Review user process and memory utilization
lveinfo --user usuario_cpanel --show-columns "EP,LCPU,MEM"

Forcing log ingestion with AWStats#

To update web statistics with restored client IPs:

/scripts/runweblogs usuario_cpanel
tail -f /usr/local/cpanel/logs/cpanellogd_log

From edge handshakes to kernel traces: practical takeaways#

Error 521 on Cloudflare does not always mean Apache stopped running or that an IP was blocked. In this scenario, TCP connections succeeded at the network level, but Apache workers crashed milliseconds later when compressing responses with a faulty shared library.

Three diagnostic steps proved most useful during this incident:

Check system logs with dmesg whenever TCP sockets vanish from netstat without leaving entries in Apache access logs.
Configure RemoteIPInternalProxy in Apache to maintain proper resource isolation in CloudLinux, preventing edge traffic from exhausting concurrent process limits.
Validate backend SNI presentation with openssl s_client so strict CDN SSL settings do not fail on fallback certificates.

Tracking requests across every layer, from packet handling in the raw table to shared libraries loaded by the web server, reveals the root cause and prevents recurring downtime.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

806 reads

Creative Commons Attribution-NonCommercial

This post is licensed under CC BY-NC.