The ghost IP mystery: when CSF ignores whitelists, NAT, and connection state

The Ghost IP Mystery: When CSF Ignores Whitelists, NAT, and Connection State#

Recently, I faced one of the most frustrating scenarios for any infrastructure analyst: a client being systematically blocked by a cPanel server, even though their IP address was explicitly and correctly registered in both csf.allow and csf.ignore.

This type of incident defies basic firewall logic. If the IP is permitted, the packet should pass. However, in modern Linux environments, CSF is merely an abstraction layer over iptables. When you introduce NAT, complex virtualization, aggressive conntrack (connection tracking) settings, packet sanity filters, and hybrid backends toggling between iptables-legacy and nftables, the "obvious" answer often lies buried under layers of kernel logic.

The client reported a Connection Timed Out specifically when accessing the WHM/cPanel administration interfaces on ports:

2087
2083

The low-level response would be to whitelist the IP, restart the CSF daemon, and move on. But in this case, the IP was already whitelisted - both in the static allow list and the ignore list:

/etc/csf/csf.allow
/etc/csf/csf.ignore

This required a forensic transition: moving beyond the surface-level control panel and treating the issue as a problem of packet pathing, connection state integrity, and kernel chain evaluation order.

The symptom that broke the firewall's logic#

While auditing the lfd log, I found a strange log line:

/var/log/lfd.log: Mar 1 14:57:22 server lfd[4638]: Incoming IP 1.2.3.4 temporary allow removed

The key detail here is that the IP 1.2.3.4 shouldn't have depended on any temporary allow status. It was already in the permanent allow list. This caught my attention because lfd (Login Failure Daemon) is a Perl process that monitors logs and manages iptables chains dynamically through system calls.

When lfd removes a temporary allow, it can trigger a reload, partial flush, rule re-ordering, or rebuilding of chains. In normal environments, this shouldn't affect the permanent whitelist. However, in configurations with heavy custom chains, integrity filters, and INVALID state drops, a packet might get dropped before reaching the chain containing the whitelisted IPs.

This was the first turning point of the diagnosis: the question wasn't whether the IP was in the whitelist, but whether the packet ever reached that whitelist rule.

Initial validation with csf -g#

I ran the standard search command:

csf -g 1.2.3.4

The result showed the IP correctly associated with the ALLOWIN and ALLOWOUT chains. Practically, this is the equivalent of:

iptables -L -n | grep 1.2.3.4

But here lies a trap: csf -g only proves that the rule exists in the memory table. It does not prove that the packet actually reaches that rule. A firewall is a set of cascading conditions; if a "DROP" condition higher up the chain matches the packet first - such as a global stealth filter or a sanity check - the whitelist rule in the user-defined chain becomes unreachable.

Analyzing other iptables chains#

To understand the actual execution order of the rules, you need to list all chains along with their line numbers:

iptables -L -n --line-numbers

Check if there are global packet drops evaluated before the permission chains (like ALLOWIN). At the very top of the iptables INPUT chain, CSF's priority flow is inserted in the following typical order:

# List the first few rules of the INPUT chain
iptables -L INPUT -n -v --line-numbers | head -20

If there is a drop chain for invalid packet states (e.g. INVALID) or DoS protection evaluated before the jump to the ALLOWIN chain, the IP whitelist is simply ignored. Also, audit whether there are active custom chains that could be dropping traffic preemptively:

iptables -L -n | grep -E "^Chain (CUSTOM|WHITELIST|BLOCK)"

To inspect where the packet from IP 1.2.3.4 might be discarded before reaching the whitelist, verify the packet counters on the higher-level chains:

iptables -L INPUT -n -v | grep -B5 "1.2.3.4"

Route and NAT diagnosis#

The first networking layer check was to find how the kernel saw the return path routing for the client's IP:

ip route get 1.2.3.4

The output was similar to:

[root@server ~]# ip route get 1.2.3.4
1.2.3.4 via 192.168.x.1 dev eth0 src 192.168.x.124

This output points to an environment where the server responds using an internal IP as the source: src 192.168.x.124.

This is common in private clouds, data centers behind NAT, hosts behind internal routers, hypervisors, edge appliances, or setups where the public IP is not configured directly as the primary address of the network interface.

NAT and routing validation#

When client traffic passes through NAT (e.g., behind an edge firewall or a local router), the server's iptables address translation tables must be validated. Verify the active NAT rules to ensure there are no conflicts or unexpected source IP (SNAT) or destination IP (DNAT) rewrites:

# List rules in the nat table
iptables -t nat -L -n -v

# Check if SNAT/DNAT is active in postrouting
iptables -t nat -L POSTROUTING -n -v

Verify if the client's active IP session exists in the NAT connection pool and how the public address translates to the internal interface:

# Query active NAT connections for the IP
conntrack -L | grep "1.2.3.4"

# Validate the server's public egress IP vs local interface settings
curl ifconfig.me
ip addr show | grep "inet "

Reverse path filtering (rp_filter) and martian packets#

One of the most common causes of "invisible blocks" is RP Filtering. The kernel validates if the interface a packet arrived on is the same interface that would be used to reach that packet's source address according to the routing table.

If the routing table suggests a different path (perhaps due to a multi-homed setup or an internal VPN), the kernel suspects spoofing and drops the packet as a Martian Packet.

Verification of kernel parameters (sysctl)#

To audit the kernel's networking behavior and reverse path validations, list all relevant sysctl parameters:

# Validate rp_filter settings (0 = disabled, 1 = strict, 2 = loose)
sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.default.rp_filter
sysctl net.ipv4.conf.eth0.rp_filter

If rp_filter is set to 1 (strict) in an environment with multiple routes or asymmetric edge NAT, legitimate packets will be discarded before passing through the firewall's filtering layer. Additionally, verify other parameters essential for forwarding and security:

# Check forwarding and ICMP redirect configurations
sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.all.accept_redirects
sysctl net.ipv4.conf.all.send_redirects
sysctl net.ipv4.conf.all.accept_source_route

If the kernel discards the packet too early due to a reverse path violation, the CSF rules are never consulted. The packet dies before hitting the LOCALINPUT chain, before the ALLOWIN rule, or before appearing clearly in CSF logs.

The impact of the INVALID state#

Next, I investigated packet integrity and connection states.

CSF can enable filters that drop malformed packets, out-of-order packets, suspicious TCP flags, or packets flagged as invalid by the kernel connection tracker. The critical configuration is:

# In /etc/csf/csf.conf
PACKET_FILTER = "1"

With PACKET_FILTER enabled, CSF inserts rules at the very top of the INPUT chain using the state or conntrack modules:

-m state --state INVALID -j DROP

or:

-m conntrack --ctstate INVALID -j DROP

This is the crux of the issue. These drop rules are placed before the IP-specific whitelist rules. The firewall's design logic is to clean up junk traffic before processing access permissions.

Thus, even if an IP is listed in ALLOWIN, if the kernel classifies the incoming packet state as INVALID, it gets discarded before the whitelist rule can ever be processed:

Important: A whitelisted IP does not save a packet with an INVALID state if the INVALID drop rule is evaluated first.

This is highly common when clients are behind aggressive NAT systems, unstable links, CGNAT, egress load balancers, middleboxes that manipulate TCP headers, corporate firewalls rewriting sessions, or unstable mobile connections.

Conntrack saturation (nf_conntrack) and tracking exhaustion#

If the server's connection tracking table is full, the firewall's ability to identify "valid" sessions breaks down, causing legitimate packets to be marked as INVALID.

Verifying and fixing conntrack table overflow#

When the kernel connection tracker table saturates, new connections are flagged as INVALID and dropped.

Diagnosing Conntrack Overflow:

# Verify the total number of currently tracked connections
cat /proc/sys/net/netfilter/nf_conntrack_count

# Verify the maximum limit of the conntrack table
cat /proc/sys/net/netfilter/nf_conntrack_max
sysctl net.netfilter.nf_conntrack_max

If the value in nf_conntrack_count is close to or equal to nf_conntrack_max, the server is experiencing conntrack overflow. Check if the kernel log reports table overflows:

dmesg | grep -i "conntrack\|nf_conntrack"
grep -i "table full" /var/log/messages
grep -i "nf_conntrack: table full" /var/log/syslog

Resolving Conntrack Overflow:

If an overflow is detected, temporarily increase the conntrack limits at runtime:

echo 262144 > /proc/sys/net/netfilter/nf_conntrack_max

# Reduce the timeout for established connections to free up table slots faster
echo 600 > /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established

To make these conntrack limits permanent across reboots, add the directives to /etc/sysctl.conf and apply them:

echo "net.netfilter.nf_conntrack_max=262144" >> /etc/sysctl.conf
echo "net.netfilter.nf_conntrack_tcp_timeout_established=600" >> /etc/sysctl.conf
sysctl -p

Monitoring iptables counters directly#

To prove whether the packets were reaching the allow rules, I monitored iptables rule counters in real time:

watch -n 1 "iptables -L -n -v | grep 1.2.3.4"

This test is simple yet highly informative. If the client tries to connect and the packet counter on the whitelist rule does not increase, the packet is getting dropped earlier.

I also checked kernel logs:

dmesg | grep "1.2.3.4"

When there are martian source logs, invalid state messages, or subsystem drops, dmesg can reveal what CSF hides. CSF does not log everything; drops from PACKET_FILTER might not appear in /var/log/lfd.log. Relying solely on CSF logs is an operational blind spot.

Service-layer interrogation: Dovecot#

I also audited the upper layers of the stack to ensure the block wasn't occurring after the firewall:

When a client reports a block, do not stop at the firewall. Also validate that the traffic reaches the host:

ss -ntp | grep '1.2.3.4'

or:

netstat -atun | grep '1.2.3.4'

If you see ESTABLISHED sessions on ports like 993, 995, 143, 110, 2083, or 2087, the traffic is arriving. In this case, the problem may lie in service limits, authentication, application failures, Dovecot, cPHulk, or another component above the network layer.

For Dovecot, validate live sessions per IP:

doveadm who | grep '1.2.3.4'

And check the configuration limits:

dovecot -n | grep mail_max_userip_connections

In offices sharing a single NAT'd IP, low default values like 10 or 15 are insufficient. A single public IP may represent dozens of devices, each running desktop clients, mobile sync, webmail, and automatic reconnection daemons.

A typical configuration adjustment for NAT'd environments:

mail_max_userip_connections = 60

remote 127.0.0.1 {
  mail_max_userip_connections = 150
}

Then apply with:

dovecot -n
systemctl restart dovecot
systemctl status dovecot --no-pager

Critical services: DNS resolution and ssl/tls#

If the client remains unable to connect to the WHM/cPanel administrative ports (2083 and 2087), it is necessary to validate whether local SSL/TLS communication is functional and whether local name resolution is not creating conflicts.

1. DNS resolution check#

Validate whether the client's IP resolves correctly on the server and whether the hostname of the machine responds to local queries:

# Validate reverse DNS resolution for the client's IP
nslookup 1.2.3.4
dig -x 1.2.3.4 +short

# Verify IPs resolved by the local hostname
hostname -I

# Check for conflicting static overrides in hosts
grep "1.2.3.4" /etc/hosts

2. Ssl/tls connection check on administrative ports#

Verify if the local service is responding to encrypted connections on ports 2083 or 2087:

# Test local SSL connection on cPanel
openssl s_client -connect localhost:2083 </dev/null 2>/dev/null | grep -i "ssl"

# Verify validity of the host's SSL certificate
openssl x509 -in /etc/ssl/certs/hostname.crt -noout -dates 2>/dev/null || echo "Certificate missing at default path."

Search for SSL handshake errors in Apache's error logs:

grep -i "ssl\|tls" /usr/local/apache/logs/error_log | tail -10

Cphulk and application-layer blocking#

I also audited cPHulk because it can block login attempts to WHM/cPanel without it looking like a traditional firewall drop.

In incidents involving ports 2087 and 2083, consider:

• Blocks due to failed logins;
• Brute-force bans;
• Users repeating an old password;
• Automated services attempting authentication;
• Entire offices sharing a single NAT public IP.

To ensure that cPHulk is not causing silent blocks for the IP on cPanel ports:

/usr/local/cpanel/scripts/cphulkdwhitelist 1.2.3.4

Advanced SELinux and AppArmor triage#

In hardened distributions, Linux mandatory access control (MAC) modules, such as SELinux and AppArmor, can deny essential system calls (syscalls) like recvmsg() or block Apache and Dovecot from binding to or accessing specific network ports.

1. SELinux auditing#

Check the state and violations of SELinux:

# Check current mode
getenforce

# Search for recent SELinux AVC denials
ausearch -m avc -ts recent | head -20

If AVC denials are present, make sure Apache's network booleans are enabled:

getsebool -a | grep -i "httpd\|network"

Verify the security context of the rules files and Apache:

ls -Z /etc/csf/
ls -Z /usr/local/apache/

2. AppArmor auditing#

If using Ubuntu or Debian-based systems running AppArmor, check for profiles enforcing restrictions on the relevant daemons:

aa-status

Legacy iptables, nftables, and CSF obsolescence#

CSF is a set of Perl scripts managing iptables rules, often using iptables-legacy wrappers.

On modern distributions (AlmaLinux 8/9, Rocky Linux 8/9, newer Debian/Ubuntu), the native firewall engine is nftables. The translation layer:

CSF -> iptables wrapper -> iptables-nft or iptables-legacy -> nftables/kernel

can introduce subtle behavior mismatches in high-load scenarios, NAT setups, or complex tracking states. In some environments, replacing automated blockers with modern native structures is the most robust fix.

Pre-disable backup protocol#

If it becomes necessary to disable CSF for migration or deep debugging, ensure operational integrity by first taking a complete backup of all CSF and iptables configurations:

# Backup CSF configurations
tar czf /root/csf-backup-$(date +%Y%m%d).tar.gz /etc/csf/

# Backup current iptables rules to disk
iptables-save > /root/iptables-backup-$(date +%Y%m%d).rules

# Save a friendly listing of CSF rules
csf -l > /root/csf-rules-before.txt

Once the backups are saved, temporarily disable CSF for isolation:

csf -x

Hardware and network interface auditing#

At the physical infrastructure level, packet loss or intermittent disconnections can be caused by network card failures, MTU mismatches, or physical link issues.

Audit the error statistics of the server's physical network cards:

# Show network interface status
ip link show

# Show detailed transmission and error statistics on the interface
ip -s link show eth0

# Check direct discard (drops) counters in the kernel
cat /proc/net/dev

Validate speed, duplex mode, and physical link integrity using ethtool:

ethtool eth0 | grep -i "speed\|duplex\|link"

CSF rollback plan#

After completing your diagnosis or migration, revert the settings to their original state and re-enable CSF to restore server security:

# Re-enable the CSF engine
csf -e

# If necessary, restore the previous configuration from backup
cp /root/csf-backup-*/csf.conf /etc/csf/csf.conf
csf -r

# Verify the current status of CSF
csf -s

# Validate if the client's IP is in CSF's active allow list
csf -g 1.2.3.4

Post-change production monitoring#

After disabling CSF or changing conntrack and network rules, continuously monitor the client IP's traffic and server connections for at least 15 to 30 minutes:

# Monitor system authentication logs
tail -f /var/log/secure | grep "1.2.3.4"

# Monitor HTTP access on Apache/cPanel administrative ports
tail -f /usr/local/apache/logs/access.log | grep "1.2.3.4"

# Monitor activity logs of the LFD daemon (if kept active)
tail -f /var/log/lfd.log | grep "1.2.3.4"

# Observe general packet drops in real time
watch -n 1 "iptables -L -n -v | grep DROP"

Checklist: IP blocked with "clean" firewall#

Follow this structured checklist to triage and fix ghost connection issues:

1. CSF verification#

• [ ] csf -g IP - check if the IP is listed in the active allow chains.
• [ ] grep IP /etc/csf/csf.allow - confirm the static entry in the whitelist.
• [ ] grep IP /etc/csf/csf.ignore - confirm the entry in the ignore list.
• [ ] grep "IP" /var/log/lfd.log - audit logs for brute-force detections or temporary blocks.

2. Route and network verification#

• [ ] ip route get IP - validate the return route and origin IP address (src).
• [ ] Verify physical link integrity and network errors with ip -s link and ethtool.
• [ ] Analyze NAT rules with iptables -t nat -L.

3. Tracking and conntrack#

• [ ] Compare nf_conntrack_count against nf_conntrack_max to identify saturation.
• [ ] Check for "table full" messages in kernel logs with dmesg.
• [ ] Adjust conntrack timeout values to prevent excessive session retention.

4. Rp_filter and kernel policies#

• [ ] Verify if rp_filter is active on the network interfaces.
• [ ] Validate accept_redirects and accept_source_route parameters.

5. iptables chain priority#

• [ ] List rules with iptables -L -n --line-numbers.
• [ ] Verify if PACKET_FILTER rules or INVALID state drops are evaluated above the ALLOWIN chain.

6. Counter monitoring#

• [ ] Run real-time counter monitoring to check if traffic hits the whitelist.

7. Service and application audit#

• [ ] Validate logins blocked in the administrative layer via cPHulk.
• [ ] Audit Dovecot connections via doveadm who and limits in mail_max_userip_connections.
• [ ] Test local SSL/TLS connections on ports 2083 and 2087 using openssl.

8. Mandatory access control (MAC)#

• [ ] Check SELinux AVC Denials and AppArmor status.

9. Operational decision#

• [ ] Evaluate if CSF should be maintained or if migrating to native nftables/iptables is the best choice for stability.

Comparative problems and severity table#

See also

Infrastructure IP blocked on the server? How to identify the causing email account via WHM and cPanel

Infrastructure Debugging pure-ftpd stuck waiting for welcome message on cpanel/ovh

Infrastructure How to bypass the global email limit in cPanel and CloudLinux: resolving error XID psafa9

This post is licensed under CC BY-NC.