The day Rocky Linux 8 kernel "broke" my iptables (and how i fixed it)

If you work with Linux infrastructure, you know that a kernel update should be routine. But sometimes a security plugin or an automatic update decides to install a new kernel, and suddenly your firewall stops talking to it. This exactly happened to me recently on a Rocky Linux 8.10 server.

I'll share the struggle and the technical solution so you don't lose the same amount of time I did.

1. Initial diagnostics: the kernel incompatibility error#

It all started when I needed to add a simple NAT rule for a Docker container:

# The command that should have worked:
/usr/sbin/iptables -w10 -t nat -A DOCKER -p tcp --dport 38888 -j DNAT --to-destination 192.168.x.x:8888 ! -i br-generic

The response was a punch to the gut: iptables: No chain/target/match by that name.

I tried listing the rules, and the error changed to something even more concerning: iptables: Incompatible with this kernel.

Running uname -r, I saw I was on kernel 4.18.0-553.97.1.el8_10.x86_64. It was a very recent version that an anti-intrusion plugin had forced onto the system.

Rocky Linux 8 (like RHEL 8) uses nftables as the backend. The iptables binary is just a compatibility layer. When the kernel updates and this layer "breaks," it's usually because:

1. The kernel and networking tools are out of sync.
2. The NAT modules (iptable_nat, nf_nat) were not installed for this specific kernel version.

In my case, when trying to load the module manually: modprobe: FATAL: Module nft_chain_nat_ipv4 not found in directory /lib/modules/4.18...

Bingo! The new kernel was there, but the "muscles" (the modules, specifically the -modules-extra package) were not installed. The kernel was "crippled."

2. DNF history auditing#

Before applying any fix, it is essential to check the package manager's history to see exactly which update caused the missing modules:

# 1. Check dnf transaction history
dnf history list

# 2. Check details of the latest kernel transaction (replace <ID> with the transaction ID)
dnf history info <ID>

# 3. Check currently installed kernel modules packages
rpm -qa | grep kernel-modules

3. Preventive iptables rules backup#

Before installing packages and reloading kernel modules, perform a complete backup of the active firewall rules to prevent configuration loss:

# 1. Back up active IPv4 rules
sudo iptables-save > /root/iptables-backup-$(date +%Y%m%d).rules

# 2. Back up active IPv6 rules
sudo ip6tables-save > /root/ip6tables-backup-$(date +%Y%m%d).rules

# 3. If you need to restore rules:
# sudo iptables-restore < /root/iptables-backup-$(date +%Y%m%d).rules

4. Checking available and loaded modules#

Verify if the extra modules packages are available in repositories for the active kernel, and check which modules are loaded:

# 1. List available extra modules for the active kernel version
dnf list available kernel-modules-$(uname -r) kernel-modules-extra-$(uname -r)

# 2. Check currently loaded networking modules
lsmod | grep -E "iptable|nf_nat|br_netfilter|ip_tables"

# 3. Verify if the module file exists physically under modules directory
ls /lib/modules/$(uname -r)/kernel/net/netfilter/ | grep nat

5. The solution: installing extra modules and fixing the kernel#

If rebooting to an older kernel fails (as detailed in Section 10), the only solution is to install the missing modules directly for the active running kernel.

# Step 1: Install the modules and extra modules packages
dnf install kernel-modules-$(uname -r) kernel-modules-extra-$(uname -r) -y

# Step 2: Force load the NAT modules in real-time
modprobe iptable_nat
modprobe nf_nat
modprobe br_netfilter

6. Post-load module verification#

After loading the modules manually, verify that the kernel has successfully resumed communication with the xtables/iptables binary:

# 1. Verify that the modules appear as loaded
lsmod | grep -E "iptable_nat|nf_nat|br_netfilter"

# 2. Run a simple listing command to check if the incompatibility error is gone
iptables -L -n

# 3. Verify Docker specific forwarding chains
iptables -t nat -L DOCKER -n

# 4. Check kernel ring buffer logs for netfilter warnings
dmesg | grep -iE "nf_nat|iptable" | tail -n 10

7. Persistent module loading on boot#

To ensure that the required modules remain loaded after system reboots, create a persistent load configuration file in systemd modules-load directory:

# 1. Create the configuration file with the required modules
echo "iptable_nat" | sudo tee /etc/modules-load.d/iptables.conf
echo "nf_nat" | sudo tee -a /etc/modules-load.d/iptables.conf
echo "br_netfilter" | sudo tee -a /etc/modules-load.d/iptables.conf

# 2. Test if systemd can load these configurations
sudo systemd-modules-load --test

8. Firewalld status and verification#

On Rocky Linux, the firewalld daemon can conflict with manual rules written in iptables or with Docker automatic redirects.

# 1. Verify active firewalld status
systemctl status firewalld

Resolution options:#

• Option A: Disable firewalld (Recommended for servers managing rules directly in iptables/Docker)

    sudo systemctl stop firewalld
    sudo systemctl disable firewalld

• Option B: Map ports using Firewalld commands (Alternative to raw iptables rules)

    sudo firewall-cmd --zone=public --add-port=38888/tcp --permanent
    sudo firewall-cmd --reload

9. SELinux restriction checks#

Tight SELinux security policies can prevent the iptables binary from interacting with the kernel hooks.

# 1. Verify SELinux active mode
getenforce

# 2. If Enforcing, test permissive mode temporarily
sudo setenforce 0

# 3. Test iptables rules. If the error is gone, SELinux is the culprit.
# Restore SELinux mode immediately:
sudo setenforce 1

# 4. Check audit logs for iptables denials
sudo ausearch -m avc -ts recent | grep iptables

10. Docker and host network verification#

After modules are installed and loaded, ensure that the Docker bridge networks are working properly:

# 1. Restart Docker to recreate network NAT chains
sudo systemctl restart docker

# 2. Verify Docker daemon status
systemctl status docker

# 3. List active Docker networks and bridges
docker network ls
ip link show type bridge

# 4. Verify host interfaces and routing tables
ip addr show
ip route show

11. Kernel rollback guide using grubby#

If the module fixes fail, the best temporary production workaround is rolling back to the previous stable kernel. Rocky Linux uses the grubby utility to manage boot configurations.

# 1. List all available kernel entries and indices
grubby --info=ALL | grep -E "index|kernel"

# 2. Set the older kernel as default using its absolute path
grubby --set-default=/boot/vmlinuz-4.18.0-553.80.1.el8_10.x86_64

# 3. Alternatively, set the default entry by index
grubby --set-default-index=0

# 4. Verify that the default kernel has been updated
grubby --default-kernel

Note: In cloud environments or VPS setups, if the default kernel configuration does not persist across reboots, audit the network boot settings or check if the GRUB is overridden by the host hypervisor.

12. Version locking prevention#

To prevent system updates from upgrading the kernel automatically without modules-extra:

# 1. Install the versionlock plugin
dnf install 'dnf-command(versionlock)' -y

# 2. Lock the current working kernel version
dnf versionlock add kernel-4.18.0-553.80.1.el8_10.x86_64

After all this, the redirection rule went through on the first try:

iptables -w10 -t nat -A DOCKER -p tcp --dport 38888 ! -i br-0f2060aef134 -j DNAT --to-destination 192.168.192.3:8888

If you ever run into this, don't despair at the "Incompatible kernel" error. Before formatting or fighting with GRUB, check if the networking modules are actually installed for the version you are running.

Network and iptables diagnostic script (diagnose-iptables.sh)#

This diagnostic shell script automatically maps and audits all kernel levels, loaded modules, and firewalls on Rocky Linux 8:

#!/bin/bash
# diagnose-iptables.sh
# Diagnostic script for Iptables errors on Rocky Linux 8.
# Must be executed as root.

set -euo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0;37m'

log_info() { echo -e "[${GREEN}INFO${NC}] $1"; }
log_warn() { echo -e "[${YELLOW}WARN${NC}] $1"; }
log_error() { echo -e "[${RED}ERROR${NC}] $1"; }

# 1. Verify root privileges
if [ "$EUID" -ne 0 ]; then
    log_error "This script must be executed as root."
    exit 1
fi

log_info "=== Starting Iptables Diagnostics on Rocky 8 ==="
log_info "Active Kernel version: $(uname -r)"

# 2. List Installed Modules Packages
log_info "--- Installed Kernel Modules Packages ---"
rpm -qa | grep kernel-modules || log_error "No kernel-modules packages found."

# 3. Check Loaded Modules Status
log_info "--- Network Modules Audit ---"
for mod in iptable_nat nf_nat br_netfilter ip_tables; do
    if lsmod | grep -q "$mod"; then
        log_info "  ✅ Module $mod: loaded in memory."
    else
        log_warn "  ❌ Module $mod: NOT loaded in memory."
    fi
done

# 4. Check Iptables Binary Health
log_info "--- Iptables Binary Execution Test ---"
if iptables -L -n >/dev/null 2>&1; then
    log_info "  ✅ Iptables binary is responding normally."
else
    log_error "  ❌ Iptables binary failed or returned incompatibility with the kernel."
fi

# 5. Check NAT and Docker chains
log_info "--- NAT and Docker Chains Check ---"
if iptables -t nat -L >/dev/null 2>&1; then
    log_info "  ✅ NAT Table is available."
    if iptables -t nat -L DOCKER -n >/dev/null 2>&1; then
        log_info "  ✅ DOCKER NAT chain exists."
    else
        log_warn "  ❌ DOCKER NAT chain is missing. Docker containers might be unreachable."
    fi
else
    log_error "  ❌ NAT Table is unavailable."
fi

# 6. Check Firewalld Service
log_info "--- Firewalld Status ---"
if systemctl is-active firewalld >/dev/null 2>&1; then
    log_warn "  ⚠️ Firewalld is active. It might conflict with native raw iptables/Docker rules."
else
    log_info "  ✅ Firewalld is inactive."
fi

# 7. Check SELinux Status
log_info "--- SELinux Status ---"
if command -v getenforce >/dev/null 2>&1; then
    log_info "  SELinux Mode: $(getenforce)"
else
    log_info "  SELinux is not installed."
fi

# 8. Gather Recent Logs
log_info "--- Recent dmesg lines related to netfilter ---"
dmesg | grep -iE "nf_nat|iptable|netfilter" | tail -n 5 || echo "No netfilter related lines in dmesg."

log_info "=== Diagnostics Completed ==="

See also

Infrastructure From LSM to Netfilter in OCI: Demystifying SELinux and Opening Ports on Ubuntu Minimal 24.04

Servers My VPS crashed: why UFW firewall logs don't mean a DDoS attack (and how i found the real cause)

Infrastructure Anatomy of a silent deadlock in CloudLinux and cPanel: phantom processes, session locks, and TCP queues

This post is licensed under CC BY-NC.