Incident anatomy: mitigating simultaneous L7 attacks, Dovecot DoS, and CPU bottlenecks

The day three fronts collapsed together: L7, Dovecot, and CPU bottlenecks#

In production servers hosting dozens of active applications, infrastructure failures rarely occur in isolation. I recently managed a cascading incident: multiple parallel attack vectors saturating host CPU, pending connections exhausting the mail service, and the security agent itself consuming over 56% of a CPU core to process the surge of incoming events.

Here is the step-by-step diagnostic process, common log-streaming traps, and the CLI commands used to contain malicious traffic.

1. Initial trigger: high process churn in /proc#

The initial alert reported severe CPU saturation. Checking top consumers with ps:

PID     PPID USER     %CPU COMMAND
2360882    1 root     56.9 python3
2455792  ... client1  50.0 lsphp
2451123  ... client2  46.9 lsphp
2455756  ... client3  40.5 lsphp
   5395    1 mysql    20.6 mariadbd

Multiple lsphp processes exhibited high utilization alongside a persistent python3 process consuming 56.9% CPU running under PPID 1 (systemd).

Attempting to inspect /proc/PID/environ for a specific PHP worker failed:

cat /proc/2417685/environ
# No such file or directory

Under high churn conditions, workers finish requests or get recycled by timeouts within milliseconds. To capture active scripts before process termination, I used a continuous low-latency sampling loop:

while true; do 
  pids=$(pgrep -u client1 lsphp)
  for pid in $pids; do 
    if [ -d /proc/$pid ]; then 
      strings /proc/$pid/environ | grep SCRIPT_FILENAME
    fi
  done
  sleep 0.5
done

2. First vector: distributed layer 7 attack on xmlrpc.php#

Inspecting processes matching common WordPress endpoints (ps faux | egrep 'xml|wp-cron' | sort) exposed the source of the workload:

client1  2622104 10.2  ...  \_ lsphp:2/domains/domain1.com.br/public_html/xmlrpc.php
client2  2672865 30.3  ...  \_ lsphp:/domain2.com.br/private_html/xmlrpc.php
client3  2664014 12.9  ...  \_ lsphp:ns/domain3.com.br/private_html/xmlrpc.php
client4  2673451 28.0  ...  \_ lsphp:domain4.com/public_html/xmlrpc.php

Processes ran in R<s state (Running, High Priority, Session Leader). A distributed botnet executed high-volume system.multicall requests against xmlrpc.php across multiple hosted domains simultaneously, bootstrapping the framework and generating continuous database queries.

Real-time tracking and immediate relief#

To log affected domains without blocking the terminal:

while true; do 
    ps faux | grep '[x]mlrpc.php' | awk '{print $NF}' | while read line; do 
        if ! grep -q "$line" xmlrpc.txt 2>/dev/null; then 
            echo "$(date '+%Y-%m-%d %H:%M:%S') - $line" >> xmlrpc.txt
        fi
    done
    sleep 1
done

To immediately terminate stuck workers:

ps aux | grep xmlrpc.php | awk '{print $2}' | xargs kill -9

3. Second vector: Dovecot dos and the stream pipe trap#

In the mail subsystem, maillog reported simultaneous connection anomalies:

dovecot: imap-login: Disconnected: Inactivity (auth failed, 1 attempts in 179 secs): user=<user1@domain.com.br>, rip=179.x.x.57
dovecot: auth-worker: shadow(user2,186.x.x.225): unknown user
dovecot: imap-login: Disconnected: Connection closed (no auth attempts in 0 secs): rip=64.15.129.109 ... TLS handshaking: Connection closed

Three abusive patterns were identified:

1. TLS exhaustion: Subnets opened TLS handshakes and aborted immediately, forcing high cryptographic CPU load without reaching authentication.
2. Inactivity descriptor locks: Connections held open for up to 179 seconds without traffic, exhausting socket tables.
3. Short username lookups: Queries lacking domain suffixes forced auth-worker into /etc/shadow lookups.

The continuous stream pipe trap#

Piping tail -f /var/log/maillog | sort | uniq freezes output because sort waits for EOF. For real-time deduplication on live streams, use associative arrays in awk:

tail -f /var/log/maillog | grep -E "auth-worker|login" | awk '!x[$0]++'

4. The 56.9% Imunify360 Python CPU spike#

High python3 usage was a direct byproduct of the traffic volume. The Imunify360 agent inspected every PHP execution via Proactive Defense while processing WAF queues. Under sustained flood conditions, the security agent began competing for CPU cycles with legitimate user processes.

5. User enumeration and googlebot preservation#

Access logs revealed the botnet enumerating authors using forged search engine referers (https://www.google.com):

GET /wp-json/wp/v2/users HTTP/1.1" 200 ... "https://www.google.com"
GET /?author=1 HTTP/1.1" 200 ... "https://www.google.com"
GET /wp-login.php HTTP/1.1" 200 ... "https://www.google.com"

Whitelisting legitimate googlebot ranges#

Extracting attacker IPs also surfaced official Googlebot ranges (66.249.64.0/19 and 2001:4860::/32). Blocking these would cause SEO deindexing.

To whitelist Googlebot officially in Imunify360:

imunify360-agent ip-list local add --ip 66.249.64.0/19 --purpose white --comment "Googlebot Official Range - Priority"
imunify360-agent ip-list local add --ip 2001:4860::/32 --purpose white --comment "Google IPv6 Range - Priority"
imunify360-agent whitelisted-crawlers add --name google

6. Mass IP mitigation pipeline#

With the sanitized list excluding Googlebot (grep -v), attacker IPs were committed to the local firewall blacklist:

grep -v "66.249." resultados.txt | awk '{print $1}' | sort -u | while read ip; do
    imunify360-agent ip-list local add --ip $ip --purpose black --comment "Botnet WP Scan / Bruteforce"
done

Technical takeaways from the incident#

1. Do not use sort on continuous streams: With tail -f, use awk '!x[$0]++' to deduplicate live output without waiting for EOF.
2. Monitor WAF resource costs: When security agents exceed 50% CPU inspecting Layer 7 traffic, shift mitigation to subnet-level network blocks.
3. Whitelist legitimate crawlers before batch banning: Always filter verified Googlebot ranges before executing bulk IP bans.

Frequently Asked Questions (FAQ)#

Why does piping tail -f into sort hang without printing output?#

sort buffers the entire input stream until it encounters an End-Of-File (EOF) signal before it can sort lines alphabetically. Because tail -f keeps the pipe open indefinitely, sort never outputs any data. Use awk '!x[$0]++' instead for real-time deduplication.

How do incomplete TLS handshakes cause a Denial of Service on Dovecot?#

Attackers establish TCP handshakes on IMAP/POP3 SSL ports (993/995) but deliberately withhold TLS negotiation packets. This traps login processes (imap-login) in pending states, exhausting Dovecot's max concurrent login connection slots and denying access to legitimate mail clients.

How can I prevent bulk WAF bans from blacklisting search engine crawlers?#

Botnets frequently forge Googlebot User-Agents and HTTP Referers. Before running bulk mitigation scripts, perform reverse DNS validation against official Google IP blocks (66.249.64.0/19 and 2001:4860::/32) and explicitly whitelist them in your WAF (imunify360-agent whitelisted-crawlers add --name google).

See also

Cybersecurity Troubleshooting Fail2ban status 255 exception: migrating to systemd backend and Nginx modular includes

Cybersecurity Debugging: resolving the "access denied by Imunify360 bot protection" block in automation workflows

Databases The day MySQL disappeared in production: low-level forensics and Laravel hardening

This post is licensed under CC BY-NC.