Crisis management and hardening: anatomy of a Linux incident response

In this incident, a routine audit in a development environment exposed behavior consistent with RCE exploitation through an outdated dependency (React2Shell), followed by persistence and privilege escalation attempts.

This section documents the exact operational sequence used to recover safely.

1) Immediate detection and containment

Initial signs:

• anomalies around system inspection commands (ps, top)
• suspicious startup shell artifacts and executable files in /tmp

Initial triage commands:

ps auxf
top -b -n 1 | head -n 40
ss -lntup
journalctl -xe --no-pager | tail -n 200

Containment actions:

1. block non-essential outbound/inbound network paths
2. freeze deployment activity for affected project
3. preserve incident evidence before cleanup

Persistence indicators were found in shell initialization paths and temporary execution vectors.

2) Rescue-mode forensic recovery

To avoid executing compromised runtime state, recovery was performed in rescue mode with passive disk access.

Execution model:

1. boot into rescue environment
2. mount affected disk read-only for initial analysis
3. export trusted assets only
4. review and sanitize SSH trust paths

Representative commands:

lsblk
mount /dev/vda2 /mnt/sysroot
mount -o remount,ro /mnt/sysroot

Backup policy applied:

• include source code and database dumps
• exclude rebuildable artifacts (node_modules, caches, transient binaries)

SSH trust audit:

cat /mnt/sysroot/root/.ssh/authorized_keys
cat /mnt/sysroot/home/*/.ssh/authorized_keys

Unauthorized key material was removed and documented.

3) Clean-slate rebuild decision

Because escalation attempts targeted privileged paths, partial cleanup was rejected in favor of full OS rebuild.

Technical reason: script cleanup alone cannot guarantee integrity of shared libraries, low-level binaries, or hidden persistence.

4) Post-incident hardening on Rocky Linux

A) Privilege segregation

Dedicated non-root runtime user:

sudo adduser --system --group --home /home/domain_user domain_user

B) Temporary filesystem controls

Enforced noexec and nosuid for /tmp:

echo "tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev 0 0" >> /etc/fstab
mount -o remount /tmp

C) SSH hardening

Key controls in sshd_config:

• disable password authentication
• disable root login
• custom port and access restrictions

Validation:

sshd -t && systemctl restart sshd

D) Network perimeter hardening

Restrictive inbound policy with explicit allowlist:

firewall-cmd --permanent --set-default-zone=drop
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --reload

E) Logical data isolation

Application data retained under /home, separated from system root:

mkdir /home/www && ln -s /home/www /www

5) Runtime restoration and controlled deploy

After rebuild, runtime dependencies were restored with deterministic workflow, including package gaps typical in RHEL-like distributions.

Process stack:

• Bun for runtime/build flow
• PM2 for process supervision and auto-restart

Validation commands:

pm2 status
pm2 logs --lines 100
systemctl status sshd firewalld

6) Final operational checklist

• [x] network containment and change freeze
• [x] evidence-first forensic triage
• [x] rescue-mode selective recovery
• [x] full host rebuild
• [x] hardening baseline across access, filesystem, and network
• [x] monitored service restoration

In escalation-suspected incidents, disciplined phased response is more important than speed alone. The combination of containment, evidence-preserving recovery, clean rebuild, and hardening delivered a stable and auditable return to service.

This post is licensed under CC BY-NC.