Deployment Troubleshooting: Port Conflicts, Config Automation, and Safe Validation

In web application deployments, two incidents appear frequently: a previous service still hogging the port and configuration replacements performed manually and insecurely. The result is downtime, human error, and improvised rollbacks. This guide is the workflow I apply to solve these in a standardized way.

Incident Scenario

Error received when starting Node.js:

Error: listen EADDRINUSE: address already in use :::3000

This error indicates that the port already has a process listening. In environments with frequent restarts, this is usually an orphaned process, a duplicate instance of PM2/systemd, or an old container.

Step 1: Precise Port and Environment Diagnosis

1.1 Verifying Port Availability and Listen States

Before initiating any new app daemon, verify if the destination port is free. Doing so prevents start-up conflicts and race conditions:

# Validate port availability via ss
if ss -lntp | grep -q ":3000"; then
    echo "ERROR: Port 3000 is already in use."
    ss -lntp | grep ":3000"
fi

# Or using fuser
if fuser 3000/tcp >/dev/null 2>&1; then
    echo "ERROR: Port 3000 is bound by PID: $(fuser 3000/tcp 2>&1)"
fi

1.2 Identifying Sockets and PIDs

If the port is occupied, locate the active PID binding the socket before sending terminal signals:

# List listening TCP ports
ss -lntp | grep ':3000'

# Fallback using lsof — always use sudo for full process visibility
sudo lsof -i :3000

Trace the origin and execution line of the PID using ps:

ps -fp <PID>

This avoids bringing down vital processes on shared or multi-tenant hosts.

1.3 Identifying Orphaned or Zombie Processes

Orphaned processes (processes running with PPID 1 that are not standard system init/systemd services) or zombie processes (dead processes maintaining kernel system structures) can hold sockets. Inspect these cases:

# Trace orphaned processes (excluding system services)
ps -eo pid,ppid,comm | awk '$2 == 1 && $3 != "systemd" && $3 != "init" && $3 != "sshd"'

# Find processes in Z (Zombie) status
ps -eo pid,stat,comm | grep -w Z

1.4 Checking Firewall Barriers

If no local process binds to the port, but external connection attempts still drop, check if host firewall rules block the path:

# Check UFW rules (Debian/Ubuntu)
sudo ufw status | grep 3000

# Inspect iptables rules with interface details (-v flag)
sudo iptables -L -n -v | grep 3000

# Open target port if needed (UFW)
sudo ufw allow 3000/tcp

Step 2: Controlled Port Conflict Resolution

2.1 Graceful Shutdown Workflow (`kill -15` vs `kill -9`)

To release an occupied port, never jump straight to force signals. Follow a structured shutdown sequence instead:

# 1. Send SIGTERM (15) to request a graceful shutdown
kill -15 <PID>

# 2. Pause to allow the process to clean up RAM, flush logs, and release sockets
sleep 5

# 3. Check process existence and force SIGKILL (9) only if necessary
kill -0 <PID> 2>/dev/null && kill -9 <PID> 2>/dev/null || true

2.2 Safe Usage of Socket Clearing Utilities

Avoid running the blind command fuser -k 3000/tcp. The -k flag kills every process associated with the TCP socket, which might include auxiliary threads or neighboring applications. On shared systems, this can cause collateral damage.

Instead, collect PIDs dynamically and require human validation:

# Identify port-associated PIDs
PORT_PIDS=$(lsof -t -i:3000)

if [ -n "$PORT_PIDS" ]; then
    for PID in $PORT_PIDS; do
        echo "Process on port 3000: $(ps -p $PID -o comm=) (PID: $PID)"
        read -p "Would you like to terminate this process gracefully? (y/N): " CONFIRM
        if [ "$CONFIRM" = "y" ]; then
            kill -15 $PID
            sleep 2
            kill -0 $PID 2>/dev/null && kill -9 $PID || true
        fi
    done
else
    echo "No process binding to port 3000."
fi

Finally, confirm the socket is clear:

ss -lntp | grep ':3000' || echo 'port 3000 free'

Step 3: Prevent Recurrence with a Process Manager

Do not use node app.js & in production. Use pm2 or systemd.

Example with PM2:

pm2 start ecosystem.config.js --env production
pm2 save
pm2 startup

Example with systemd (summary):

[Service]
ExecStart=/usr/bin/node /opt/app/server.js
Restart=always
RestartSec=3

Step 4: Safe Configuration Automation

During migrations, it's common to change prefixes or users in wp-config.php or .env. A classic error is using single quotes, which prevents variable expansion in shell scripts.

Wrong (does not expand variables):

sed -i 's/$OLD/$NEW/g' wp-config.php

Correct (uses double quotes for expansion):

sed -i "s/$OLD/$NEW/g" wp-config.php

4.1 Mandatory Change Pipeline Sequence

To prevent downtime and file corruption in production environments, follow this mandatory pipeline sequence:

flowchart LR A["📦 1. Create\nBackup"] --> B["🔒 2. Escape\nVariables"] B --> C["👁️ 3. Dry-run\n(stdout)"] C --> D["✏️ 4. Apply\nwith sed"] D --> E{"✅ Validate\nSyntax & DB"} E -->|"OK"| F["📝 Audit\nLog"] E -->|"❌ Fail"| G["🔄 Auto\nRollback"] style G fill:#7f1d1d,color:#fca5a5 style F fill:#14532d,color:#86efac

Create Backup: Save a timestamped backup before touching the file.
Escape Variables: Sanitize inputs to prevent conflicts with delimiters.
Validate Dry-run: Test the replacements without modifying the file.
Apply with sed: Run replacements in-place using alternative delimiters.
Validate Syntax & DB: Test file syntax and database connection.

4.2 Hardened Script Implementation with Error Handling

If $OLD_VAL or $NEW_VAL variables are empty or contain special characters like backslashes (\), forward slashes (/), or the replacement operator (&), sed will fail or corrupt files.

Also, use alternative delimiters (such as |) in your sed command to prevent collisions with directory paths.

# 1. Locate config file and extract target strings
CONFIG_FILE="wp-config.php"
OLD_VAL=$(grep DB_NAME "$CONFIG_FILE" | cut -d"'" -f4 | cut -d_ -f1)
NEW_VAL=$(pwd | cut -d/ -f3)

# 2. Check for empty variables
if [ -z "$OLD_VAL" ] || [ -z "$NEW_VAL" ]; then
    echo "ERROR: Configuration variables OLD_VAL or NEW_VAL are empty. Aborting."
    exit 1
fi

# 3. Escape special characters for sed pattern safety
OLD_ESCAPED=$(printf '%s\n' "$OLD_VAL" | sed 's/[&/\]/\\&/g')
NEW_ESCAPED=$(printf '%s\n' "$NEW_VAL" | sed 's/[&/\]/\\&/g')

# 4. CREATE MANDATORY BACKUP FIRST
BACKUP_FILE="${CONFIG_FILE}.bak.$(date +%F-%H%M%S)"
cp "$CONFIG_FILE" "$BACKUP_FILE"
echo "✅ Security backup created: $BACKUP_FILE"

# 5. RUN DRY-RUN TO VALIDATE CHANGES (Outputs to stdout)
echo "=== DRY-RUN OUTPUT ==="
sed "s|${OLD_ESCAPED}|${NEW_ESCAPED}|g" "$CONFIG_FILE" | head -n 25
echo "======================"

# 6. APPLY CHANGES IN-PLACE AFTER DRY-RUN
sed -i "s|${OLD_ESCAPED}|${NEW_ESCAPED}|g" "$CONFIG_FILE"

# 7. RUN-TIME VALIDATIONS
# Validate syntax
if ! php -l "$CONFIG_FILE" >/dev/null 2>&1; then
    echo "❌ ERROR: Invalid PHP syntax on modified config! Starting auto-rollback..."
    cp "$BACKUP_FILE" "$CONFIG_FILE"
    exit 1
fi

# Validate DB connectivity (if WP-CLI is active)
if command -v wp &>/dev/null; then
    if ! wp db check >/dev/null 2>&1; then
        echo "❌ ERROR: Database connection failed! Starting auto-rollback..."
        cp "$BACKUP_FILE" "$CONFIG_FILE"
        exit 1
    fi
else
    # Native PHP connection check fallback
    # Uses realpath for absolute path to avoid CWD-dependent require failures
    ABS_CONFIG=$(realpath "$CONFIG_FILE")
    if ! php -r "require '${ABS_CONFIG}'; \$conn = new mysqli(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME); if (\$conn->connect_error) { exit(1); }" > /dev/null 2>&1; then
        echo "❌ ERROR: PHP MySQL connection test failed! Starting auto-rollback..."
        cp "$BACKUP_FILE" "$CONFIG_FILE"
        exit 1
    fi
fi

# 8. AUDIT LOGGING
# chmod 640 + group adm: restricts log reads to root and adm group members.
# Prevents exposing executor identity (whoami) and file paths to unprivileged users.
LOG_FILE="/var/log/deploy-changes.log"
sudo touch "$LOG_FILE" && sudo chown root:adm "$LOG_FILE" && sudo chmod 640 "$LOG_FILE" || true
{
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Modification performed by: $(whoami)"
    echo "  File:   $CONFIG_FILE"
    echo "  Old:    $OLD_VAL"
    echo "  New:    $NEW_VAL"
    echo "  Status: Validated & Applied"
} >> "$LOG_FILE" 2>/dev/null || true

echo "✅ Configuration changes successfully applied and validated."

4.3 Auditing Configurations via Auditd

To monitor alterations to sensitive production files, register a watch rule in the Linux audit subsystem:

# Monitor write operations and attribute changes on target configurations
sudo auditctl -w /var/www/html/wp-config.php -p wa -k config_change

Fast Manual Rollback

If the configuration changes fail:

cp wp-config.php.bak.YYYY-MM-DD-HHMM wp-config.php
systemctl restart php-fpm

5) Reference Tables and Safety Templates

Safe Deployment Script Template (`deploy-safe.sh`)

Create this script inside your project workspace as a wrapper for updates:

#!/bin/bash
# deploy-safe.sh - Safe deployments with port clearance and config validations
set -euo pipefail

PORT="${1:-3000}"
CONFIG_FILE="${2:-wp-config.php}"

echo "=== Deploying Safe Workflow ==="

# 1. Create Timestamped Backup
BACKUP="${CONFIG_FILE}.bak.$(date +%F-%H%M%S)"
cp "$CONFIG_FILE" "$BACKUP"
echo "✅ Security backup generated: $BACKUP"

# 2. Check Port and Handle Active Processes
if ss -lntp | grep -q ":${PORT}"; then
    echo "⚠️ Port $PORT is occupied:"
    ss -lntp | grep ":${PORT}"
    read -p "Would you like to terminate active socket processes? (y/N): " CONFIRM
    if [ "$CONFIRM" = "y" ]; then
        PORT_PIDS=$(lsof -t -i:${PORT})
        if [ -n "$PORT_PIDS" ]; then
            kill -15 $PORT_PIDS 2>/dev/null || true
            sleep 3
            kill -0 $PORT_PIDS 2>/dev/null && kill -9 $PORT_PIDS 2>/dev/null || true
        fi
    else
        echo "Aborted by operator."
        exit 1
    fi
fi

echo "✅ Port $PORT is free and ready."

# 3. Perform replacements (escaped configurations)
# sed -i "s|${OLD_ESCAPED}|${NEW_ESCAPED}|g" "$CONFIG_FILE"

# 4. Validate syntax
if ! php -l "$CONFIG_FILE" >/dev/null 2>&1; then
    echo "❌ ERROR: Syntax validation failed."
    cp "$BACKUP" "$CONFIG_FILE"
    exit 1
fi

# 5. Check database connection
if command -v wp &>/dev/null; then
    if ! wp db check >/dev/null 2>&1; then
        echo "❌ ERROR: Database check failed."
        cp "$BACKUP" "$CONFIG_FILE"
        exit 1
    fi
fi

echo "✅ Configuration validation successful. Resuming service..."
# systemctl restart app
echo "✅ Deployment finished successfully."

Linux Signals Reference for Process Management

Signal	Code	Name	Intended Usage
SIGHUP	`1`	Hangup	Reload daemon configurations without terminating execution.
SIGINT	`2`	Interrupt	Stop processes from control terminal (Ctrl+C).
SIGKILL	`9`	Kill	Immediate, uncatchable termination. Last resort.
SIGTERM	`15`	Terminate	Graceful shutdown request. Recommended first step.

Network Diagnostics Utilities

Utility Command	Purpose	Example
`ss -lntp`	List active listening sockets and mapping PIDs	`ss -lntp \	grep :3000`
`lsof -i :PORT`	Display process details currently running on a port	`lsof -i :3000`
`fuser PORT/tcp`	Returns the PIDs binding to a TCP port	`fuser 3000/tcp`
`netstat -tulpn`	List sockets (legacy interface)	`netstat -tulpn`

Deployment Validation Checklist

[ ] Security backup generated prior to modifications.
[ ] Variables validated to ensure they are not empty.
[ ] Special character symbols escaped for sed parsing.
[ ] Dry-run executions verified in stdout.
[ ] File syntax validated post-modifications.
[ ] Target network socket ports verified for availability.
[ ] Firewall parameters verified and open.
[ ] Active service processes gracefully terminated (SIGTERM).
[ ] Automated rollbacks validated in case of failures.
[ ] Change logs registered in centralized log directories.

Conclusion

Stable deployment depends on method, not trial and error. In my workflow, I always follow: identifying the correct process, releasing the port with control, automating replacements with dry-runs, validating syntax, and keeping a rollback ready. This standard has reduced repeat incidents and standardized delivery across the team.

This post is licensed under CC BY-NC.