The day MySQL disappeared in production: low-level forensics and Laravel hardening

Few operational moments cause as much stress as realizing that entire MySQL database tables have vanished from a production host.

The stack consisted of Laravel, an aaPanel management layer on Linux, and several active subdomains. When databases disappear unexpectedly, initial suspicions often point toward ransomware or external network breaches. In critical incidents, however, a structured forensic approach is key: auditing listening sockets, analyzing binary logs (binlogs), sniffing loopback traffic, and implementing permanent safeguards.

Here is the step-by-step investigation, disk space recovery, and security hardening implemented to prevent recurrence.

1. Initial triage and network containment#

The immediate question was whether the table drops originated externally (open port 3306) or internally (local application worker execution).

Auditing listening sockets with netstat:

netstat -tunap | grep 3306

Terminal output:

tcp6       0      0 :::3306                 :::*                    LISTEN      104467/mysqld

MySQL was bound to global IPv6 (:::3306). In Linux, when net.ipv6.bindv6only is disabled, IPv6 sockets also accept mapped IPv4 connections. If system firewall rules omit explicit IPv6 filters, the database is exposed to external port scans.

The immediate fix was restricting the bind exclusively to loopback in my.cnf:

[mysqld]
bind-address = 127.0.0.1
mysqlx-bind-address = 127.0.0.1

Next, I enabled general_log and registered auditd watches on the MySQL data directory:

SET GLOBAL general_log = 'ON';
SET GLOBAL log_output = 'TABLE';

# Monitor changes in the actual MySQL/aaPanel datadir (/www/server/data/)
auditctl -w /www/server/data/ -p wa -k mysql_data_change

# Make the audit rule persistent across reboots
echo "-w /www/server/data/ -p wa -k mysql_data_change" | sudo tee /etc/audit/rules.d/mysql.rules
sudo augenrules --load

2. Definitive forensic evidence in MySQL binary logs#

Because the incident occurred minutes before enabling the general log, I inspected the binary logs (binlogs) in /www/server/data/:

mysqlbinlog --no-defaults -v /www/server/data/mysql-bin.000010 | grep -iE "DROP|DELETE|TRUNCATE|RENAME" -B 5 -A 5

The output revealed the exact queries executed at 00:04:12:

#260403  0:04:12 server id 1  Query  thread_id=211
SET TIMESTAMP=1775174652/*!*/;
SET @@session.foreign_key_checks=0/*!*/;
DROP TABLE `blacklist_cache`,`blacklist_history`,`domain_dns`,`domain_whois`,`failed_jobs`,`ip_histories`,`jobs`,`migrations`,`monitors`,`users` /* generated by server */;

#260403  0:04:13 server id 1  Query  thread_id=211
alter table `user_agreements` add constraint `user_agreements_user_id_foreign` foreign key (`user_id`) references `users` (`id`) on delete cascade;

The forensic evidence was conclusive:

1. A batch DROP TABLE statement deleted all tables (including the migrations table).
2. Immediately following the DROP, subsequent queries attempted to rebuild foreign key constraints.
3. This signature matches the exact behavior of php artisan migrate:fresh or the RefreshDatabase testing trait.

3. Auditing web logs and sniffing local loopback traffic#

To determine whether the command was triggered via a web exploit, I checked the web server access logs (/www/wwwlogs/) at the exact timestamp (00:04:12):

grep "03/Apr/2026:00:04" /www/wwwlogs/dash.domain.com.br_ols.access_log

The result was completely empty. No external HTTP requests arrived during the drop.

To trace local queries in real time, I ran a packet capture on the loopback interface:

tcpdump -i lo -n -s 0 -l -A dst port 3306 | grep -iE "DROP|DELETE|TRUNCATE|RENAME|SCHEMA"

The capture picked up queue cleanup queries (delete from failed_jobs) originating from 127.0.0.1, confirming the execution came from local PHP background workers.

4. Resolving disk space bottlenecks before database restore#

While preparing the database restore, df -h revealed a critical risk: the root partition was at 92% utilization (only 8.5 GB free). Restoring a large InnoDB dump in a low-disk environment can trigger deadlock timeouts or table corruption from temporary transaction rollback logs.

Mapping the largest files on disk:

find / -type f -exec du -h {} + 2>/dev/null | sort -rh | head -n 10

I identified and cleared redundant storage consumers:

• Redundant swap files: /swapfile_novo (8 GB) and /home/www/swap (2 GB).
• Anomalous SQLite panel log: /www/server/panel/logs/sqlite_easy (3 GB).
• aaPanel Recycle Bin: /.Recycle_bin/ (3.6 GB).
• Docker volume temp files: /var/lib/docker/volumes/.../tmp/ (3.8 GB).

# Deactivate and delete extra swap files
swapoff /swapfile_novo 2>/dev/null && rm -f /swapfile_novo
swapoff /home/www/swap 2>/dev/null && rm -f /home/www/swap

# Clean recycle bin and logs
rm -rf /.Recycle_bin/*
rm -f /www/server/panel/logs/sqlite_easy

Free disk space increased from 8.5 GB to 28 GB (73% usage), providing the safety margin needed for InnoDB table reconstruction.

5. Fixing Nginx temporary cache permissions#

Nginx error logs reported recurrent write failures:

[crit] rename() "/www/server/nginx/proxy_temp_dir/..." to "/www/server/nginx/proxy_cache_dir/..." failed (13: Permission denied)

I reset ownership and directory permissions for user www:

chown -R www:www /www/server/nginx/proxy_cache_dir /www/server/nginx/proxy_temp_dir
chmod -R 755 /www/server/nginx/proxy_cache_dir /www/server/nginx/proxy_temp_dir

6. Definitive hardening and secure restoration#

With the environment clean and isolated, I enforced security boundaries before restoring the dump:

A. Revoke destructive privileges in MySQL#

Production application database users should not hold permissions to drop or alter schemas. In MySQL, DROP covers TRUNCATE TABLE (TRUNCATE does not exist as a standalone grant privilege in MySQL's syntax):

-- Revoke destructive structural commands (DROP covers TRUNCATE)
REVOKE DROP, ALTER ON *.* FROM 'laravel_user'@'localhost';
FLUSH PRIVILEGES;

B. Enforce Laravel framework protections#

In app/Providers/AppServiceProvider.php, enable Laravel's built-in destructive command guard in boot():

namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Illuminate\Support\Facades\DB;

class AppServiceProvider extends ServiceProvider
{
    public function boot(): void
    {
        // Blocks db:wipe, migrate:fresh, migrate:refresh, and migrate:reset in production
        DB::prohibitDestructiveCommands($this->app->isProduction());
    }
}

C. Clear queues and restore the database dump#

Because sql_log_bin is strictly a session-level variable in MySQL (SET GLOBAL sql_log_bin is not valid), disable binlog generation within the restore session to speed up importation without bloating replication logs:

# Purge residual queue jobs
php artisan queue:clear
php artisan cache:clear

# Import backup disabling binlog inside the session
(echo "SET SESSION sql_log_bin = 0;"; cat /path/to/backup.sql; echo "SET SESSION sql_log_bin = 1;") | mysql -u root -p database_name

Or within an interactive MySQL shell:

SET SESSION sql_log_bin = 0;
SOURCE /path/to/backup.sql;
SET SESSION sql_log_bin = 1;

Once the session disconnects, any new connection resumes binary logging automatically per my.cnf.

Practical lessons and post-incident hardening#

1. Principle of least privilege: Production database users should strictly hold data manipulation privileges (SELECT, INSERT, UPDATE, DELETE), never schema destruction grants like DROP or ALTER (noting that DROP covers TRUNCATE).
2. Lock down framework destructive commands: Ensure DB::prohibitDestructiveCommands() is present in your standard AppServiceProvider deployment checklist.
3. Binary logging is non-negotiable: Without mysqlbinlog, determining the exact second and query signature responsible for dropped tables would be nearly impossible.
4. Maintain disk headroom: Alert proactively on storage thresholds; InnoDB crash recovery and bulk imports require substantial free disk space for temporary transaction logging and index building.

Conclusion#

In this post, I documented the complete forensic trail — from listening sockets on :::3306 to the DROP TABLE queries in binary logs — and how Laravel's signature (migrate:fresh) emerged from the combination of foreign_key_checks=0 and / generated by server /. With bind restricted to loopback, MySQL privileges pruned, and the framework guard enforced in production, the incident was resolved and documented to prevent any recurrence.

See also

WordPress How to Detect and Clean Compromised WordPress Sites: Complete Forensic & Sanitization Guide

Cybersecurity Troubleshooting Fail2ban status 255 exception: migrating to systemd backend and Nginx modular includes

Cybersecurity Incident anatomy: mitigating simultaneous L7 attacks, Dovecot DoS, and CPU bottlenecks

This post is licensed under CC BY-NC.