Running fastpanel and aaPanel on Podman (Rocky Linux): the UID mapping nightmare and real solutions

This is a technical account of one of those journeys that every "die-hard" infrastructure analyst eventually faces: the attempt to run a traditional web control panel inside containers without compromising host stability.

If you are trying to deploy Fastpanel or aaPanel via Podman/Docker on Rocky Linux, this article will save you hours of frustration regarding filesystem permissions and UID mapping.

1. The initial block: "unsupported OS" and the fastpanel trap#

Everything started with the need to deploy a control panel on a Rocky Linux VM, while keeping the host clean by using containers. When attempting to run the Fastpanel installation script, the first "wall" appeared: the script checks the /etc/os-release file and refuses the installation if the OS version isn't on their exact whitelist (usually focused on Debian/Ubuntu or pure CentOS).

The technical fix: os-release hijacking with systemd#

The solution wasn't to force the script onto Rocky Linux, but to create a contained environment that the installer would accept as "native." I opted for an Ubuntu 22.04 image with native Systemd support. This is non-negotiable, as panels like Fastpanel rely heavily on systemctl to manage the stack (Nginx, MariaDB, PHP-FPM).

# Spawning the Ubuntu container to "deceive" the installer
podman run -d \
  --name control-panel \
  --privileged \
  --tmpfs /tmp --tmpfs /run --tmpfs /run/lock \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  -p 80:80 -p 443:443 -p 8888:8888 \
  docker.io/jrei/systemd-ubuntu:22.04

The --privileged flag and precisely mapped /sys/fs/cgroup mounts are essential for internal Systemd functionality under Podman.

2. The permission nightmare: UID errors and innodb (ibdata1)#

After bypassing the OS barrier, the next challenge was data persistence. Mapping host volumes to the container (/home/user/mysql -> /var/lib/mysql) immediately resulted in the dreaded OS errno 13: Permission denied.

Auditing the MariaDB/MySQL logs revealed that the internal process could neither read nor write to the ibdata1 file.

2.1 mandatory preventive backup routine#

Before modifying any permissions or altering user namespaces, perform a complete backup of the host data directories to prevent data loss:

# Create a secure timestamped backup directory
BACKUP_DIR="/root/panel-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Back up the host directory into a compressed tarball
tar czf "$BACKUP_DIR/data-backup.tar.gz" /home/percio/data/

echo "Backup successfully saved to: $BACKUP_DIR"

2.2 the root cause: rootless Podman and UID namespaces#

In a rootless Podman environment, the user you see as root inside the container is actually your standard unprivileged user on the host. However, services like MySQL run as a specific user (usually UID 106 inside the Ubuntu container).

When you map a volume, Rocky Linux sees the file as belonging to your host user (e.g., UID 1000). The container's internal UID 106 does not have write permissions to this file because of the way subUID mapping handles ownership boundaries.

2.3 the definitive fix via podman unshare#

Using chmod 777 is a security failure and often doesn't even work due to SELinux. The correct solution is to use the user namespace to adjust ownership on the host in a way the container understands:

# 1. Ensure the host user physically owns the data initially
sudo chown -R percio:percio /home/percio/data

# 2. Use unshare to translate the container's UID 106 (mysql) to the host
podman unshare chown -R 106:106 /home/percio/data/mysql

# 3. Adjust for the primary panel user (usually internal UID 1000)
podman unshare chown -R 1000:1000 /home/percio/data/config

Furthermore, the :Z suffix on the volume is mandatory on Rocky Linux: -v /home/percio/data/mysql:/var/lib/mysql:Z

This instructs Podman to label the file with the correct SELinux security context, enabling the container process to access the disk.

2.4 post-uid mapping verification#

To ensure that namespace permissions were applied correctly, run the following verification checks:

# 1. Verify file ownership inside the container
podman exec -it control-panel ls -la /var/lib/mysql/

# 2. Verify MySQL process UID inside the container
podman exec -it control-panel ps aux | grep mysql

# 3. Verify container startup logs for permission errors
podman logs control-panel 2>&1 | grep -i "permission\|error"

3. Third layer: SELinux verification#

On Rocky Linux, SELinux prevents container access to host paths by default. The :Z mount option labels host paths with container_file_t.

SELinux audit and diagnostic commands:#

# 1. Verify current SELinux labels on host directories
ls -Z /home/percio/data/mysql/

# 2. Query host audit logs for recent container/mysql AVC denials
sudo ausearch -m avc -ts recent | grep mysql

# 3. Check global host SELinux status
getenforce

# 4. If contexts are misconfigured, restore contexts recursively
sudo restorecon -Rv /home/percio/data/

4. Fourth layer: systemd and MariaDB container diagnostics#

Since these control panels depend on internal Systemd daemons, we must audit container startup health and database connectivity.

Check systemd and running services:#

# 1. Check if Systemd is running as PID 1 (should return "systemd" or "init")
podman exec -it control-panel ps -p 1 -o comm=

# 2. Check general Systemd status inside the container
podman exec -it control-panel systemctl status

# 3. List active services inside the container
podman exec -it control-panel systemctl list-units --type=service --state=running

Check mariadb/mysql:#

# 1. Check MySQL daemon status inside the container
podman exec -it control-panel systemctl status mysql

# 2. Test internal database connectivity and execute statements
podman exec -it control-panel mysql -u root -p"your_password" -e "SHOW DATABASES;"

5. Fifth layer: alternatives to --privileged and hardening#

The --privileged flag should be deprecated in secure production environments. Secure your panel container using capabilities and sandbox hardening:

5.1 using specific capabilities#

Grant only required network and system capabilities to run the panels:

podman run -d \
  --name control-panel \
  --cap-add SYS_ADMIN \
  --cap-add NET_ADMIN \
  --cap-add SYS_PTRACE \
  --security-opt label=type:container_runtime_t \
  --tmpfs /tmp --tmpfs /run --tmpfs /run/lock \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  -v /home/percio/data:/data:Z \
  -p 80:80 -p 443:443 -p 8888:8888 \
  docker.io/jrei/systemd-ubuntu:22.04

5.2 using rootful Podman (sudo)#

Running systemd containers under rootful Podman (sudo podman) retains native seccomp profiles and capability filtering, which is significantly more secure than rootless container execution under --privileged.

5.3 resource limiting and hardening#

Apply resource limits on the container to prevent denial-of-service (DoS) states on the host system:

# Limit memory to 2GB and CPUs to 2 cores, with a read-only root filesystem
podman run -d \
  --name control-panel \
  --memory=2g \
  --cpus=2 \
  --read-only \
  --tmpfs /tmp --tmpfs /run --tmpfs /run/lock \
  -v /home/percio/data:/data:Z \
  ...

6. Sixth layer: backups and container lifecycle updates#

A resilient containerized setup requires safe database backups and automated updates without losing state.

6.1 container-specific backups#

Generate database logical dumps and snapshot container layers:

# Export a database dump from the container to the host
podman exec -it control-panel mysqldump -u root -p"your_password" --all-databases > /root/db-backup-$(date +%Y%m%d).sql

# Snapshot and save the container image state
podman commit control-panel panel-snapshot:$(date +%Y%m%d)
podman save panel-snapshot:$(date +%Y%m%d) -o /root/panel-snapshot-$(date +%Y%m%d).tar

6.2 safe update lifecycle#

To upgrade your control panel base image or operating system:

# 1. Stop and remove the old container instance
podman stop control-panel
podman rm control-panel

# 2. Pull the latest base image
podman pull docker.io/jrei/systemd-ubuntu:22.04

# 3. Spin up the container referencing the persistent volume mappings
podman run -d --name control-panel ... (mapped volumes)

7. Seventh layer: network port and firewall verification#

Validate port exposures:#

# Audit exposed port mappings in Podman
podman port control-panel

# Test local responses on ports 80 and 8888
curl -I http://localhost:80
curl -k https://localhost:8888

Configure host firewall (Rocky Linux):#

# Check existing active ports in Firewalld
sudo firewall-cmd --list-all

# Permit persistent traffic to panel services
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=443/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8888/tcp --permanent
sudo firewall-cmd --reload

8. Eighth layer: container log auditing#

For active troubleshooting or interface timeouts:

# Tail logs in real-time
podman logs -f control-panel

# Output last 100 entries with precise ISO timestamps
podman logs --tail 100 --timestamps control-panel

# Search container runtime for error logs
podman logs control-panel 2>&1 | grep -iE "error|fail|warning"

9. The authentication mystery: CLI vs. web in fastpanel#

At this stage, MySQL was running, but a bizarre paradox emerged: I would reset the database or panel password via the command line (using internal scripts like mogwai), receive a "Success" message, but when trying to log in via the web interface, it would return "Incorrect Password."

After a forensic audit of the container's filesystem, I identified three probable causes:

1. Internal SQLite Inconsistency: Fastpanel maintains its config in an internal SQLite database (/etc/fastpanel/fastpanel.sqlite) which can fall out of sync with MariaDB if there were write errors during the initial bootstrap.
2. Hash Corruption: Because the installer initially failed due to UID permissions, it populated the database tables with corrupted seeds/hashes.
3. Session Locks: The container failed to persist PHP session files due to the same UID mapping issue, invalidating any successful login attempt immediately.

10. Pivot to aaPanel: the modular alternative#

My conclusion after 10 years in infrastructure: Fastpanel is excellent on bare metal, but its deep dependencies on Systemd and its monolithic stack make it hostile to clean container isolation.

I migrated the lab to aaPanel. aaPanel is modular and its installation scripts handle Docker/Podman environments significantly better because they don't demand total control over PID 1.

The critical port 22 (SSH) warning#

Be wary of generic community tutorials. Many suggest mapping -p 22:21 due to a typo. Never do this. Mapping the container's FTP port (21) to your Host's port 22 will knock out your remote SSH access. Use safe mappings like: -p 2121:21 (FTP) and -p 8888:8888 (Panel).

The 404 not found (security entrance) error#

When accessing the aaPanel IP for the first time, you will likely see a 404 Not Found. This is not a failure; it's a security feature. aaPanel generates a random entry token (e.g., /8f2a1b) that must be appended to the URL.

How to recover this URL from within the container:

# Check the bootstrap logs
podman logs aapanel

# Or call the internal aaPanel CLI
podman exec -it aapanel bt 14

11. SRE conclusion and final playbook#

Running control panels in containers on Rocky Linux requires understanding the "handshake" between Host and Guest through User Namespaces and SELinux layers.

Podman container security checklist:#

• [ ] Security: Avoid using --privileged by running containers with specific capabilities or rootful Podman.
• [ ] Backups: Ensure datestamped backups of the physical directory /home/percio/data are taken before changes.
• [ ] UID Mapping: Adjust file permissions on the host using podman unshare for container services.
• [ ] SELinux: Ensure volumes use the :Z suffix and resolve permission denials using restorecon.
• [ ] Validation: Verify internal Systemd, MariaDB daemon state, and ports after container launch.
• [ ] Hardening: Set memory/CPU bounds and secure interfaces.

Symptoms vs. causes correlation table#

12. Automated diagnostic script (diagnose-podman-panel.sh)#

Use this script to audit container volume mappings, security contexts, firewall ports, and internal daemon status on the host system:

#!/bin/bash
# diagnose-podman-panel.sh
# Diagnostic script for control panels running on Podman (Rocky Linux).
# Must be executed as root or a user with Podman access.

set -euo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0;37m'

log_info() { echo -e "[${GREEN}INFO${NC}] $1"; }
log_warn() { echo -e "[${YELLOW}WARN${NC}] $1"; }
log_error() { echo -e "[${RED}ERROR${NC}] $1"; }

CONTAINER_NAME="control-panel"
if [ $# -ge 1 ]; then
    CONTAINER_NAME="$1"
fi

log_info "Starting diagnostics for container: $CONTAINER_NAME..."

# 1. Verify container existence and state
if ! podman ps -a --format "{{.Names}}" | grep -qw "$CONTAINER_NAME"; then
    log_error "Container '$CONTAINER_NAME' not found."
    exit 1
fi

CONTAINER_STATUS=$(podman inspect --format "{{.State.Status}}" "$CONTAINER_NAME")
log_info "Container Status: $CONTAINER_STATUS"

# 2. Privilege Audit (Privileged check)
IS_PRIVILEGED=$(podman inspect --format "{{.HostConfig.Privileged}}" "$CONTAINER_NAME")
if [ "$IS_PRIVILEGED" = "true" ]; then
    log_warn "The container is running with full privileges (--privileged=true). Consider migrating to specific capabilities."
else
    log_info "Container is running without full privileges (Secure)."
fi

# 3. Volumes and SELinux Audit
log_info "--- Volume & SELinux Auditing ---"
VOLUMES=$(podman inspect --format "{{range .Mounts}}{{.Source}}:{{.Destination}} {{end}}" "$CONTAINER_NAME")
for vol in $VOLUMES; do
    src=$(echo "$vol" | cut -d':' -f1)
    dest=$(echo "$vol" | cut -d':' -f2)
    if [ -d "$src" ] || [ -f "$src" ]; then
        context=$(ls -Zd "$src" | awk '{print $4}' 2>/dev/null || ls -Zd "$src" | awk '{print $1}')
        log_info "Volume: $src -> $dest | SELinux Context: $context"
        if [[ "$context" != *"container_file_t"* ]]; then
            log_warn "Volume '$src' is not labeled as container_file_t. Append :Z or run restorecon."
        fi
    else
        log_error "Volume source path does not exist on host: $src"
    fi
done

# 4. Host SELinux Status
if command -v getenforce >/dev/null 2>&1; then
    log_info "Host SELinux Status: $(getenforce)"
fi

# 5. Ports and Firewall Audit
log_info "--- Port & Network Mapping ---"
podman port "$CONTAINER_NAME" || log_warn "No exposed ports found."

if command -v firewall-cmd >/dev/null 2>&1; then
    log_info "Host Firewalld Open Ports:"
    firewall-cmd --list-ports || true
fi

# 6. Systemd and MariaDB Internal Container Check
if [ "$CONTAINER_STATUS" = "running" ]; then
    log_info "--- Internal Container Diagnostics ---"
    
    # PID 1 Check
    PID1=$(podman exec "$CONTAINER_NAME" ps -p 1 -o comm= 2>/dev/null || echo "Error")
    log_info "Container PID 1: $PID1"
    
    # Systemd Status
    if podman exec "$CONTAINER_NAME" systemctl is-system-running >/dev/null 2>&1 || true; then
        systemd_status=$(podman exec "$CONTAINER_NAME" systemctl is-system-running 2>/dev/null || echo "Unknown")
        log_info "Container Systemd Status: $systemd_status"
    fi
    
    # MariaDB Status
    if podman exec "$CONTAINER_NAME" systemctl status mysql >/dev/null 2>&1; then
        log_info "MariaDB/MySQL Service: Running (OK)"
    elif podman exec "$CONTAINER_NAME" systemctl status mariadb >/dev/null 2>&1; then
        log_info "MariaDB/MySQL Service: Running (OK)"
    else
        log_error "MariaDB/MySQL service is inactive or failing inside the container."
    fi
else
    log_warn "Container is not running. Skipped internal audits."
fi

log_info "Diagnostics complete."

See also

Infrastructure The Day Python 3.10 "broke" my aaPanel on Podman (and how i fixed it)

Infrastructure From Podman to Docker: how i cleaned my environment and resolved "device or resource busy" port conflicts

Linux Optimizing memory and killing SWAP exhaustion on Linux servers with aaPanel

This post is licensed under CC BY-NC.