Resolving phantom IPs after manual DNS zone

Q: Forensic diagnosis: isolating the root cause?

When sysadmins run standard dig queries without specifying a name server, the results are queried from the local resolver configuration (such as Google, Cloudflare, or local ISP caches)-which are heavily anchored on Cache (TTL).

The phantom IP puzzle: manual edits and BIND cache desync#

System administrators managing corporate servers on the cPanel/WHM stack quickly learn that graphical interface limitations often demand direct shell interventions. During recent maintenance routines, I removed a hostile DNS record by editing the zone file directly via SSH. I saved the file and considered the task complete. However, testing showed that the removed IP persisted, "haunting" global resolver queries.

If you have experienced this issue, you know the frustration of running a dig query only to see an IP that was completely deleted from /var/named/. In this report, I document the isolation steps I applied, the common pitfalls of raw zone edits, and how to force absolute synchronization across cPanel DNS Cluster nodes.

The scenario and the problem layer#

The goal was to stop traffic by removing a specific IPv4 from the A-records of an active domain (domain.com). Logged in as superuser, I opened the BIND zone file.

However, before any direct intervention, it is mandatory to perform a security backup to allow fast rollbacks in case of critical syntax errors that could crash BIND:

# Backup the individual zone file before making changes
cp /var/named/meudominio.com.br.db /var/named/meudominio.com.br.db.bak.$(date +%Y%m%d)

# Create a full compressed backup of the BIND named directory
tar czf /root/named-backup-$(date +%Y%m%d).tar.gz /var/named/

Locating the correct zone file#

Sometimes, the DNS zone file for a domain might be named slightly differently or reside in alternative paths depending on the cPanel server's customizations. To identify and confirm the exact location of the .db file:

# 1. List all registered zones in the default BIND directory
ls -la /var/named/*.db

# 2. Check the named configuration to find the zone's 'file' directive
named-checkconf | grep -i "file"

# 3. Via WHM, view the path at: DNS Functions → Edit DNS Zone → Select domain → View path
# Or via cPanel CLI backend utilities:
cat /var/cpanel/users/usuario/dns

Once located and backed up, proceed with the edit:

vi /var/named/meudominio.com.br.db

I deleted the line containing the targeted IP (for example: 200.100.50.25), saved the file to disk, and verified the changes by querying the resolver. To my surprise, the terminal returned the following debug outputs:

[root@server ~]# dig +short A meudominio.com.br
10.20.30.40
10.20.30.41
200.100.50.25

The output clearly shows the issue: the IP 200.100.50.25 was still visible and routable. The physical file was updated in /var/named/, but the resolver on port 53 continued to serve the deleted record.

Forensic diagnosis: isolating the root cause#

When sysadmins run standard dig queries without specifying a name server, the results are queried from the local resolver configuration (such as Google, Cloudflare, or local ISP caches)-which are heavily anchored on Cache (TTL).

To determine if the issue was due to external ISP propagation latency or local server resolver caching, I ran a loopback query directly targeting the local BIND daemon:

dig @localhost +short A meudominio.com.br

This isolated query helped identify which of two scenarios were at play:

Scenario a: effective local update (clean @localhost)#

If the local query (@localhost) does not return the deleted IP, BIND has successfully read the updated zone file. The delay in public resolution is purely due to the TTL (Time To Live) configuration of the original zone.

If the .db file has a $TTL of 14400 (4 hours) or 3600 (1 hour), public resolvers will cache the old record until the TTL expires naturally. This is normal behavior and not a server configuration failure.

Scenario b: persistent local cache (old IP on @localhost)#

This was the actual issue: the local daemon continued to serve the obsolete IP, returning records I had deleted from the disk file. This indicated a mismatch between the Disk Storage State and the Active Memory Cache. Why would cPanel/BIND ignore direct manual edits? Below are the root causes.

Pitfalls of direct manual edits on cPanel#

Editing zone files directly via shell bypasses the automated validation hooks that trigger when changes are made via the WHM/cPanel GUI. Here are the two main issues to resolve:

1. Missing serial number increment (SOA failures)#

Unlike WHM local API operations, manual edits through text editors (like vi or nano) do not trigger automatic reloads. BIND does not monitor file updates on disk unless the SOA (Start of Authority) serial number is incremented.

The opened .db file requires manually increasing the serial number, formatted as YYYYMMDDnn (Year, Month, Day, Revision):

2024030801 ;Serial <- If this number is not increased, the change will be ignored

I incremented the serial to the next revision (2024030802) and saved the file.

Verifying the SOA Serial Number#

To confirm that the serial has been correctly modified in both the physical file and the local SOA response:

# Search for the serial number line in the zone file
grep -i "serial" /var/named/meudominio.com.br.db

# Query the local resolver for the SOA record to verify response
dig SOA meudominio.com.br @localhost +short
# Example output: ns1.meudominio.com.br. admin.meudominio.com.br. 2024030802 3600 900 604800 86400

2. Hidden alias cloned records (duplicate auditing)#

Partial removals can cause issues; make sure the target IP does not exist in other records (like CNAME or TXT aliases) within the file. I verified this using grep:

# Find all A records containing the old IP to be removed
grep "200.100.50.25" /var/named/meudominio.com.br.db

# Search for all active A records (matches optional host or leading space)
grep -E "^\s*(\S+\s+)?IN\s+A\s+" /var/named/meudominio.com.br.db

# Count total active A records accurately (prevents AAAA false matches)
grep -cE "^\s*(\S+\s+)?IN\s+A\s+" /var/named/meudominio.com.br.db

Safe operational sync procedure#

After correctly incrementing the serial number, the changes must be loaded into memory and propagated to the rest of the DNS Cluster. Here is the CLI procedure:

1. Syntax validation (safety net override)#

Ensure there are no syntax errors that could crash the local name server:

named-checkzone meudominio.com.br /var/named/meudominio.com.br.db

If the command returns "OK", you can safely proceed with BIND reloading. If it fails, resolve the errors before continuing. Common BIND syntax errors include:

ignoring out-of-zone data: Occurs when trying to define a record for a domain outside the zone's origin (typically because a domain name is missing the trailing dot . at the end, e.g., myotherdomain.com instead of myotherdomain.com.).
record with no data type: Occurs when a record line lacks the proper type configuration (e.g., missing IN A, misaligned TTL fields, etc.).

2. Zone permissions verification#

Zone files edited using root credentials can end up with permission mismatches, preventing BIND (named) from reading changes:

# Audit the current permissions
ls -la /var/named/meudominio.com.br.db

# File permissions must be 644 (-rw-r--r--) with owner named:named (or root:root depending on OS)
# Correct permissions if necessary:
chown named:named /var/named/meudominio.com.br.db
chmod 644 /var/named/meudominio.com.br.db

3. Reload BIND cache#

Instruct the local daemon to reload the updated zone:

rndc reload meudominio.com.br

Checking `rndc` and BIND Service Connectivity#

If reloading hangs or times out, check the health of rndc and BIND:

# Verify rndc and BIND operational statuses
rndc status

# Flush the local BIND cache to verify rndc connection
rndc flush

# Automatically detect the active BIND service name (named vs named-chroot on cPanel)
BIND_SVC="named"
systemctl is-active named-chroot >/dev/null 2>&1 && BIND_SVC="named-chroot"
systemctl is-active named >/dev/null 2>&1 && BIND_SVC="named"

# Confirm that the BIND daemon is active
systemctl status "$BIND_SVC"

Inspecting BIND System Logs#

Always tail the logs corresponding to the active service to verify that the zone reloaded cleanly:

# Follow BIND logs in real-time
journalctl -u "$BIND_SVC" -f

# View error logs specifically
journalctl -u "$BIND_SVC" -p error

# Filter for warnings and errors during zone reading
journalctl -u "$BIND_SVC" | grep -i "error\|warning" | tail -20

4. Sync DNS cluster#

If the server is part of a WHM DNS Cluster (Master-Slave model via WHM DNS Roles), manual updates are not automatically pushed to slave nodes. Instead of performing a heavy full cluster synchronization (/scripts/dnscluster syncall), sync only the specific zone:

/scripts/dnscluster synczone meudominio.com.br

Verifying DNS Cluster Status#

To confirm that the DNS cluster is operational and running correctly:

# Check the current status of the cluster
/scripts/dnscluster status

# List the active zones synced on the cluster
/scripts/dnscluster listzones

# Monitor the DNS cluster log in real-time
tail -50 /usr/local/cpanel/logs/dnscluster.log

# Force global synchronization of all zones if the cluster is out of sync:
/scripts/dnscluster syncall

5. Post-editing: external propagation verification#

After syncing BIND and the DNS cluster, ensure that changes have propagated to public DNS servers:

# Query public resolver IP targets
dig @8.8.8.8 meudominio.com.br A +short
dig @1.1.1.1 meudominio.com.br A +short

# Inspect the remaining TTL cache time on public resolvers
dig meudominio.com.br | grep -A1 "ANSWER SECTION"

# Quick shell loop to query multiple public DNS servers
for dns in 8.8.8.8 1.1.1.1 208.67.222.222; do
    echo "DNS $dns: $(dig @$dns meudominio.com.br +short)"
done

Advanced BIND & cPanel considerations: DNSSEC, dynamic zones, and WHM templates#

A. DNSSEC signed zones#

If the domain has DNSSEC enabled, the zone records are cryptographically signed. Any manual changes to the .db file will break the signature chain (RRSIG records), causing validating DNS resolvers to return a SERVFAIL error. If you manually edit a signed zone, you must re-sign it. If cPanel manages the keys:

# Re-sign the zone using cPanel's internal tools
/usr/local/cpanel/bin/dnssec-signzone meudominio.com.br

Or manually using the BIND tools:

dnssec-signzone -o meudominio.com.br -N INCREMENT /var/named/meudominio.com.br.db

B. Dynamic DNS zones (DDNS) and concurrency (`rndc freeze/thaw`)#

If the zone receives dynamic updates (e.g., via APIs, DHCP, or auto-updating hooks), BIND maintains changes in a journal file (.jnl). Editing the .db file directly while the named daemon is active will cause synchronization conflicts, and named will overwrite the .db file from cache or the journal, discarding your edits. To safely edit dynamic zones, use the freeze/thaw commands:

# 1. Freeze the zone (dumps the journal to the .db file and suspends dynamic memory updates)
rndc freeze meudominio.com.br

# 2. Make the manual edits in the .db file

# 3. Thaw the zone (reloads BIND with the edited file and re-enables dynamic updates)
rndc thaw meudominio.com.br

C. WHM zone templates#

cPanel automatically rebuilds DNS zone files during various administrative routines (e.g., changing account IPs, domain name edits, or backup restorations) based on templates stored in /var/cpanel/templates/dnszones/. Direct modifications in /var/named/ that diverge from the structural templates might be completely overwritten. If you are applying persistent structural additions, consider adapting them inside the WHM zone templates.

Troubleshooting and resolution flowchart#

The following diagram maps the troubleshooting sequence for resolving local caching and BIND cluster synchronization conflicts:

flowchart TD A["dig +short meudominio.com.br returns old IP"] --> B[dig @localhost +short] B --> C{"@localhost returns old IP?"} C -->|No| D["Public Cache/TTL issues. Wait for TTL to expire or reduce value"] C -->|Yes| E["Was SOA Serial incremented?"] E -->|No| F["Increment Serial YYYYMMDDnn and save"] E -->|Yes| G[named-checkzone OK?] G -->|No| H["Fix syntax errors: missing trailing dot in FQDN, $ORIGIN missing"] G -->|Yes| I{Is the zone dynamic?} I -->|Yes| J["rndc freeze edit zone file rndc thaw"] I -->|No| K[rndc reload meudominio.com.br] J --> L["Verify file permissions named:named + chmod 644"] K --> L L --> M{Is zone signed with DNSSEC?} M -->|Yes| N[Re-sign zone with dnssec-signzone] M -->|No| O[/scripts/dnscluster synczone] N --> O O --> P[dnscluster status OK?] P -->|No| Q[tail -f /usr/local/cpanel/logs/dnscluster.log] P -->|Yes| R["dig @8.8.8.8 +short verify external propagation"]

SOP: safe DNS zone edits on cpanel/bind#

Below is the standard operating procedure checklist to ensure no validation or safety steps are missed:

Pre-editing#

[ ] Detect Service: Identify active named service: systemctl list-units | grep named
[ ] Zone Backup: Create a copy of the original database file: cp /var/named/zona.db /var/named/zona.db.bak
[ ] Verify Current Serial: Run local SOA query: dig SOA dominio.com @localhost +short
[ ] Dynamic Zone Check: If dynamic, freeze zone first: rndc freeze dominio.com

Editing#

[ ] Edit File: Open the file in a shell editor: vi /var/named/zona.db
[ ] Check $ORIGIN: Ensure the $ORIGIN directive is defined.
[ ] FQDN Trailing Dot: Ensure all fully-qualified names end with a trailing dot ..
[ ] Increment Serial: Increase the SOA serial number using the format YYYYMMDDnn.
[ ] Check Duplicates: Confirm the target IP is not configured in any other CNAME aliases.
[ ] Save changes: Save and quit the editor.

Validation & permissions#

[ ] Thaw Zone: If frozen, restore updates: rndc thaw dominio.com
[ ] Validate Syntax: Run named-checkzone: named-checkzone dominio.com /var/named/zona.db
[ ] Fix Permissions: Ensure files are writable and owned by the correct user (chmod 644, chown named:named).

Activation & clustering#

[ ] Reload Cache: Load the updated zone into BIND: rndc reload dominio.com
[ ] Sign DNSSEC: Re-sign zone keys if DNSSEC is active.
[ ] Sync DNS Cluster: Push the specific zone updates to the cluster: /scripts/dnscluster synczone dominio.com
[ ] Monitor System Logs: Verify no errors are thrown during zone load on the correct active service (named or named-chroot): journalctl -u "$BIND_SVC" -n 50 --no-pager

Post-editing (propagation checks)#

[ ] Local Verification: Test local resolution: dig @localhost dominio.com +short
[ ] External Verification: Query public DNS resolvers: dig @8.8.8.8 dominio.com +short
[ ] Monitor DNS: Monitor the domain performance over the next 24 hours.

Automated DNS zone validation script#

To minimize manual errors, use this validate-dns.sh diagnostic script to audit local BIND zone file updates:

#!/bin/bash
# validate-dns.sh - BIND DNS Zone Diagnostics on cPanel
# Usage: ./validate-dns.sh <domain.com>

ZONE="${1:-}"
DB_FILE="/var/named/${ZONE}.db"

if [ -z "$ZONE" ] || [ ! -f "$DB_FILE" ]; then
    echo "Error: Domain or zone database file not found."
    echo "Usage: $0 <domain.com>"
    exit 1
fi

echo "=========================================================="
echo " RUNNING DIAGNOSTICS FOR DNS ZONE: ${ZONE}"
echo "=========================================================="

# 1. Detect the active BIND service name automatically on cPanel/WHM
BIND_SVC="named"
systemctl is-active named-chroot >/dev/null 2>&1 && BIND_SVC="named-chroot"
systemctl is-active named >/dev/null 2>&1 && BIND_SVC="named"

# 2. Check file metadata, active service ($BIND_SVC), and permissions
echo -e "\n[1] Zone File Metadata, Active Service ($BIND_SVC), and Permissions:"
ls -la "$DB_FILE"

# 3. Extract declared serial number
echo -e "\n[2] Serial Stated in DB File:"
grep -i "serial" "$DB_FILE" | head -1

# 4. Run syntax validation
echo -e "\n[3] Syntax Validation (named-checkzone):"
CHECK_OUT=$(named-checkzone "$ZONE" "$DB_FILE")
CHECK_RES=$?
echo "$CHECK_OUT"
if [ $CHECK_RES -ne 0 ]; then
    echo -e "\n--> [ERROR] Critical syntax error found in zone file. DO NOT reload BIND!"
    exit 1
fi

# 5. Check ORIGIN and trailing dot sufixes
echo -e "\n[4] Check of $ORIGIN and DNS suffixes:"
grep -i '^$ORIGIN' "$DB_FILE" || echo "Warning: ORIGIN directive not declared in zone file."
echo "Displaying first FQDN lines for visual check:"
grep -E "^[^$;]" "$DB_FILE" | grep -v "^$" | head -5

# 6. List active A records
echo -e "\n[5] Active A Records in Zone:"
grep -E "^\s*(\S+\s+)?IN\s+A\s+" "$DB_FILE" | head -15

# 7. Local resolution query
echo -e "\n[6] Local Resolution (Query localhost):"
dig "@localhost" "$ZONE" A +short

# 8. External resolution query
echo -e "\n[7] External Resolution (Query Google DNS 8.8.8.8):"
dig "@8.8.8.8" "$ZONE" A +short

# 9. rndc status check
echo -e "\n[8] Local rndc Status:"
rndc status | head -5

echo -e "\n=========================================================="
echo " DIAGNOSTICS COMPLETED SUCCESSFULLY"
echo "=========================================================="

Forensic summary#

Manually editing files in /var/named/ is fast but requires adhering to core BIND and cPanel workflows, including SOA serial incrementing, BIND service detection (named vs named-chroot), DNSSEC re-signing, dynamic zone freeze/thaw, and cluster synchronization. "Phantom IPs" in queries are simple synchronization failures that can be resolved by this checklist: Remove Record -> Increment SOA Serial -> Reload BIND -> Sync Cluster.

By performing local loopback tests (@localhost), you can isolate propagation issues from local server failures, confirming that the BIND memory state is aligned with the disk configuration.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

This post is licensed under CC BY-NC.

Resolving phantom IPs after manual DNS zone edits on cPanel

The phantom IP puzzle: manual edits and BIND cache desync#

The scenario and the problem layer#

Locating the correct zone file#

Forensic diagnosis: isolating the root cause#

Scenario a: effective local update (clean @localhost)#

Scenario b: persistent local cache (old IP on @localhost)#

Pitfalls of direct manual edits on cPanel#

1. Missing serial number increment (SOA failures)#

Verifying the SOA Serial Number#

2. Hidden alias cloned records (duplicate auditing)#

Safe operational sync procedure#

1. Syntax validation (safety net override)#

2. Zone permissions verification#

3. Reload BIND cache#

Checking `rndc` and BIND Service Connectivity#

Inspecting BIND System Logs#

4. Sync DNS cluster#

Verifying DNS Cluster Status#

5. Post-editing: external propagation verification#

Advanced BIND & cPanel considerations: DNSSEC, dynamic zones, and WHM templates#

A. DNSSEC signed zones#

B. Dynamic DNS zones (DDNS) and concurrency (`rndc freeze/thaw`)#

C. WHM zone templates#

Troubleshooting and resolution flowchart#

SOP: safe DNS zone edits on cpanel/bind#

Pre-editing#

Editing#

Validation & permissions#

Activation & clustering#

Post-editing (propagation checks)#

Automated DNS zone validation script#

Forensic summary#

Was this article helpful?

Comments

The phantom IP puzzle: manual edits and BIND cache desync#

The scenario and the problem layer#

Locating the correct zone file#

Forensic diagnosis: isolating the root cause#

Scenario a: effective local update (clean @localhost)#

Scenario b: persistent local cache (old IP on @localhost)#

Pitfalls of direct manual edits on cPanel#

1. Missing serial number increment (SOA failures)#

Verifying the SOA Serial Number#

2. Hidden alias cloned records (duplicate auditing)#

Safe operational sync procedure#

1. Syntax validation (safety net override)#

2. Zone permissions verification#

3. Reload BIND cache#

Checking rndc and BIND Service Connectivity#

Inspecting BIND System Logs#

4. Sync DNS cluster#

Verifying DNS Cluster Status#

5. Post-editing: external propagation verification#

Advanced BIND & cPanel considerations: DNSSEC, dynamic zones, and WHM templates#

A. DNSSEC signed zones#

B. Dynamic DNS zones (DDNS) and concurrency (rndc freeze/thaw)#

C. WHM zone templates#

Troubleshooting and resolution flowchart#

SOP: safe DNS zone edits on cpanel/bind#

Pre-editing#

Editing#

Validation & permissions#

Activation & clustering#

Post-editing (propagation checks)#

Automated DNS zone validation script#

Forensic summary#

Was this article helpful?

See also

Comments

Checking `rndc` and BIND Service Connectivity#

B. Dynamic DNS zones (DDNS) and concurrency (`rndc freeze/thaw`)#