Debugging DNS resolution inconsistency: split-horizon and race conditions on Linux

The ghost of the legacy IP: split-horizon and race conditions in DNS resolution#

DNS is frequently the primary suspect behind erratic network behavior. Recently, I investigated a peculiar case: reverse "Split-Horizon" symptoms on a Linux server running DirectAdmin and BIND.

Here is the step-by-step diagnostic breakdown, from latency spikes and misrouted packets to the operating system level fix.

The issue: the ghost of the legacy IP#

The scenario was clear: the domain meudominio.com.br had migrated to the local server IP (203.0.113.50). Public Internet resolvers had already picked up the new record. However, running a standard ping from the server terminal (srv-core) resulted in ICMP echo requests routing to the legacy external IP (198.51.100.10).

Terminal output during the initial check:

[root@srv-core ~]# ping meudominio.com.br
PING meudominio.com.br (198.51.100.10) 56(84) bytes of data.
64 bytes from srv-old.provedor.com (198.51.100.10): icmp_seq=1 ttl=49 time=27.7 ms

Traffic was leaving the instance and fetching the domain across the WAN, introducing latency and breaking internal service communication.

Deep-dive analysis: the resolution path#

To identify why the kernel routed packets to the external gateway, I audited the system resolution stack. Commands like ping do not build raw DNS queries; they delegate name resolution to the standard C library (glibc via getaddrinfo()).

First, I checked /etc/hosts:

cat /etc/hosts

The file was clean, containing only standard loopback entries (127.0.0.1) and the server hostname. Nothing pinned the old IP.

Next, I checked the local BIND zone file managed by DirectAdmin:

cat /var/named/meudominio.com.br.db

The SOA serial was updated, and all A records (www, mail, ftp) pointed to the new local IP 203.0.113.50. Local BIND authoritative records were valid.

The divergence was located in /etc/resolv.conf:

cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 203.0.113.99
nameserver 1.1.1.1

The operating system was configured to bypass its local resolver entirely: 127.0.0.1 was omitted. The OS dispatched UDP queries straight to external nameservers.

Ruled-out hypotheses#

Before altering configurations, I isolated variables:

1. Global Propagation Delay:

Using dig (which bypasses glibc and nsswitch to query nameservers directly):

   dig @8.8.8.8 meudominio.com.br +short
   # Returns: 203.0.113.50

Both Google (8.8.8.8) and Cloudflare (1.1.1.1) returned the updated IP immediately. Public propagation was complete, unlike incidents involving stale glue records in BIND.

1. Local Name Service Caching Daemons (nscd / systemd-resolved):

Neither nscd nor systemd-resolved were running. Resolution followed standard libc paths.

Identifying the race condition#

To clear any stale route entries in the kernel, I flushed the route cache:

ip route flush cache

Executing ping immediately revealed the core symptom: a resolution race condition between nameservers:

[root@srv-core ~]# ping meudominio.com.br
PING meudominio.com.br (203.0.113.50) ... time=0.024 ms
...
[root@srv-core ~]# ping www.meudominio.com.br
PING www.meudominio.com.br (198.51.100.10) ... time=27.9 ms

Results alternated: when the query hit a fast resolver, it resolved locally with 0.024ms latency. When timing out or falling back to the secondary nameserver (203.0.113.99), it received an unpropagated response from an upstream forwarder with 27.9ms latency.

Practical fix#

On servers hosting local authoritative DNS via DirectAdmin and BIND, adding nameserver 127.0.0.1 as the primary resolver in /etc/resolv.conf is best practice.

To eliminate external network timeouts and guarantee zero-latency internal communication, I utilized /etc/nsswitch.conf. In Linux, the standard hosts: files dns directive forces the operating system to consult static files before querying external network resolvers.

I appended the static binding:

echo "203.0.113.50 meudominio.com.br www.meudominio.com.br" >> /etc/hosts

By adding this entry to /etc/hosts:

1. getaddrinfo() resolves the local IP instantly from memory.
2. The race condition ceased, and application traffic remained internal.

Practical DNS resolution takeaways#

Debugging DNS discrepancies requires distinguishing the test utility from the system call executed by application runtimes:

• dig and nslookup craft raw DNS packets and send them directly to target nameservers, bypassing /etc/hosts and /etc/nsswitch.conf.
• ping, curl, and language runtimes (PHP, Node.js, Python) invoke glibc getaddrinfo(), adhering strictly to /etc/nsswitch.conf precedence.
• When hosting local zones on the same machine, ensure /etc/resolv.conf references 127.0.0.1 or define static bindings in /etc/hosts to prevent local traffic from traversing external networks.

See also

Infrastructure DNS Troubleshooting at Layer 3: When the daemon loads the zone but the internet does not respond

Infrastructure Gmail rejecting DirectAdmin system emails: authenticating server hostnames with SPF and DKIM

Infrastructure Case Study (RCA): Investigating a Global Email Routing Outage Post-Migration in cPanel

This post is licensed under CC BY-NC.