Investigating Linux network connections: why i abandoned netstat for SS and lsof at kernel level

Anyone operating busy web servers and production backends knows the exact moment when CPU usage spikes, request queues fill up, and you need immediate answers: what is happening with system connections right now?

For years, the default reaction was running legacy netstat. However, under high concurrency or network saturation, relying on net-tools binaries can freeze your terminal during the incident.

Below is a technical breakdown of why I permanently transitioned my diagnostic workflow to ss (Socket Statistics) and lsof, exploring system calls (syscalls) and Linux Kernel architecture.

The netstat bottleneck under load: the /proc trap#

When diagnosing hung ports or ghost sockets, diagnostic tool performance is critical.

netstat collects socket data by parsing virtual filesystem entries in /proc/net/tcp and /proc/net/udp. On systems handling tens of thousands of active connections, the kernel must format every socket descriptor into text before delivering it to userspace:

1. Virtual I/O and CPU overhead: The kernel spends valuable cycles serializing strings.
2. High query latency: The command takes seconds to render while the system is already constrained.
3. Non-atomic reads: Socket table states can shift mid-read across massive text buffers.

To bypass this overhead, modern Linux environments use ss (from iproute2).

The default diagnostic command (ss)#

To extract a fast, comprehensive view of sockets without network hangs:

ss -tupna

Flag breakdown:#

• -t: Filter TCP sockets.
• -u: Filter UDP sockets.
• -p: Display process name and PID owning the descriptor (requires sudo).
• -n: Numeric mode. Disables DNS resolution to prevent timeout delays during incidents.
• -a: Display all sockets (both listening and established).

Under the hood: netlink IPC and tcp_diag#

The advantage of ss is structural. Rather than reading /proc, ss interfaces directly with the Linux Kernel tcp_diag (or inet_diag) subsystem via Netlink IPC sockets.

Tracing ss with strace:

strace -e socket,sendto,recvmsg ss

The kernel opens an AF_NETLINK socket to deliver an atomic binary memory dump:

socket(AF_NETLINK, SOCK_RAW, NETLINK_INET_DIAG) = 3
sendto(3, {len=72, type=INET_DIAG_GET /* or TCPDIAG_GETSOCK */, ...}) = 72
recvmsg(3, {msg_name=..., msg_iov=[{...}]}, ...) = 4096

Direct Netlink IPC delivers socket states sub-millisecond, even during SYN floods or severe traffic spikes.

Hunting stuck ports and conflicts with lsof#

While ss excels at socket metrics, lsof (List Open Files) is unmatched for tracking down open file descriptors and resolving Address already in use (EADDRINUSE) errors.

1. Filter specific web ports without host resolution:#

lsof -i :80,443 -nP

2. Isolate the exact PID listening on a port:#

lsof -i TCP -s TCP:LISTEN

This immediately reveals orphan worker processes preventing Nginx, Apache, or Node.js from binding ports.

Useful fast filters and metrics in ss#

Instant TCP state summary:#

Check for elevated SYN_RECV (potential SYN flood) or TIME_WAIT (ephemeral port exhaustion):

ss -s

Inline native filters (no grep pipes needed):#

ss -o state established '( dport = :http or dport = :https )'

Unix domain sockets (PHP-FPM, MySQL):#

ss -x 'src unix:/tmp/php-fpm.sock' # or portably: ss -xlp | grep php-fpm.sock

Kernel tuning: essential sysctl parameters#

When inspecting network bottlenecks, review these kernel limits to prevent silent packet drops:

1. net.core.somaxconn: Maximum listen socket backlog queue length. Legacy distributions defaulted to 128 or 1024, whereas modern kernels (5.4+) set this to 4096. Raising it to 65535 prevents packet drops during heavy SYN bursts.
2. net.ipv4.ip_local_port_range: Allocatable ephemeral port range for outgoing connections (vital for reverse proxies routing to backends).

Check runtime values directly:

sysctl -a | grep -E 'somaxconn|port_range'

Conclusion#

Adopting Netlink-based tools like ss combined with lsof descriptor tracking delivers the speed and accuracy needed for reliable Linux network diagnostics.

See also

Linux Desktop freeze post-mortem: from futex lockups to Cgroups v2 on Linux

Linux Why your server needs SWAP even with free disk space: kernel memory mechanics and 4GB setup

Infrastructure Anatomy of a silent deadlock in CloudLinux and cPanel: phantom processes, session locks, and TCP queues

This post is licensed under CC BY-NC.