When the server froze: anatomy of a memory thrash#
Few scenarios in everyday development are as disruptive as watching a Linux server suddenly freeze with CPU pinned at 100% without an immediate, obvious code-level error.
Recently, I hit a recurring bottleneck on my Linux development server: whenever I ran VS Code Remote (SSH) in parallel with Antigravity agent workflows, CPU utilization maxed out, degrading terminal responsiveness to the point of a near system freeze.
Below is the technical post-mortem: moving past initial assumptions of pure compute saturation, identifying memory thrashing and swap stalls, analyzing kernel metrics via CLI, and implementing containment strategies to stabilize the host.
1. Initial symptoms and early hypotheses#
The symptom was immediate: opening VS Code Remote alongside Antigravity triggered maxed-out CPU and unresponsive SSH sessions.
Under these workloads, three initial hypotheses are common:
- File Watcher and Indexing Churn (
inotifyandripgrep): The VS Code Server relies heavily on the kernelinotifysubsystem and backgroundrg(ripgrep) workers to index workspace directories. Large repositories with deep unignored trees (node_modules,.git) generate continuous CPU cycles. - Context Switching and Thread Contention: Highly concurrent multithreaded binaries can cause excessive thread scheduling overhead inside the kernel scheduler.
- Aggressive Garbage Collection: Node.js runtimes (powering the VS Code Server and IDE subsystems) triggering non-stop GC cycles under memory pressure.
To isolate the root cause, the first step was collecting raw system metrics.
2. False CPU saturation: discovering i/o wait#
Checking top showed 98% CPU utilization, but inspecting individual breakdown fields revealed what was actually happening:
%Cpu(s): 4.2 us, 38.5 sy, 0.0 ni, 1.1 id, 55.8 wa, 0.0 hi, 0.4 si, 0.0 st
User-space execution (us) was only at 4.2%. The actual utilization was driven by:
sy(Kernel / System Time) at 38.5%: The kernel actively managing page tables, memory reclaim, and interrupt queues.wa(I/O Wait) at 55.8%: The CPU spending over half its time completely idle, stalled waiting for disk read/write requests to complete.
Verifying with vmstat 1 5:
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 4 1845248 182412 24108 412580 1420 2840 3200 4100 4810 9820 5 39 1 55 0
5 5 1845248 179320 24108 411200 1890 3120 4100 4800 5120 11400 4 42 0 54 0
The si (swap-in) and so (swap-out) columns showed thousands of blocks moving per second. The server was trapped in Memory Thrashing: physical RAM was exhausted, forcing the kernel to page active processes to disk swap. Every terminal keystroke or IDE event required swapping pages back and forth from disk, creating a massive I/O bottleneck.
3. Mapping memory-consuming processes#
To audit which processes held high Resident Set Size (RSS) and pushed the system into swap:
ps -eo pid,ppid,pmem,rss,comm,args --sort=-rss | head -n 12
The primary consumers were:
node(VS Code Server / Extension Host): Multiple processes consuming over 1.8 GB RSS while scanning unfiltered trees.antigravity-server/ parallel agents: Retaining memory buffers for context caching.rg(ripgrep): Search loops actively indexing workspace logs and temporary artifact folders.
4. Containment and optimization steps#
Stabilizing the environment required adjustments across three layers: editor configuration, kernel memory tunables, and cgroups containment.
1. Hardening VS code file watchers#
Create or update .vscode/settings.json in the remote workspace to prevent inotify and rg from indexing heavy directories:
{
"files.watcherExclude": {
"**/.git/objects/**": true,
"**/.git/subtree-cache/**": true,
"**/node_modules/**": true,
"**/.gemini/**": true,
"**/tmp/**": true,
"**/logs/**": true
},
"search.exclude": {
"**/node_modules": true,
"**/.git": true,
"**/.gemini": true,
"**/tmp": true
},
"files.autoSave": "afterDelay",
"files.autoSaveDelay": 2000
}
2. Tuning kernel swappiness and cache pressure#
Linux defaults to vm.swappiness = 60. On development servers hosting active in-memory runtimes, reduce swap aggressiveness:
Edit /etc/sysctl.d/99-dev-tuning.conf:
# Reduce premature disk swapping
vm.swappiness = 10
# Maintain stable directory and inode caches in RAM
vm.vfs_cache_pressure = 50
# Configure dirty page background flush thresholds
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
Apply immediately:
sudo sysctl --system
3. Setting up high-priority zram SWAP#
Instead of allowing thrashing to hit physical disk storage, configure compressed RAM swap (zram) with higher priority:
# Allocate 4GB compressed zram block device
sudo zramctl --find --size 4G --algorithm lz4
sudo mkswap /dev/zram0
sudo swapon /dev/zram0 -p 100
Because zram compresses memory pages directly in RAM with microsecond latency, inactive pages are compacted without incurring disk I/O penalties.
4. Limiting agent runtimes with Cgroups v2#
To ensure parallel AI agent runs do not exhaust host memory, wrap executions inside a systemd resource slice:
systemd-run --user --slice=agents.slice --property=MemoryMax=3G --property=CPUWeight=50 ./antigravity-server
Practical containment takeaways#
When a remote server appears pegged at 100% CPU during intensive development workflows, do not assume pure computational load:
- Always check the
wa(I/O wait) metric intopand thesi/sofields invmstat. In many scenarios, the CPU is simply stalled waiting for disk-bound swap thrashing. - Configure explicit exclusion rules for VS Code watchers across heavy subdirectories.
- Combine
zramwithsystemdresource slices to ensure neither the editor nor concurrent agents can exhaust the host and disrupt SSH connectivity.
Was this article helpful?
Leave a quick reaction to help prioritize future technical guides:
This post is licensed under CC BY-NC.



Comments
Join the discussion below.
0 comments