Photo by Laura Ockel on Unsplash
Zone Reclaim Mode: NUMA Memory Policy Footgun
On multi-socket servers, a single kernel parameter can turn predictable millisecond latencies into second-long stalls. The culprit is vm.zone_reclaim_mode, a NUMA memory policy setting that forces Linux to reclaim local memory pages rather than reach across the interconnect to another socket’s RAM. What sounds like a reasonable locality optimization becomes a performance disaster under real workloads.
What Zone Reclaim Does
NUMA systems have multiple memory nodes, each attached to a specific CPU socket. Accessing local memory is fast—typically 70-90 nanoseconds. Remote memory access crosses the inter-socket fabric and costs roughly 1.5-2× the latency. Zone reclaim mode tells the kernel: when a NUMA node runs low on free pages, prefer reclaiming local pages over allocating from a remote node.
The logic seems sound. Keep allocations local to avoid remote access penalties. But the implementation creates a worse problem: when zone reclaim kicks in, the kernel scans local memory looking for pages to evict. It walks page tables, flushes dirty pages to disk, and breaks up active working sets—all while the application waits. A remote memory access might add 50 nanoseconds. Zone reclaim can block for milliseconds or longer, triggering I/O and cache disruption that cascades through the workload.
The Performance Cliff
Zone reclaim creates two pathological behaviors. First, it turns memory pressure into synchronous I/O. When the reclaim scanner decides a dirty page must go, it forces writeback before the allocation completes. An application thread trying to allocate a few kilobytes suddenly waits on disk. Second, it fragments the working set across nodes unpredictably. As reclaim evicts pages, subsequent access patterns pull data back from disk or recreate it, often landing on a different node. The kernel fights to keep allocations local while simultaneously destroying locality.
Databases and in-memory caches hit this constantly. A query allocates result buffers, zone reclaim fires, pages get evicted from the buffer pool, and the next query faults them back in—possibly from a remote node anyway. The overhead isn’t the remote access penalty you tried to avoid. It’s the reclaim scan, the I/O, and the refault storm.
When Remote Access Wins
Modern server fabrics are fast. AMD’s Infinity Fabric and Intel’s UPI deliver remote memory access in well under 200 nanoseconds on current platforms. Remote access is uniform cost, paid once per access. Zone reclaim is variable cost: sometimes it’s cheap, sometimes it triggers a multi-millisecond stall, and you don’t know which until it happens.
The math favors remote access for most workloads. A 50-nanosecond remote penalty on every access is cheaper than a single 10-millisecond reclaim stall. Even if 30% of your memory references go remote, you’re paying a consistent tax. Zone reclaim gives you 70% local access with occasional catastrophic pauses.
The Right Default
Most distributions now ship with vm.zone_reclaim_mode=0, disabling the behavior. The kernel allocates from remote nodes when local memory is tight, accepting the latency cost to avoid reclaim. For workloads where remote access genuinely matters—HPC jobs with carefully pinned memory, or real-time systems with strict latency bounds—you can enable it selectively and tune node affinity by hand.
If you inherit a server with zone reclaim enabled, the fix is immediate: sysctl vm.zone_reclaim_mode=0. For latency-sensitive applications on multi-socket hardware, check numastat or perf for remote access patterns. A general guideline: remote memory traffic above 15-20% is worth investigating, but focus first on whether you’re seeing reclaim stalls. Remote access is a predictable cost. Zone reclaim is a trapdoor.
Measuring the Impact
You can observe zone reclaim with /proc/vmstat. Look for numa_zone_reclaim_failed and pgsteal_* counters rising during allocation bursts. If you see zone reclaim activity correlating with latency spikes, you’ve found the problem. Disabling it shifts those allocations to remote nodes, which you can confirm by watching per-node memory stats in numastat. The remote access overhead is real, but it’s visible, consistent, and far cheaper than the alternative.
Zone reclaim is a textbook example of optimizing the wrong thing. It minimizes remote access at the cost of determinism, trading a small constant factor for unbounded stalls. On modern servers, the interconnect is fast enough that remote memory is almost always the better default.