Photo by Laura Ockel on Unsplash
NUMA Distance Metrics: Why Hop Count Matters for Memory Access
Non-Uniform Memory Access (NUMA) architectures are often described with a simple binary: local memory is fast, remote memory is slow. But on modern multi-socket servers with dozens of cores per socket, this framing hides a critical nuance. Memory access latency varies continuously based on how far the request must travel through the interconnect fabric, not just whether it crosses a socket boundary.
Beyond Local and Remote
A two-socket server presents the simplest NUMA topology: each socket has its own memory, and cross-socket access incurs a fixed penalty as requests traverse the inter-socket link (typically Intel’s UPI or AMD’s Infinity Fabric). But add a third or fourth socket, and the picture changes. A CPU accessing memory on an adjacent socket experiences different latency than accessing memory two hops away through an intermediate socket.
This matters because the memory controller sits on the destination socket, not on some neutral backplane. Every request must be routed hop-by-hop through the fabric, with each traversal adding latency and consuming interconnect bandwidth. A four-socket system has multiple paths between nodes, and the physical topology—ring, mesh, or fully-connected—determines the hop count for each source-destination pair.
Distance Tables and System Locality
Linux exposes this topology through the numactl --hardware command, which reports the relative distance between each NUMA node as a unitless integer. A local access is distance 10 (the baseline), while remote nodes are assigned higher values proportional to their latency cost. On a two-socket system, cross-socket access might be distance 21. On a four-socket ring topology, adjacent sockets might be 21, but opposite sockets could be 31 or higher.
These distance values guide the kernel’s memory allocation decisions. When a process requests memory, the kernel prefers the local node, but under memory pressure it consults the distance table to choose the next-best node. A workload running on node 0 that exhausts local memory will see allocations come from node 1 (distance 21) before node 3 (distance 31), minimizing the average access latency.
Why Hop Count Compounds
Each hop through the interconnect consumes not just time but also bandwidth. A memory-intensive workload spread across all four sockets generates cross-traffic that competes for the same physical links. If node 0 and node 2 both need to access memory on node 3, their requests may traverse the same link, creating contention that wouldn’t exist if the memory were evenly distributed by distance.
This is why distance-aware scheduling can outperform simple NUMA-aware allocation. Pinning a process to node 0 and its memory to node 1 is better than scattering allocations across nodes 1, 2, and 3 with an average higher hop count, even if the total memory is split evenly. Consistent, short paths reduce both latency and fabric congestion.
Implications for Database and HPC Workloads
Applications that assume uniform remote access costs can make poor placement decisions. A database that stripes its buffer pool across all NUMA nodes may inadvertently create hot paths through the interconnect, as query threads on node 0 repeatedly reach across two hops to node 3. Recognizing that distance varies allows smarter partitioning: place frequently co-accessed data on nearby nodes, and isolate independent working sets on distant nodes to avoid interference.
High-performance computing workloads often pin threads and pre-allocate memory to enforce strict locality, but even then, understanding hop distance helps when scaling out. A parallel simulation that spawns tasks on all cores must account for the fact that synchronization between distant nodes costs more than between adjacent ones. Minimizing cross-traffic on the longest paths can improve scalability on large NUMA systems.
Observability and Tuning
Profiling tools like numastat and perf can reveal remote access patterns, but they rarely break down access by distance. Identifying whether remote accesses are primarily one-hop or multi-hop requires correlating hardware topology with allocation maps, often through vendor-specific performance counters. Some systems expose interconnect utilization per link, making it possible to spot bottlenecks on specific hops.
For workloads that can’t avoid remote access, tuning interleaving policy by distance—spreading allocations across near nodes only—can reduce tail latency compared to round-robin interleaving across all nodes. It’s a middle ground between strict local allocation and naive distribution, preserving some parallelism without paying the full multi-hop penalty.
NUMA distance isn’t an academic detail. On systems with three or more sockets, hop count directly affects memory latency, bandwidth contention, and ultimately application throughput. Treating NUMA as a binary loses information that matters when every microsecond counts.