Photo by Franck V. on Unsplash

NUMA-Aware Scheduling: Thread Placement and Cross-Socket Penalties


Modern multi-socket servers don’t present a uniform view of memory to all CPU cores. Each socket has its own attached memory, creating multiple NUMA (Non-Uniform Memory Access) nodes within a single system. A core can access memory attached to its local socket quickly, but accessing memory attached to a different socket requires traversing an interconnect like Intel’s UPI or AMD’s Infinity Fabric. This cross-socket hop adds latency, often 1.5x to 3x compared to local access.

The scheduler’s job becomes more complex in this environment. Simply distributing threads evenly across all available cores can inadvertently create performance problems if those threads end up accessing memory that’s far away. The best thread placement depends on where the memory lives, but memory moves, threads migrate, and the optimal layout shifts as the workload evolves.

Default Scheduler Behavior

Most operating systems implement some form of NUMA awareness in their schedulers. Linux, for instance, tries to keep threads running on the same NUMA node where they were created and where their initial memory was allocated. The kernel tracks which node allocated each memory page and attempts to schedule threads on cores within that node.

This works reasonably well for processes that allocate memory once at startup and then run hot loops over that data. A database buffer pool, for example, often gets allocated early in the process lifecycle. If the scheduler keeps the database threads pinned to the node that owns that memory, access patterns remain local and fast.

The strategy breaks down when threads regularly access memory from multiple nodes, when memory gets allocated on one node but primarily accessed from another, or when thread migration is necessary for load balancing. At that point, the scheduler faces a tradeoff: should it migrate the thread to the memory, migrate the memory to the thread, or accept the cross-socket penalty?

Thread Placement Strategies

Some applications bypass the default scheduler and implement explicit NUMA-aware placement. High-performance databases and network applications often partition work by NUMA node. Each node runs its own set of worker threads that only touch memory allocated from that node’s local banks. Incoming requests get routed to the appropriate thread pool based on which node owns the relevant data.

This partitioning avoids cross-socket traffic but introduces its own challenges. Work distribution becomes uneven if some nodes have more active data than others. Connection affinity or session stickiness is required to ensure that multiple requests for the same data land on the same node. And the partitioning scheme itself adds complexity to application logic.

Another approach is work stealing with NUMA consideration. Thread pools allow idle threads to steal work from busy threads, but the stealing policy accounts for NUMA distance. A thread will preferentially steal from queues on the same node, only reaching across to remote nodes when local work is exhausted. This maintains good CPU utilization while minimizing cross-socket memory access under normal load.

Memory Interleaving and Performance

Some systems offer an alternative configuration: interleave memory allocations across all NUMA nodes in a round-robin fashion. This spreads memory evenly and prevents any single node from becoming a bottleneck, but it guarantees that roughly half of all memory accesses (in a two-socket system) will be remote.

Interleaving can improve performance for memory-bandwidth-bound workloads where the aggregate bandwidth of all memory controllers matters more than individual access latency. But for latency-sensitive applications with good locality, interleaving throws away the advantage that NUMA topology provides.

The decision between node-local and interleaved allocation isn’t binary across the entire system. Modern allocators can apply different policies to different memory regions. Critical hot paths get node-local memory, while cold or rarely-accessed data structures use interleaved allocation to avoid exhausting any single node’s capacity.

Runtime and Language Considerations

Language runtimes add another layer of complexity. A garbage collector moving objects between memory regions can silently break NUMA locality. An object initially allocated on node 0 might get promoted and relocated to memory that happens to be backed by node 1, even though the threads accessing it still run on node 0.

NUMA-aware runtimes address this by partitioning the heap per node or by tracking object affinity during garbage collection. The JVM, for example, can be configured to prefer allocating objects on the same node where the allocating thread is running, and the GC can attempt to preserve that placement during compaction.

Thread pools within runtimes also make placement decisions. An async runtime scheduling thousands of lightweight tasks needs a NUMA-aware work distribution strategy, or it risks scattering tasks randomly across nodes and destroying locality. Pinning tasks to specific nodes works for some workloads, but flexible scheduling with NUMA-conscious stealing often provides a better balance.

Measuring the Impact

The performance difference between good and bad NUMA placement varies widely. Workloads with tight loops over small datasets that fit in cache won’t notice much difference because they rarely hit main memory. Workloads that stream through large datasets with poor reuse see dramatic effects, sometimes 50% or more performance variation based solely on thread placement.

Profiling tools can surface NUMA issues. Linux perf can report the ratio of local versus remote memory accesses, and many hardware performance counters expose cross-socket traffic. A workload that shows high cross-socket bandwidth or elevated memory access latency despite low cache miss rates likely has a NUMA placement problem.

The fix might be explicit thread pinning, memory binding with numactl, or application-level partitioning. But before optimizing, it’s worth measuring. NUMA effects are real and significant on multi-socket systems, but on single-socket machines or workloads with naturally good locality, the complexity of manual NUMA tuning often isn’t justified.