Photo by Brian Kostiuk on Unsplash
Packet Steering: Directing Network Traffic Across CPU Cores
When a network interface card receives packets at line rate, a single CPU core quickly becomes the bottleneck. Modern systems solve this by distributing packet processing across multiple cores, but deciding which core handles which packet involves subtle tradeoffs between throughput, latency, and cache efficiency.
The Single-Core Bottleneck
Without packet steering, all incoming packets interrupt a single CPU core. That core must handle the interrupt, pull data from the NIC, traverse the network stack, and deliver the packet to the application. At 10 Gbps and beyond, this serialized approach saturates one core while the rest sit idle. The NIC can deliver packets faster than one core can process them, creating a queue that grows until packets drop.
Receive-Side Scaling
Receive-Side Scaling (RSS) moves the distribution decision into hardware. The NIC examines each packet’s header and computes a hash over the IP addresses and port numbers. This hash determines which receive queue the packet enters, and each queue is associated with a specific CPU core. The operating system sets up multiple interrupt vectors, one per core, so each core independently processes its assigned queue.
The hash function ensures that all packets from the same TCP connection land on the same core. This affinity is critical: processing the same connection on the same core keeps TCP state warm in that core’s cache. If consecutive packets from a connection bounced between cores, each would trigger cache misses as the new core loaded the connection’s state from memory.
Software Steering Options
Not all NICs support RSS, and even those that do may benefit from additional software-level steering. Receive Packet Steering (RPS) implements similar logic in the kernel. When a packet arrives, the kernel hashes its header and enqueues it to the appropriate per-core backlog. RPS provides the benefits of RSS without requiring hardware support, though it consumes more CPU cycles since the first core still handles the interrupt before forwarding the packet.
Receive Flow Steering (RFS) extends RPS by considering application locality. Rather than using a static hash-to-core mapping, RFS tracks which CPU core is running the application that owns each socket. It steers packets for that socket to the core where the application is executing. This keeps both the kernel’s network state and the application’s data structures in the same CPU cache, reducing cross-core traffic.
Tradeoffs in Distribution
Perfect load balancing would give each core an equal share of packets, but connection affinity complicates this goal. A few heavy connections (elephant flows) might hash to the same core while many small connections land elsewhere. The result is an imbalanced workload where one core runs hot while others idle.
Some advanced NICs allow dynamic rebalancing, rehashing flows when cores become saturated. This improves balance but risks breaking connection affinity mid-stream, potentially causing packet reordering if old packets are still in flight on the original core when new packets arrive on the new core.
NUMA Considerations
On multi-socket servers, memory locality adds another dimension. If a NIC is attached to socket 0, steering packets to a core on socket 1 means crossing the inter-socket link for every memory access. The latency penalty can be significant. RSS and RPS configurations typically prefer cores local to the NIC’s socket, accepting some load imbalance to preserve memory locality.
The Busy Polling Alternative
Interrupt-driven packet processing, even when distributed, still incurs the cost of context switches and interrupt handling. High-frequency trading systems and low-latency applications often switch to busy polling. A dedicated core continuously polls the NIC’s receive queue, eliminating interrupt latency entirely. This sacrifices CPU efficiency for predictable, low-latency packet processing.
Practical Implications
Default steering configurations work well for general-purpose workloads, but tuning becomes important at higher throughput. If a few cores are pegged while others idle, examining the RSS hash distribution or enabling RFS can help. For workloads with a mix of elephant and mice flows, combining RSS with application-level load balancing (like SO_REUSEPORT for accepting sockets) distributes work more evenly.
Packet steering is invisible when it works but becomes the limiting factor when traffic patterns and core assignments misalign. Understanding the mechanisms and their tradeoffs helps diagnose throughput bottlenecks that don’t appear as network congestion or application slowness, but as lopsided CPU usage across cores.