Cache Hierarchy: Why CPUs Have Multiple Levels of Cache


Modern CPUs don’t just have one cache—they have three or four distinct levels, each with different sizes, speeds, and purposes. This hierarchy exists because of fundamental physics: you can have memory that’s fast or memory that’s big, but not both at the same time.

The core problem is that DRAM main memory operates at speeds measured in tens of nanoseconds, while CPU cores execute instructions in fractions of a nanosecond. Without cache, every memory access would stall the processor for hundreds of cycles. Cache hierarchy solves this by placing progressively larger but slower memory layers between the CPU and main memory.

Why Multiple Levels Instead of One Big Cache

A single large fast cache would be ideal, but it’s physically impossible. Cache speed depends on proximity to the CPU core and the number of transistors that need to be searched. SRAM, the technology used for CPU caches, is fast but expensive in terms of die area and power consumption.

L1 cache sits directly on each CPU core and provides access in 3-4 cycles. It’s typically split into separate instruction and data caches of 32-64 KB each. This split allows the CPU to fetch the next instruction while simultaneously accessing data, a technique called Harvard architecture.

L2 cache is larger—usually 256 KB to 1 MB per core—and sits slightly further away, responding in 10-12 cycles. It’s often unified, holding both instructions and data. The extra latency comes from the larger size: more memory cells means longer wire paths and more complex addressing logic.

L3 cache is shared across all cores, ranging from a few MB to over 100 MB on server chips. Access takes 40-50 cycles, but that’s still an order of magnitude faster than going to main memory. The sharing creates interesting dynamics: one core can warm up cache for another, but cores also compete for the same space.

The Inclusion Problem

Cache hierarchy originally used an inclusive design: anything in L1 was also in L2, and anything in L2 was also in L3. This simplified coherence—if a cache line wasn’t in L3, it wasn’t in any lower level either, making lookups faster.

But inclusion wastes space. If L3 holds 32 MB and duplicates everything in L1 and L2 across 16 cores, several megabytes are effectively lost. Modern processors increasingly use non-inclusive or exclusive policies where each level holds unique data. Intel dropped inclusivity in some recent architectures, AMD uses a victim cache approach, and ARM designs vary by implementation.

The tradeoff is complexity. Non-inclusive caches must track where data lives across multiple levels, adding metadata overhead and complicating coherence protocols. When core A modifies data that core B has cached, the system must invalidate or update B’s copy. Without inclusion, finding that copy requires snooping or directory structures that track precise locations.

Why L3 Exists at All

L2 is per-core and fast, so why bother with a slower shared L3? The answer is working set size and inter-core communication.

Most applications access far more than 256 KB of active data. Database query engines might touch dozens of megabytes of index nodes. Video encoders stream through large frame buffers. Without L3, these workloads would constantly evict useful data from L2, thrashing between cache and DRAM.

L3 also reduces coherence traffic. When cores share data—like threads updating a shared data structure—L3 provides a fast path for cache-to-cache transfers. Without it, every shared access could trigger a round trip to main memory or require complex core-to-core communication.

Victim Caches and L4

Some architectures add additional layers. AMD uses L3 as a victim cache that only stores data evicted from L2, maximizing unique data stored across levels. A few server chips include L4 cache, typically as large eDRAM or off-die memory in the 128-256 MB range. This helps workloads with enormous datasets, though it adds another 100+ cycles of latency.

The emerging trend is heterogeneous cache: some cores get more L2, or different cache configurations for performance versus efficiency cores. Apple’s M-series chips do this extensively, with large L2 caches on performance cores and smaller caches on efficiency cores.

Implications for Software

Cache hierarchy is mostly transparent, but its structure leaks through in performance. Algorithms that fit in L1 run dramatically faster than those that spill to L2, which in turn beat L3-bound code. The cutoffs matter: designing a hash table with 48 KB buckets instead of 64 KB can keep critical structures entirely in L1.

Data layout matters more than algorithmic complexity for many workloads. Linked lists perform poorly not because traversal is O(n), but because each node likely lives in a different cache line, triggering a new cache miss. Arrays enable prefetching and keep related data close.

Understanding cache hierarchy doesn’t require hand-optimizing every data structure, but it explains why certain patterns—tight loops, sequential access, small working sets—consistently win in benchmarks. The hardware is physically structured to reward locality, and multiple cache levels are the mechanism that makes it possible.