Photo by Mariia Berezovsky on Unsplash
Unified Memory: Why GPUs and CPUs Are Sharing Address Spaces
Every generation of AI hardware runs into the same wall: models keep growing, but the memory attached to a single accelerator doesn’t grow nearly as fast. The industry’s response has been a quiet but significant shift in how GPUs and CPUs share memory, moving from strictly separate address spaces toward unified and pooled memory architectures. Understanding this shift explains a lot about why recent hardware generations look the way they do.
The Old Model: Two Separate Worlds
For most of the GPU’s history as a compute accelerator, the CPU and GPU lived in completely separate memory domains. The CPU had its system RAM, the GPU had its own dedicated VRAM (typically GDDR or HBM), and moving data between them meant an explicit copy over PCIe. A program would allocate a buffer on the host, fill it with data, then call something like cudaMemcpy to push it across the bus before the GPU could touch it.
This model is simple and predictable, but it has two costs. First, PCIe bandwidth is far lower than either side’s local memory bandwidth, so every transfer is a potential bottleneck. Second, and more importantly for today’s workloads, it means the GPU is capped at whatever VRAM is physically installed on the card. If a model’s parameters, activations, and optimizer state don’t fit in that VRAM, you’re stuck sharding across multiple GPUs or falling back to slower paths, regardless of how much system RAM is sitting idle nearby.
Unified Memory: One Address Space, Managed Automatically
Unified memory (sometimes called managed memory) changes the programming model rather than the physical hardware. Instead of explicit copies, the CPU and GPU share a single virtual address space. A pointer allocated once is valid on both sides, and the runtime migrates pages between host and device memory on demand, triggered by page faults much like virtual memory works between RAM and disk.
This doesn’t eliminate the underlying data movement. Physically moving a page across PCIe still takes time. What it removes is the burden on the programmer to reason about exactly when and where data needs to live. For workloads with irregular or hard-to-predict access patterns, this can actually outperform hand-tuned manual transfers, because the system only moves the pages that are actually touched rather than copying entire buffers speculatively.
Why AI Workloads Pushed This Further
Large model training and inference exposed the limits of both the old copy-based model and basic unified memory. A single large language model can have parameters, gradients, optimizer state, and activation memory that collectively dwarf any single GPU’s VRAM, even on top-tier accelerators. Techniques like offloading optimizer state to host RAM, or streaming parameters in and out of GPU memory layer by layer, are really just software workarounds for the fact that hardware memory pools are still separate and finite.
This is the pressure driving hardware vendors toward tighter CPU-GPU memory coherence and, more recently, memory pooling technologies that let multiple accelerators and CPUs address a shared pool of memory over a fast, coherent interconnect rather than PCIe. The goal is to make the boundary between “GPU memory” and “system memory” less rigid, so that a workload can use whatever memory is available in a rack rather than only what’s soldered onto its own card.
The Coherence Problem
None of this is free. The hard part isn’t moving bytes, it’s keeping caches consistent when multiple processors can write to the same memory region. If a CPU core and a GPU streaming multiprocessor can both cache the same page, someone has to guarantee that a write on one side is visible on the other in a well-defined order. This is the same cache coherence problem that multi-socket CPU systems have dealt with for decades, just extended across a fundamentally more heterogeneous set of processors with very different memory access patterns and latency tolerances.
Coherent interconnects designed for this purpose add hardware protocol overhead to maintain that guarantee, which is a real cost. The bet the industry is making is that the flexibility of a larger, shared memory pool is worth more than the raw bandwidth of a smaller, dedicated, incoherent one, at least for workloads that are memory-capacity constrained rather than pure bandwidth constrained.
What This Means Going Forward
The direction is fairly clear even if the exact mechanisms vary by vendor: memory is increasingly treated as a disaggregated, poolable resource rather than something bolted permanently to a specific chip. This mirrors a broader trend in data center design toward disaggregating compute and storage, just applied one layer lower in the stack. For anyone building or tuning AI infrastructure, the practical implication is that memory capacity planning is becoming less about “how much VRAM does this GPU have” and more about “how much memory can this workload reach, and how fast.” That’s a more complicated question, but it’s also a more accurate one.