Photo by Luke Chesser on Unsplash
Memory Fragmentation in Long-Lived Services
Long-running services—databases, message brokers, web servers running for weeks or months—often exhibit a slow, insidious form of memory bloat that isn’t a traditional leak. The application isn’t losing references to data. Instead, the memory allocator gradually fragments the heap until available free space becomes scattered across thousands of tiny unusable regions. The service keeps requesting more memory from the kernel even though much of what it already holds is technically free.
This problem is distinct from a reference leak. A leak leaves objects on the heap with no way to reclaim them. Fragmentation, by contrast, means memory is free but split into fragments too small for the current workload. Both symptoms look identical from outside: resident memory grows steadily, garbage collection pauses lengthen, and eventually the service either slows to a crawl or crashes.
How Fragmentation Happens
Memory allocators hand out fixed-size blocks or variable-size regions from a contiguous heap. As a long-lived service allocates and frees objects over days of operation, a pattern emerges. Early allocations—long-lived objects like connection pools, caches, or global state—tend to survive. Shorter-lived objects—per-request buffers, temporary work structures, parsed JSON—get freed frequently.
The problem: if a long-lived object is surrounded by freed short-lived objects, those freed regions can’t be reused for large allocations. An allocator might have 100 MB of free memory across the heap, but if it’s fragmented into 1 MB chunks and a request needs a 10 MB buffer, the allocator has no contiguous region to satisfy it. It asks the kernel for more memory instead.
Over time, this creates a ratchet effect. Memory usage climbs, rarely descends, and the heap becomes increasingly fragmented.
Why Allocators Can’t Always Solve This
Modern allocators like jemalloc and tcmalloc use sophisticated strategies to minimize fragmentation: thread-local caches, size classes, arena segregation, and periodic compaction. Yet no allocator can perfectly solve heap fragmentation without either stopping the world to move objects around (which requires language-level support like Java’s garbage collection) or leaving fragmentation as an inherent tradeoff.
The core issue is that the allocator doesn’t control what patterns the application creates. If the workload intrinsically generates short-lived allocations scattered between long-lived ones, fragmentation is baked in. Some allocators trade off speed for fragmentation resistance by maintaining more metadata or doing more bookkeeping. Others optimize for allocation speed and accept that fragmentation will emerge over time.
In C and C++, there’s no general solution. A process-local malloc can’t move objects after they’re allocated; the application holds raw pointers to them. Rust’s ownership model helps by encouraging move semantics and by making it obvious when allocations are long-lived, but fragmentation can still occur in long-running Rust services.
The Practical Impact
For services that rarely restart, fragmentation compounds. A Kubernetes pod running for weeks before eviction or a bare-metal database running for months sees this effect accumulate. The service consumes twice as much RAM as its actual dataset. Its GC pause times grow because collectors need to scan more fragmented memory. CPU cache efficiency drops because data is spread across more physical pages.
The typical response is to restart the service periodically—a practice known as “jittered restarts” in reliability engineering. It’s a workaround, not a fix. A fresh process starts with a clean heap, and fragmentation gradually rebuilds.
Strategies That Help
Some services mitigate fragmentation through deliberate design choices. Using memory pools—pre-allocating fixed-size buffers at startup and reusing them—eliminates the dynamic allocation patterns that trigger fragmentation. This trades allocation flexibility for predictability.
Others use alternative memory layouts: arena allocators that group related objects, or bump allocators for short-lived allocations that can be freed in bulk. These techniques work when the application’s lifecycle matches the allocator’s assumptions.
For services where restarts are cheap, accepting periodic recycling is pragmatic. For stateful systems like databases or brokers where restart costs are high, minimizing fragmentation through careful allocation patterns becomes crucial.
The deeper lesson: memory isn’t just about consumption. How it’s managed—allocated, freed, and reused—shapes the system’s behavior over time. A growing heap can indicate correctness problems, but it often indicates an allocator and workload pattern that have gradually fallen out of sync.