Photo by Franck V. on Unsplash

Pointer Swizzling: Converting Between Disk and Memory References


When you store a data structure with pointers to disk and later load it back into memory, those pointers break. The addresses they contained refer to locations in a process that no longer exists. Pointer swizzling is the technique for fixing this: translating disk-relative offsets into valid memory addresses when loading persistent data structures.

The problem appears anywhere you persist complex data structures—databases, object stores, serialization frameworks, memory-mapped file systems. Any system that saves pointer-rich data and needs to reconstruct the graph of references when loading it back.

The Core Problem

A pointer is just a memory address. When you write a pointer to disk, you’re writing a number that was meaningful in one address space but becomes meaningless when read into a different process or even the same process at a different time. Virtual memory means even the same binary launched twice will likely place the heap in different locations.

A simple example: you have a binary tree node with left and right child pointers. In memory, those pointers might be 0x7f4a2000 and 0x7f4a2040. Write that structure to disk and load it back later—those addresses are now either invalid or pointing at completely unrelated memory.

The naive solution is to not use pointers at all. Store everything as offsets from a base address or use handles/indices into arrays. This works but forces you to abandon the performance benefits of direct pointer traversal while the data is in memory.

Swizzling Strategies

Eager swizzling fixes all pointers at load time. Walk the entire data structure once, translating every disk offset into a memory address before allowing access. The upside is simple runtime access—once swizzled, the data structure works exactly like a normal in-memory structure with zero additional indirection. The downside is load time cost. For large structures, this means touching every page immediately, potentially causing a storm of page faults.

Lazy swizzling translates pointers on first access. Each pointer includes a tag bit indicating whether it’s already been swizzled. On access, check the tag; if unswizzled, translate the offset to an address and set the tag. This spreads the translation cost over time and avoids touching data that’s never accessed. The cost is a check on every pointer dereference and more complex bookkeeping.

No swizzling keeps everything as offsets forever. Access requires adding the base address to every offset on every dereference. This is the simplest approach implementation-wise but adds arithmetic to the critical path. Memory-mapped databases sometimes take this route, trading pointer dereference speed for simplicity.

Tag Bits and Encoding

The swizzled/unswizzled distinction needs to be encoded somewhere. The classic trick uses the high bit of the pointer. On most architectures, user-space addresses don’t use all 64 bits—only the low 48 or so are actually significant. Use bit 63 as a flag: 0 for memory addresses, 1 for unswizzled disk offsets.

When accessing a potentially-unswizzled pointer, check the bit. If set, mask it off, interpret the remaining bits as an offset from the data structure’s base address, compute the real address, write it back, and proceed. If clear, it’s already a valid pointer.

This encoding constrains your offset range to whatever fits in 63 bits minus whatever you need for the base address representation, but that’s rarely the limiting factor.

Database Index Structures

B-tree implementations in databases face this problem constantly. The tree’s structure exists partly on disk and partly in memory. Nodes are paged in as needed. Each node contains pointers to child nodes, which might be represented as page IDs on disk.

Some systems maintain parallel structures: disk pages with page IDs, memory nodes with real pointers, and a translation table mapping page IDs to memory addresses. Access checks the translation table first; if the page is already loaded, follow the cached pointer. Otherwise, load the page and populate the translation.

Other systems use uniform references that work in both contexts—effectively offset-based access with caching layers on top. The “pointer” is always a page ID plus offset, but frequently accessed pages stay pinned with their addresses cached.

Performance Considerations

Swizzling is fundamentally a space-time trade-off with a side of memory access patterns. Eager swizzling optimizes for access speed at the cost of load latency. Lazy swizzling balances the two but adds conditional overhead to every access. No swizzling keeps load times minimal but penalizes every traversal.

The right choice depends on access patterns. Data structures with high locality that see repeated traversals—working set fits in cache, same nodes accessed repeatedly—benefit from eager swizzling. Sparse access patterns where most data is never touched favor lazy approaches. Transient structures loaded and discarded quickly may be better served by offset-based access entirely.

Swizzling also interacts with garbage collection and memory management. Swizzled pointers are real pointers, which means they participate in GC pointer scanning. Offset-based references don’t, which can simplify collector implementation but complicates object graph traversal.

Modern Alternatives

Contemporary systems often avoid explicit swizzling by using more abstract reference types from the start. Capabilities, fat pointers with metadata, and handle-based systems provide uniform reference semantics that work across persistence boundaries without translation. The cost is indirection, but modern hardware makes that increasingly affordable.

Memory-mapped file systems and persistent memory architectures push toward representations that work identically on disk and in memory, eliminating the translation problem by making virtual memory do the work. The data structure’s pointers are always offsets from a fixed base, and the OS maps that base to a consistent virtual address.

Still, for systems working with legacy data formats or optimizing for absolute minimal pointer-chasing overhead, swizzling remains a relevant technique. Understanding the trade-offs helps when designing any system that bridges persistent and volatile representations of complex data structures.