Most databases manage their own buffer pools with explicit read() and write() calls, maintaining fine-grained control over when pages enter memory and when dirty data hits disk. Memory-mapped database engines take a fundamentally different approach: they hand buffer management to the operating system by mapping data files directly into the process address space with mmap(). The result is simpler code and fewer abstractions, but the tradeoff introduces performance and reliability challenges that only become apparent under load.

How Memory-Mapped Engines Work

When a database maps its files into memory, data pages become accessible as regular memory addresses. Reading from a table means dereferencing a pointer; the page fault mechanism brings blocks into RAM transparently when needed. Writes update memory directly, and the kernel decides when to flush dirty pages back to disk through its own page cache logic.

This design eliminates the need for a custom buffer pool, eviction policies, and explicit I/O scheduling. LMDB, a widely used embedded database, builds its entire architecture on memory-mapped files. SQLite supports memory-mapped I/O as an option. The simplicity is real: the database can treat persistent storage as if it were an array, and the kernel handles the rest.

The Control You Lose

The appeal of delegating to the kernel comes with a loss of visibility and control. Traditional databases maintain detailed statistics on buffer pool hit rates, page evictions, and I/O patterns. They can pin hot pages in memory, prioritize certain workloads, and schedule writes to minimize interference. Memory-mapped engines surrender these levers.

The kernel’s page replacement algorithm—typically a variant of LRU—is generic and doesn’t understand database access patterns. It can’t distinguish between a sequential scan that should bypass the cache and a frequently accessed index page that should stay resident. When memory pressure arises, the kernel may evict critical pages while keeping less important data in RAM, and the database has no mechanism to influence this decision.

Write-back behavior is similarly opaque. The kernel flushes dirty pages on its own schedule, often in large asynchronous bursts. This can create unpredictable I/O spikes that interfere with foreground queries. Databases with explicit buffer pools batch and order their writes deliberately, smoothing I/O load and ensuring crash-consistent checkpoints.

Crash Recovery and Durability

Ensuring durability in a memory-mapped engine is more complex than calling msync(). The kernel’s page cache introduces multiple layers of buffering, and write ordering is not guaranteed. A transaction might write to several mapped pages, but the kernel may flush them in arbitrary order—or not at all before a crash.

LMDB solves this with a copy-on-write B-tree structure and atomic metadata updates, avoiding traditional write-ahead logging. Each transaction creates a new root, and only when the root is written does the transaction become durable. This design is elegant but tightly coupled to the storage structure. Other memory-mapped engines use WAL on top of mmap(), losing some of the simplicity that motivated the approach in the first place.

Crash recovery also depends on the filesystem. On filesystems without checksums or metadata journaling, a sudden power loss can corrupt a mapped file in ways that the database cannot detect. Traditional databases with explicit I/O have more control over fsync semantics and can implement their own checksumming.

When Memory-Mapping Works

Memory-mapped engines excel in read-heavy, embedded scenarios where simplicity and small code size matter more than tuning knobs. LMDB’s use in distributed systems like OpenLDAP demonstrates this well: predictable single-writer semantics, compact implementation, and zero-copy reads make it a strong fit for configuration storage and metadata.

The approach struggles under write-heavy workloads, high concurrency, or when the working set exceeds available RAM. Large-scale transactional databases like PostgreSQL and MySQL avoid memory-mapped data files entirely for their primary storage, preferring the control that comes with managing their own buffer pools and I/O scheduling.

The Persistent Tension

The decision to memory-map a database boils down to whether you value simplicity over control. The kernel is good at generic memory management, but database workloads are rarely generic. The cost of indirection through page faults, the lack of workload-aware eviction, and the difficulty of ensuring predictable durability make memory-mapping a careful tradeoff rather than an unambiguous win.

For embedded engines in constrained environments, that tradeoff often makes sense. For large-scale, high-throughput systems where every millisecond counts, most database designers choose to do the work themselves.