When two processes on the same machine need to communicate, the default path involves serialization, system calls, and kernel-mediated data copying. For high-throughput scenarios like database clients, browser sandboxing, or containerized microservices on a single host, this overhead compounds quickly. Memory-mapped RPC sidesteps most of it by having both processes share the same region of physical memory.

How Traditional IPC Creates Overhead

Standard inter-process communication mechanisms—Unix domain sockets, TCP loopback, or named pipes—all require the kernel to copy data from one process’s address space to another. Even with Unix domain sockets, which avoid network stack processing, you still serialize your data structure into a byte stream, invoke a system call to write, context-switch to the kernel, copy bytes into a kernel buffer, context-switch to the receiving process, and deserialize on the other side.

This pattern creates several bottlenecks. Serialization and deserialization consume CPU cycles and allocate temporary buffers. System calls interrupt execution flow and flush CPU pipelines. Most critically, the kernel copies data at least once, sometimes twice, depending on buffering strategy. For small messages this is manageable, but when passing large structures or streaming bulk data between colocated processes, the cost becomes the dominant factor in latency and throughput.

Memory-Mapped Files as Shared Memory

A memory-mapped file uses mmap to project a file (or anonymous memory region) directly into a process’s address space. When multiple processes map the same file, they gain direct access to the same physical memory pages. Write to an address in one process, and the other process sees the change immediately—no serialization, no system calls, no kernel copies.

The kernel handles the mapping and page table updates, but data transfer itself happens through normal memory loads and stores. This is as close to raw memory speed as IPC gets. Latency drops from microseconds to nanoseconds for cache-resident data, and throughput scales with memory bandwidth rather than syscall frequency.

Coordination and Synchronization Challenges

Shared memory eliminates copying but introduces synchronization complexity. Without kernel mediation, processes must coordinate access themselves. This typically involves a combination of atomic operations, memory barriers, and explicit synchronization primitives like semaphores or futexes placed in the shared region itself.

A common pattern is a lock-free ring buffer mapped into shared memory. The producer writes data and atomically updates a write pointer. The consumer polls or blocks on the read pointer. Memory ordering becomes critical—architectures with weak memory models require careful fence placement to ensure visibility. Without proper barriers, one process might see stale data or partial writes.

For structured RPC rather than raw streaming, you need a protocol layer on top. This might be a simple message queue with fixed-size slots, or a more sophisticated allocator that manages variable-sized objects within the shared region. Some systems embed reference counts or generation numbers to handle object lifecycle when both sides might access the same structure concurrently.

Real-World Applications

Databases use memory-mapped IPC extensively. PostgreSQL’s shared buffers live in shared memory accessible to all backend processes. Client libraries sometimes use shared memory for result set transfer when the client runs on the same host as the server, avoiding the serialization cost of the wire protocol.

Browser multi-process architectures rely on shared memory for performance. Chromium uses shared memory to transfer rendered frames from the renderer process to the browser process, and for passing large data structures between sandboxed components. The alternative—serializing bitmaps or DOM structures through IPC channels—would collapse throughput.

Container environments create another use case. When a sidecar proxy and application container run on the same node with shared volumes, mapping a file in that volume gives both direct memory access. This enables zero-copy data plane communication for service mesh implementations or observability agents that need to process high-volume request streams.

Tradeoffs and Limitations

Memory-mapped RPC trades simplicity for performance. Error handling becomes harder because a crash in one process can corrupt shared state. Security boundaries matter more—you can’t use this across trust domains without additional protection, since any process with access can read and write the entire region.

Portability varies. While POSIX defines mmap, behavior details differ across systems. Windows has its own shared memory primitives. Cross-platform abstractions exist but add complexity. And memory-mapped IPC only works within a single host—the moment you need network communication, you’re back to serialization and sockets.

For colocated processes exchanging high volumes of data, the performance gain is substantial enough that these tradeoffs are often worthwhile. The key is recognizing when you’re paying serialization tax unnecessarily, and when shared memory’s complexity is justified by the workload.