Photo by Mika Baumeister on Unsplash
Why Databases Keep Reinventing the Write Path: A Look at Log-Structured Everything
If you look closely at modern storage engines, filesystems, message queues, and even some network protocols, you’ll notice the same trick showing up again and again: instead of updating data in place, append it to the end of a log and deal with the mess later. This is log-structured design, and it’s one of those ideas that keeps getting reinvented because the underlying hardware reality it exploits hasn’t gone away.
The core insight
Random writes are expensive. Sequential writes are cheap. That’s true on spinning disks because of seek time, and it’s still true on SSDs because of how flash memory handles erasure and wear leveling. Updating a record in place usually means reading a block, modifying a few bytes, and writing the whole block back, often in a location scattered across the device relative to the last write. Appending, by contrast, just means writing the next available slot in a contiguous region.
Log-structured systems take this observation and run with it. Instead of mutating data where it lives, every write becomes an append to a log. Updates and deletes don’t touch the original record at all. They just add a new entry that supersedes the old one, and some later process figures out what’s actually current and reclaims space from what isn’t.
This is the same core idea behind LSM trees in storage engines, write-ahead logs in traditional databases, log-structured filesystems, and even the segment-based storage used by distributed commit logs. The specifics vary, but the shape of the solution is consistent.
Why this trades one problem for another
The catch is that nothing is free. If you never overwrite data in place, your log grows forever, and most of what’s in it eventually becomes garbage: old versions of records that have since been updated or deleted. At some point you need a process that walks through the log, figures out which entries are still live, and rewrites them more compactly, discarding the dead weight.
This reclamation process goes by different names depending on the system: compaction in LSM-based databases, garbage collection in log-structured filesystems, checkpointing in write-ahead logging schemes. The mechanics differ, but the tradeoff is the same one every time. You’ve converted a random-write problem into a background bookkeeping problem. You’ve also introduced a new kind of cost: the system now has to do extra work later to clean up after itself, and that work competes for the same disk bandwidth and CPU that foreground reads and writes need.
This is why log-structured systems tend to be write-optimized but need real engineering effort to keep read performance and space usage under control. A log that’s never compacted will eventually make reads slow, because you might have to scan through many superseded versions of a record before finding the current one. A log that’s compacted too aggressively will compete with live traffic for I/O. Tuning that balance is a recurring theme in the design of these systems.
Where it shows up beyond databases
Once you recognize the pattern, you start seeing it everywhere. Filesystems designed for flash storage often avoid in-place updates for the same reason databases do: it plays nicer with how the underlying medium actually works. Message brokers that persist messages to disk tend to use append-only segment files rather than mutable records, partly because it simplifies durability guarantees and partly because sequential disk access is simply faster. Version control systems and content-addressable storage share a cousin of this idea too, where new content is added rather than old content being changed, and history is preserved by construction rather than by extra effort.
The common thread isn’t really about databases specifically. It’s about any system that needs durable, high-throughput writes and can tolerate deferring the cost of organizing that data until later. Append-only design turns a scattered, random access pattern into a sequential one, pushes the complexity of reconciliation into a separate process, and lets that process run on its own schedule rather than blocking the write path.
The tradeoff is permanent, not solved
It’s tempting to think of compaction and garbage collection as implementation details that will eventually get optimized away. They won’t, because they’re not bugs, they’re the other half of a deliberate tradeoff. Any system that chooses append-only writes for speed is implicitly choosing to pay for that speed later, in the form of background work to reclaim space and keep reads fast.
Understanding this helps explain a lot of operational behavior that otherwise looks mysterious: why a database might suddenly show elevated latency during a compaction cycle, why disk usage can balloon before it shrinks, or why some systems expose tuning knobs for how aggressively they clean up old data. These aren’t signs of a poorly built system. They’re the visible cost of a design decision that, on balance, is usually worth making.