Photo by Domaintechnik on Unsplash
Write Stalls: When SSDs Block Your Database
You’re running a high-throughput workload against your database when suddenly write latency spikes from single-digit milliseconds to seconds. Your monitoring shows CPU and disk are not maxed out, yet writes are effectively stalled. You’ve hit a write stall, a deliberate mechanism that sacrifices availability to maintain data integrity.
Write stalls occur when a storage engine’s background maintenance can’t keep pace with the incoming write rate. Rather than accept unbounded memory growth or risk data corruption, the system applies backpressure by blocking new writes until it catches up.
The Compaction Bottleneck
LSM-tree based storage engines like RocksDB, LevelDB, and Cassandra’s storage layer buffer writes in memory and flush them to immutable files on disk. Over time, these files accumulate and must be merged through compaction to reclaim space, remove deleted entries, and maintain read performance.
Compaction is fundamentally I/O bound. Each level might require reading and rewriting multiple gigabytes of data. If your application writes faster than compaction can merge files, the number of files grows without bound. This degrades read performance since queries must check more files, and eventually exhausts system resources.
The write stall is the safety valve. When file counts or memory usage crosses configured thresholds, the storage engine stops accepting new writes. It’s a controlled degradation that prevents the more catastrophic failure of running out of memory or disk space.
Triggering Conditions
Write stalls trigger under several conditions, each representing a different resource limit:
L0 file accumulation happens when memtables flush faster than L0 compaction can merge them into L1. L0 is special because its files have overlapping key ranges, so every L0 file must be checked during reads. Most engines impose strict limits on L0 file counts.
Pending compaction bytes tracks the total size of files waiting to be compacted. When this exceeds a threshold, writes slow or stop to let compaction reduce the backlog.
Memtable count limits prevent unbounded memory growth. If active and immutable memtables exceed the configured limit and flushes are backed up, writes must wait.
Soft and Hard Stalls
Many engines implement a two-tier approach. Soft stalls slow writes when approaching limits, giving compaction time to recover without completely blocking the application. Hard stalls impose full blocking when limits are exceeded.
RocksDB, for example, slows writes when L0 files reach level0_slowdown_writes_trigger and stops them entirely at level0_stop_writes_trigger. This graduated response lets operators tune the tradeoff between throughput consistency and absolute limits.
Why Not Just Queue Forever?
You might wonder why the engine doesn’t simply queue writes in memory indefinitely. The answer is resource exhaustion and durability. Unbounded queuing would eventually consume all available RAM. More critically, if the process crashes before those writes reach disk, you lose data despite acknowledging writes to clients.
Write stalls force the application to slow down rather than hide an unsustainable write rate behind ever-growing queues. They make the problem visible and synchronous rather than delaying catastrophic failure.
Mitigation Strategies
Preventing write stalls requires matching your write rate to your storage engine’s sustained compaction throughput. Increase compaction threads to parallelize the work. Tune compaction triggers to start merging earlier. Size your L1 appropriately since L0-to-L1 compaction is often the bottleneck due to L0’s overlapping structure.
For workloads with bursty writes, increase memtable size and count to absorb spikes. This trades memory for time, buffering bursts while compaction catches up. But it only defers the problem if your sustained write rate exceeds compaction capacity.
Hardware matters too. Faster SSDs with higher write bandwidth increase compaction throughput. Storage engines with multiple levels benefit from drives that handle parallel I/O well.
The Fundamental Tradeoff
Write stalls expose the fundamental tension in LSM-tree design. These engines optimize for write throughput by buffering and batching, but that optimization requires periodic maintenance. When maintenance can’t keep pace, the system must choose between integrity and availability. Write stalls choose integrity.
Understanding write stalls means understanding that your storage engine has finite capacity for disorder. You can configure where that limit sits, but you can’t eliminate it. The write rate your database can sustain isn’t just about disk bandwidth—it’s about the rate at which compaction can restore order to your data structure.