The Recovery Problem

Modern databases need to survive more than just crashes. Operators need to restore data to an exact moment before a mistake—a bad migration, an erroneous bulk delete, or application logic that corrupted records. Traditional full backups can’t help if the corruption happened hours ago and was only discovered now. This is where write-ahead log (WAL) archiving becomes critical infrastructure.

WAL archiving turns the transaction log—already written for crash recovery—into a continuous backup stream. Instead of discarding old WAL segments after checkpointing, the database ships them to durable storage. Combined with periodic base backups, archived WAL segments enable point-in-time recovery (PITR): restoring the database to any moment in the retention window.

How WAL Archiving Works

When a database performs a checkpoint, it normally recycles old WAL segments that are no longer needed for crash recovery. With archiving enabled, those segments are instead copied to an archive location—S3, a network file system, or dedicated backup storage—before recycling.

The archive grows continuously as transactions generate new WAL. Each segment is a self-contained record of all changes that occurred during its time window. The database tracks which segments have been successfully archived and will refuse to recycle any segment until archiving confirms receipt.

A base backup captures the entire data directory at a known point in time, recording the WAL position at backup start. To perform PITR, you restore the base backup and replay archived WAL segments up to the target recovery point. The database applies exactly the same transaction stream that occurred in production, stopping at the precise moment you specify.

Continuous Archiving vs. Streaming Replication

WAL archiving complements but differs from streaming replication. Streaming replication sends WAL records to standby servers in near real-time, maintaining hot replicas for high availability. Archiving stores WAL segments in cold storage for disaster recovery and historical restoration.

Streaming replication provides low RTO (recovery time objective)—failover is fast. Archiving provides low RPO (recovery point objective)—you can restore to any point within the archive retention window. Production systems typically deploy both: streaming for availability, archiving for durability and point-in-time restoration.

Standby servers can also archive WAL, providing redundancy. If the primary fails before archiving a segment, the promoted standby continues archiving from its position, preventing gaps in the archive timeline.

Operational Considerations

WAL archiving introduces several operational constraints. Archive storage must be durable and sufficiently provisioned—losing archived segments breaks the recovery chain. If archiving falls behind, WAL segments accumulate on the primary, consuming disk space. If the primary runs out of space before archiving catches up, the database will stall writes to prevent data loss.

Archive commands must be idempotent and handle retries gracefully. The database may invoke the archive command multiple times for the same segment if previous attempts failed or timed out. Using atomic operations or checking for existing files before copying prevents duplicate work and alert noise.

Encryption and compression are typically applied at the archive layer. WAL segments contain plaintext transaction data, so encrypting them before storing in shared or cloud storage is critical. Compression reduces storage costs, though it must be balanced against CPU overhead during archiving and recovery.

Recovery Time and Testing

Point-in-time recovery speed depends on the amount of WAL to replay. Restoring from a week-old base backup with a week of archived WAL can take hours or longer for large databases. Regular base backups reduce recovery time but increase storage and backup window overhead.

Testing recovery procedures regularly is essential. Restoring to a point before a known event—a specific transaction commit—validates that the archive chain is intact and that the recovery process works as expected. Discovering broken archiving during an actual incident is too late.

The Durability Trade-off

WAL archiving shifts the durability model. Without archiving, the database guarantees durability only up to the last checkpoint on disk. With archiving, durability extends to the last archived WAL segment. This creates a new failure mode: if the primary crashes and the archive is lost or corrupted, recovery depends on the base backup alone, losing all transactions after that backup.

Many deployments use versioned or immutable storage for archives, combined with replication to multiple regions. The cost of archive storage is far cheaper than the cost of data loss, making aggressive redundancy economically rational for production systems.