Photo by Kevin Ache on Unsplash
Why Schema Changes Are the Scariest Deploys in Your Stack
Most deploys are forgettable. You ship code, watch dashboards for a few minutes, move on. Schema migrations don’t get that luxury. Renaming a column or adding a NOT NULL constraint on a table with hundreds of millions of rows can lock writes for minutes, saturate replication, or break every service still running the old application code. The database doesn’t care that your deploy pipeline is fast. It cares about the physical work of rewriting or reindexing data, and that work doesn’t compress just because you’re in a hurry.
Why migrations are harder than they look
A schema change looks like a single statement, but it’s really two problems layered on top of each other. The first is mechanical: some ALTER TABLE operations require rewriting the entire table on disk, which means holding locks, generating enormous amounts of WAL or binlog traffic, and competing with live production queries for I/O and buffer pool space. Adding a column with a default value used to be one of the classic examples of this, though most modern database engines have optimized the common cases so they’re closer to instantaneous metadata changes.
The second problem is social, not mechanical: multiple versions of your application are running at once during a rolling deploy. If you drop a column the moment new code stops using it, the old code that’s still running on some instances will start throwing errors. The database schema and the application code have to stay compatible with each other across the entire deploy window, not just at the start and end of it.
The expand-contract pattern
The standard answer to the compatibility problem is expand-contract, sometimes called parallel change. Instead of mutating a column in place, you split the change into stages that are each independently safe:
- Expand: add the new column, table, or constraint alongside the old one, without removing anything.
- Migrate: backfill data and update application code to write to both the old and new structures, then to read from the new one.
- Contract: once every instance of the application is confirmed to be running the new code, drop the old structure.
Each stage is a small, reversible step rather than one irreversible jump. If something goes wrong during the migrate stage, you roll back the application code, not the database. This is the same instinct behind feature flags and progressive delivery, applied to data instead of behavior: decouple the risky change from the deploy that depends on it.
Handling the heavy lifting
For changes that genuinely require rewriting a large table, teams generally avoid running the operation directly against the live table. Common approaches include:
- Online DDL tooling built into the database or provided by third-party utilities, which creates a shadow copy of the table, applies the schema change there, and swaps it in with a brief cutover.
- Batched backfills, where an application-level job copies data in small chunks with throttling, rather than one long-running transaction that holds locks and bloats the transaction log.
- Read replicas or a maintenance window for the rare cases where a brief, well-communicated pause is actually the safer option compared to a complex live migration.
The right choice depends on table size, how much write traffic the table sees, and how tolerant the business is of degraded performance for a few minutes versus zero disruption at all.
Where this is heading
As databases increasingly support online, non-blocking DDL natively, some of this pain is disappearing at the infrastructure layer. But the social half of the problem, keeping schema and application code compatible across a rolling deploy, isn’t something a database vendor can solve for you. That part is architectural discipline: treat every schema change as a multi-step migration with a compatibility window, not a single atomic edit, and the deploy stops being scary.