The migration failed three minutes before the release window closed. The root cause stared back from the logs: a missing new column in production.
Adding a new column sounds simple. In high-traffic systems, it can be one of the most dangerous schema changes you’ll make. Schema migrations that add columns can lock tables, block writes, or trigger costly table rewrites. Even with modern databases, precision matters.
The first step is to understand your database engine’s behavior. In PostgreSQL, adding a new column with a default value rewrites the table. On MySQL with InnoDB, it can block reads and writes depending on the version and configuration. In both cases, the impact grows with table size. Design migrations to avoid full rewrites whenever possible.
Add the new column as nullable or with no default in the first step. Backfill the data in batches using an application job or an out-of-band script. Only after the data is in place should you add the default constraint. This minimizes downtime and reduces lock contention.