The migration was supposed to be small. Just a new column in a live database table. But everyone in the war room knew the stakes. One bad ALTER TABLE and the API would grind to a halt.
Adding a new column sounds simple. In practice, it can block writes, lock rows, and stall production. The size of the table, the type of the column, and the database engine all shape the risk. A schema change on a 5M-row table in PostgreSQL is not the same as one in MySQL or a cloud-native, distributed system.
When you add a new column, decide if it should have a default value. In PostgreSQL, adding a DEFAULT with a NOT NULL constraint will rewrite the table. That can take minutes or hours. Instead, add the column as NULL, backfill in small batches, then set constraints. In MySQL, an ALTER TABLE for a new column can be instant with the right storage engine—if you know the flags to use. Otherwise, use an online DDL tool to avoid downtime.
For high-traffic production systems, deploy schema changes with incremental steps: