The migration went wrong at 2:13 a.m. The logs showed a failure in the database schema update. Someone had pushed code that assumed a column existed. It didn’t.
Adding a new column sounds simple. In production, it’s not. Every choice matters: column order, defaults, nullability, indexing, locking behavior, and backfill strategy. Get it wrong and you risk downtime, corrupted data, or deadlocks under load.
The first step is to define the new column in your migration file with intent. Pick a name that won’t collide with future changes. Set the correct data type to avoid costly casts later. Decide if it should allow nulls. Setting a default value can be useful, but remember that adding non-null columns with defaults can lock large tables.
Next, consider the deployment strategy. For small tables, a direct ALTER TABLE may be fine. For large tables in high-traffic systems, use an online schema change method. Tools like pt-online-schema-change or native database features can help. Test on a replica before touching production.