The database migration hit production at 02:14. Seconds later, the error logs began to spike. A missing new column had taken the API down.
Adding a new column is one of the most common schema changes. Done wrong, it can cause downtime, block writes, or break backward compatibility. Done right, it’s invisible to end users and safe even under high load.
When planning a migration, first check the target database engine’s ability to add a new column without locking the table. PostgreSQL can add a nullable column instantly if no default is set. MySQL may require a full table rewrite depending on version and engine. Always confirm with EXPLAIN and staging tests before touching production.
If the new column requires a default value, consider adding it in two steps. Step one: add the column as nullable. Step two: backfill in small batches to avoid performance degradation. After data is populated, alter the column to set NOT NULL or apply necessary constraints.
Indexes need separate attention. Adding an index on a new column can lock writes unless performed concurrently. Use CREATE INDEX CONCURRENTLY in PostgreSQL or ALGORITHM=INPLACE in MySQL when possible. Measure the operation in terms of lock time, I/O, and impact on replication lag.