The alert fired at 2:07 a.m. A migration had failed. The logs showed a parser error. The root cause was simple: a new column had been added to a production table without a deployment plan.
A new column changes the shape of your database schema. It can break queries, invalidate caches, and cause silent data corruption. The speed of deployment does not matter if the schema is drifting. Every change must be safe, controlled, and reversible.
When adding a new column, start with a clear migration strategy. In relational databases, that often means an ALTER TABLE statement. Know your vendor’s behavior. PostgreSQL can add certain columns instantly, but others may lock the table. MySQL may rebuild the table, blocking writes. For high-volume systems, schedule schema changes during low traffic or use tools like gh-ost or pt-online-schema-change.
Default values require special care. Setting a default on a new column for millions of rows can lock the table for minutes. Instead, add the column as NULLable, backfill data in batches, then alter it to NOT NULL with a default when safe.