The query ran, and the schema broke. A missing new column had taken down the release.
Adding a new column in a live database is never just an ALTER TABLE. Done wrong, it can lock production, trigger replication lag, or corrupt critical data. Done right, it preserves uptime, keeps migrations predictable, and avoids rollback chaos.
The safest workflow starts with explicit planning. Define the new column with clear defaults and nullability rules. Avoid setting a NOT NULL constraint with a default on huge tables—it will rewrite every row. Instead, add the column as nullable, backfill in controlled batches, then apply constraints in a second migration.
Performance comes from minimizing table rewrites. Use ADD COLUMN without defaults where possible, and run data backfill in small transactions to avoid blowing up locks and logs. Indexes should be added only after the data is stable, as indexing during backfill can choke throughput.