The migration ran without errors, but the data told a different story. A single missing new column had broken the feature.
Adding a new column to a production database is simple in theory and dangerous in practice. Schema changes are permanent, and they can lock tables, stall writes, or break downstream systems. The smallest detail—data type choice, default value, indexing—can decide whether the deployment is invisible or catastrophic.
The safest way to add a new column is to treat it as a multi-step operation. First, deploy the schema change without touching application logic. Use ALTER TABLE with NULL allowed and no default when adding large columns to high-traffic tables to avoid locking issues. For integer or boolean columns, add them with defaults only if the database supports non-blocking operations.
After the new column exists, backfill it in small batches. Use a job that runs on a low-priority queue to avoid load spikes. Once backfilled, add indexes only if queries require them. In high-scale systems, even index creation can block writes or consume massive I/O, so test on staging datasets before touching production.