The fix was simple. The impact, massive. Adding a new column can break production if it’s done without precision.
A new column changes the shape of your data. It affects queries, indexes, and constraints. It can trigger unexpected load on replication. It can lock tables and block writes. On large datasets, even a single ALTER TABLE can lead to hours of degraded performance.
Before adding a new column, measure the size of the table and assess downtime risk. Use tools or migration frameworks that support online schema changes. Split changes into phases:
- Create the column with no default to avoid full table rewrites.
- Backfill in controlled batches.
- Add constraints or defaults after the data is in place.
Test the migration on a staging environment that mirrors production traffic and data scale. Confirm indexes still work as expected. Watch query plans for regressions. Monitor replication lag closely.