The query had been running for hours when the alert hit. Logs showed the schema change. Someone had added a new column.
Adding a new column should be simple. In small tables, it is. In production-scale systems, it can be dangerous. The cost is often hidden—locks, blocked writes, replication lag, or a sudden spike in CPU. A single schema migration can stall an entire service if it’s not planned and executed right.
A new column alters the table definition stored in metadata. Depending on the database engine, this can mean rewriting the table on disk. In MySQL with older storage formats, an ALTER TABLE ADD COLUMN can trigger a full table copy. PostgreSQL can handle nullable columns with defaults faster, but not all cases are instant. Large datasets magnify the risk.
To add a new column safely, measure the impact before running in production. Clone the schema, import a realistic dataset, and test the migration. Check how your database handles the operation with and without defaults. Review the execution plan and system metrics.