The database froze. Queries piled up. A missing new column was the cause.
Adding a new column seems simple. In production, it can lock tables, delay writes, and break services. Schema changes demand care. The wrong migration can take down critical systems.
To add a new column without downtime, design the change in stages. First, add the column with a default of NULL and no constraints. This prevents a full table rewrite. Then backfill data in small batches, monitoring performance. Finally, apply constraints or defaults once the column is populated.
For high-traffic databases, use tools like pt-online-schema-change or native partitioning features. On PostgreSQL, ADD COLUMN is usually instant unless you set a default. On MySQL, behavior depends on the storage engine and version. Always test changes in a staging environment with production-like data volume.