The query hit production at 02:13. We traced it. The log showed a missing column. The fix was simple but the impact was not.
Adding a new column to an existing database table is routine, but it can break systems if done carelessly. Schema changes propagate through APIs, background jobs, caches, and analytics pipelines. A ALTER TABLE may lock rows, spike CPU, or conflict with replication lag.
Before adding a new column, define its purpose and data type with precision. Check for nullability, default values, and indexing needs. Use migrations that are version-controlled and reviewed. Avoid wide tables unless the data model demands it. Keep column names consistent with existing conventions to prevent confusion in future queries.
Test the new column in staging with realistic load. Run both read and write operations to measure latency. Verify that ORM models, serializers, and database views handle the column correctly. In distributed systems, ensure all services using the table can tolerate the change before deployment.