The build was failing again. The error came from a migration script that tried to add a new column to a live production table. Every deployment was stuck until someone fixed it.
Adding a new column sounds simple. In practice, it can cause downtime, locks, data inconsistencies, and broken queries if you move too fast. Large datasets make operations slow, and schema changes can block reads and writes. The safer path requires a plan.
First, decide if the new column can be nullable. Creating a nullable column is fast on most relational databases. If it must have a default value, set it null at creation, then backfill values in batches. Avoid writing a default at the schema level on creation, as it often incurs a full-table rewrite.
Second, deploy in phases. Phase one: add the column with no constraints. Phase two: run a background job to populate the column. Phase three: add indexes or constraints only after the data is ready. This minimizes lock time.