The migration failed on the second deploy. All logs pointed to one change: a new column.
Adding a new column to a production database sounds simple, but it can break scale, lock tables, and slow queries. Done wrong, it stalls deploys and causes downtime. Done right, it is invisible to users and safe for continuous delivery.
A new column changes the database schema. This can mean schema migrations, data backfills, default values, and constraint updates. The key risks:
- Table locks during schema changes, blocking reads or writes.
- Long-running migrations on large datasets.
- Incompatible changes with existing application code.
To add a new column safely, first deploy a schema change that makes the column nullable with no default. This is fast because it only updates the table metadata. Then roll out application changes to write data into the column. Next, backfill data in small batches. Avoid table-wide locks by using background jobs or chunked updates. After the backfill completes, add constraints or defaults in a separate migration.