The migration finished at 03:17. The tests were green. But a single missing new column in the schema brought the deploy to a halt.
A new column seems simple: add it, run the migration, push to production. In reality, it demands precision. Database changes touch persistence, APIs, caching, and background jobs. Denormalized stores can double your surface area for bugs. Adding a new column to an existing table without a plan for data backfill and index updates risks downtime and data loss.
Start with the migration file. Define the new column with explicit data types and constraints. Avoid nullable fields unless they have a clear business case. Set sensible defaults to simplify deployment. If default values require computation, consider a separate backfill migration to prevent locking large tables for extended periods.
In distributed systems, schema changes must align with application code that supports both the old and new column states during rollout. This means deploying code that writes to both columns before switching reads to the new column. For zero downtime, make changes in phases: add the new column, dual-write, validate data integrity, then remove old fields.