The cause was simple: the missing new column.
Adding a new column to a production database is not just a schema change. It touches migrations, indexes, query performance, and every piece of code that reads or writes that table. Done poorly, it can lock tables, block requests, or corrupt data. Done well, it is fast, safe, and repeatable.
Start with the migration plan. Define the new column in a way that minimizes locking. In PostgreSQL, adding a nullable column without a default is near-instant. In MySQL, engine and version matter; test on a copy of the dataset.
If the column needs a default, set it in application logic, backfill in small batches, then add the database-level default. This avoids locking large tables for minutes or hours. Monitor query plans after the change. Adding an index for the new column might help, but every index carries write overhead.
Update the application code in steps. First, deploy code that can write to and read from both the old schema and the schema with the column present. After the migration runs, deploy code that depends on the new column. Always keep changes backward-compatible until the migration is complete.
For large datasets, use online schema change tools like pt-online-schema-change or gh-ost. For cloud-managed databases, check if the provider offers zero-downtime DDL.
Write tests that assert the new column exists, has expected constraints, and is populated correctly after backfill. Log metrics to confirm column usage in production.
A new column is a small change in code, but a big change in the lifecycle of data. Treat it with care, measure its impact, and automate the process so the next time is even safer.
See how schema changes like adding a new column can be deployed in minutes without downtime—try it now with hoop.dev.