The schema change went live at midnight. By morning, the first query was already breaking. A single new column had shifted execution plans, altered indexes, and changed assumptions buried deep in the code.
Adding a new column sounds simple. In practice, it touches more systems than expected. It impacts the database schema, the ORM mapping, API contracts, caching layers, and downstream consumers. Backfill strategies must be planned to avoid locking tables or spiking latency. Migrations need to run without blocking reads or writes.
Before creating a new column, define its data type with precision. Consider constraints, nullability, and defaults. Adding a NULL column with no default is fast in most database engines. Adding a column with a non-null default can rewrite the entire table, causing downtime under load. Use online DDL where possible. In MySQL, ALGORITHM=INPLACE and in Postgres, ADD COLUMN with a default can be safe if the DB version supports metadata-only changes.
Version your schema changes. Coordinate deployments so that application code can handle the presence or absence of the new column. Deploy schema changes before the code that writes to them. Deploy the reads afterward. This avoids race conditions during rollout.