The tables were clean. But the new column was missing.
Adding a new column should be simple. It often is. Yet in production, it can break query plans, lock writes, and stall deployments. A single schema change can jam the entire pipeline if handled without care.
The best practice is to design your schema changes so they are safe, reversible, and do not block critical operations. Start by creating the new column with a neutral default or allowing null values. This prevents existing rows from failing during the migration. Avoid adding NOT NULL constraints until after backfilling data.
Run backfills in controlled batches. Use indexed lookups when possible to prevent full table scans. Watch performance metrics during execution. On large datasets, schedule these changes during low-traffic windows or use an online schema change tool to avoid downtime.