The query returned fast, but the schema had already moved on. A missing field. A silent error. A stalled deployment. All because the database needed a new column.
Adding a new column should be simple. In production, it can be brutal. Locks. Migrations that run too long. Changes that ripple through the app and break downstream consumers. Teams fight this daily across SQL and NoSQL systems.
The correct approach starts with defining the migration plan. For relational databases, use ALTER TABLE with care. Avoid blocking writes on large tables by using online schema change tools like gh-ost or pt-online-schema-change. For NoSQL, add the new field at the application layer first. Let writes start populating it before any read logic depends on it.
Version your database access code. Deploy the write path for the new column before the read path. Keep both code paths live until backfill is complete. For backfills, run incremental batches to avoid load spikes. Monitor replication lag and lock durations in real time.