Two production systems were locked. Access was blocked to everyone but a few who held break glass rights. Minutes mattered. The only thing between failure and recovery was a clear break glass access procedure, tied to the right user groups, logged, and safe from abuse.
Break glass access procedures define how to grant temporary elevated privileges when normal paths fail. Without them, emergencies turn into outages. Done well, they protect sensitive environments while allowing fast, auditable intervention. Done poorly, they open the door to security gaps and chaos.
It starts with defining dedicated break glass user groups. These groups should stand apart from normal administration roles. Membership must be minimal, often single digits, with approvals tracked. Accounts need hardened authentication, long random passwords, and multi-factor login. They must remain inactive except during an approved break glass event.
Next comes the procedure itself. It needs clear triggers: what counts as an emergency, who can authorize access, and how to request it. Every step must be explicit. The process should include identity verification, precise scoping of permissions, and an automatic time-to-live to revoke access quickly.