Break Glass Process
You set up a break glass process by creating dedicated emergency roles with strict alerts. Engineers claim these roles during an outage when normal login systems fail or lack rights. Teams often build this workflow inside their cloud console or identity tool. They keep a few high-level accounts dormant until a severe incident happens. When an app goes down at night, the on-call engineer claims the emergency role. Most teams grant full admin rights to that role so the engineer can fix anything.
Break-Glass Access Procedures: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up a break glass process by creating dedicated emergency roles with strict alerts. Engineers claim these roles during an outage when normal login systems fail or lack rights.
Teams often build this workflow inside their cloud console or identity tool. They keep a few high-level accounts dormant until a severe incident happens. When an app goes down at night, the on-call engineer claims the emergency role. Most teams grant full admin rights to that role so the engineer can fix anything. Broad permissions help restore service fast, but they create major risks during high-stress events. An exhausted engineer with full admin rights can run a bad command and drop a table. They might also query production tables and pull thousands of customer emails and card numbers.
How a native break glass process works
A standard break glass process relies on isolated identity paths and clear alert rules. First, the security team creates an emergency role in the cloud provider or database. This role bypasses the normal single sign-on provider in case that login tool is down. Instead, the team stores the backup credentials or hardware keys in a safe place. Some teams use an automated ticket tool that assigns the role when an incident starts.
When an outage hits, the on-call engineer opens an emergency ticket and states the reason. The workflow grants the emergency role to that user for a set number of hours. At the exact same moment, the system sends high-priority alerts to the security team. Every login event writes a record to a central log storage bucket. When the timer runs out, the identity system revokes the role and cuts off permissions.
What native emergency roles solve and what they miss
Native emergency roles solve the problem of getting locked out during a major system failure. They also stop engineers from holding full admin rights during normal workdays. The security team gets a clear log of who claimed the role and when.
Still, this setup only controls who logs in to the host or database. Once the session starts, the native role cannot check or block individual commands in flight. A user with an emergency database role can run a delete query without a second check. They can also run a broad select query that returns raw phone numbers and national IDs. Masking does not run on the native role to rewrite matched fields in results. Native login logs show that a session happened, not what each command changed or read. Teams have to dig through scattered server logs after the incident to review every action.
Break-Glass Access Procedures: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
How to review and test your emergency workflow
An emergency workflow fails if engineers only read the steps during a real crisis. Teams should test their emergency roles every quarter during planned game days. Have the on-call engineer claim the role and run a safe test command. Check that the alert fires right away and reaches the security channel. Verify that the role drops off when the clock expires.
After every real or test event, run a quick review with the team. Check the logs to see every query and script the engineer ran during the session. If the engineer only needed to restart one service, shrink the role permissions to match. Keeping emergency scopes small lowers the chance of an accidental data drop during the next outage.
Add runtime guardrails and masking beside your databases and APIs
Login rules, role grants, and CI checks govern who connects to your stack. Yet any allowed script, engineer, or AI agent still runs commands with the full scope of its role. For workloads on Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes, SSH, or internal HTTP services, hoop.dev sits beside the service as an open-source proxy, a sidecar for your database or API.
Before a query runs, hoop.dev guardrails check the statement against your rules. An AI Analyzer you configure can score intent or hold risky calls for human review. Next, hoop.dev data masking rewrites sensitive fields in responses, and each verdict is logged per listener. Explore the MIT-licensed project on GitHub.
Frequently asked questions
When should an engineer trigger a break glass process?
An engineer should use it only during a severe outage or active security incident. If normal login tools work and have enough rights, they must use standard paths.
How long should emergency role permissions stay active?
Emergency permissions should last only for a short window, such as one to four hours. If the fix takes longer, the engineer can renew the request with a peer check.
What happens after an emergency session ends?
The system revokes the role and rotates any temporary keys or passwords right away. The security team then reviews the session logs against the incident ticket before closing it.