How to Configure K3s Break Glass Access
You set up k3s break glass access by binding an emergency role to a backup user. Teams use this path when their main login provider fails during an outage. Most teams start by sharing the local admin file that K3s creates at startup. This gives anyone with the file full control over every node and secret. Broad permissions make it easy to break production workloads by mistake. A safer setup uses dedicated emergency accounts with clear rules and audit logs. How k3s break glass access works in p
Your coding agents run with your access.
Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.
You set up k3s break glass access by binding an emergency role to a backup user. Teams use this path when their main login provider fails during an outage. Most teams start by sharing the local admin file that K3s creates at startup. This gives anyone with the file full control over every node and secret. Broad permissions make it easy to break production workloads by mistake. A safer setup uses dedicated emergency accounts with clear rules and audit logs.
How k3s break glass access works in practice
K3s writes a local admin file to the server disk when the cluster starts. This file holds a client certificate with full cluster admin rights. It bypasses external login systems like single sign-on or cloud identity providers. If your identity provider goes down, engineers can still use this local path. Still, sharing that single admin file across a whole team creates a huge risk. You cannot tell which engineer ran a command when everyone uses the same key.
To fix this, teams create separate client certificates or tokens for emergency use. You map these keys to specific roles inside the Kubernetes role-based access control system. An engineer requests the emergency key from a secure vault when an outage starts. The cluster checks the certificate signature and grants the role tied to that user. After the team fixes the outage, an admin revokes the certificate or rotates the key. This flow keeps the cluster reachable without handing out permanent admin keys to everyone.
Why you should restrict the default admin file
The default admin file on a K3s server node never expires on its own. It stays valid until you rotate the server certificates and restart the cluster service. If an engineer copies this file to a laptop, that laptop holds permanent root rights. Anyone who steals that file can read every secret or delete any running app. You must lock down file permissions on the host node right away.
Only the root user on the Linux host should read that local file. For remote emergencies, teams generate short-lived client certificates signed by the cluster certificate authority. You set a short lifetime on the certificate so it expires after a few hours. This approach stops old emergency keys from sitting on engineer laptops for months. You also give each emergency certificate a unique username for the cluster audit log. That way, the log shows who logged in during the outage.
Break-Glass Access Procedures + Customer Support Access to Production: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native cluster roles leave open
Native cluster roles and client certificates solve the problem of how to log in. They let an on-call engineer reach the API server when normal login tools fail. They also record the user identity in the basic Kubernetes API audit log. For basic server outages, this native path gives teams a working fallback plan.
Still, native roles only check who logs in and which resource they touch. Once an engineer holds an emergency admin role, the cluster trusts every single command. The native setup cannot pause a dangerous delete command to ask for a peer review. It also cannot mask sensitive fields like emails, phone numbers, or card numbers in output. If an engineer reads a secret or runs a shell in a pod, raw data appears. Native logs record that a pod shell opened, but not the exact commands run inside.
Govern kubectl and API calls in flight
Identity providers and Kubernetes RBAC decide who logs into a cluster. They cannot redact keys inside returned JSON objects or hold a risky delete call for review. By design, hoop.dev sits inline as an open-source proxy, a sidecar for your Kubernetes API, without replacing RBAC. On every production call from an engineer or AI agent, hoop.dev decides to pass, mask, block, or hold it for human approval.
First, hoop.dev guardrails refuse banned methods, paths, or headers before they run or hold risky deletes for review. Next, hoop.dev data masking rewrites sensitive keys in returned Secret, ConfigMap, and log responses in memory, and writes each decision to the audit trail. Explore the repo on GitHub.
Frequently asked questions
Where does K3s store the default admin credentials?
K3s saves the default admin file in the rancher folder on the server node. Only root users on that host machine can read the file by default.
How do you revoke an emergency client certificate in Kubernetes?
Kubernetes does not support standard certificate revocation lists out of the box. You must remove the user role binding or rotate the cluster certificate authority.
Can we audit commands run during a break glass session?
You can enable the native Kubernetes API audit log on the K3s server. Still, native logs only show API calls and miss commands typed inside container shells.
Related guides
In this post