Kubernetes Rook Ceph

Published
Length
3 min read

Luis Felipe Dias

All posts

You run a kubernetes rook ceph setup by deploying the operator to manage storage nodes. This setup turns raw node disks into shared block, file, and object pools for apps. Teams often grant broad cluster roles so operators and engineers can check storage health. When those roles are too wide, any user with pod access can alter live storage pools. A careful setup pairs storage deployment with strict role rules in the storage namespace. How a kubernetes rook ceph architecture works Rook runs as

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You run a kubernetes rook ceph setup by deploying the operator to manage storage nodes. This setup turns raw node disks into shared block, file, and object pools for apps. Teams often grant broad cluster roles so operators and engineers can check storage health. When those roles are too wide, any user with pod access can alter live storage pools. A careful setup pairs storage deployment with strict role rules in the storage namespace.

How a kubernetes rook ceph architecture works

Rook runs as an operator inside your cluster and watches custom storage resources. When you request a cluster, the operator starts Ceph daemons as normal pods. Monitor pods track cluster state while storage pods manage raw disks on your nodes. App teams do not manage these daemons or touch the storage pods. Instead, apps claim storage through standard persistent volume requests in their own namespaces.

The operator needs permissions to watch nodes, create pods, and manage storage secrets. Kubernetes RBAC controls these permissions through Role and ClusterRole objects. In Kubernetes RBAC, rules are purely additive, which means there are no deny rules. A Role sets permissions inside one namespace, such as the namespace holding your storage pods. A ClusterRole covers cluster-wide items like nodes, or resources across all namespaces. You attach these rules to users or service accounts with a RoleBinding or ClusterRoleBinding.

Managing operator and admin permissions

During incidents, engineers often use a toolbox pod to inspect the storage cluster. To run commands inside that pod, a user needs permission on the pod exec subresource. Kubernetes RBAC names subresources with a slash, such as the pod log or exec path. If you grant a ClusterRoleBinding for exec, that user can enter pods in every namespace. It is safer to use a RoleBinding scoped only to the storage namespace.

Even with a namespace Role, native RBAC has limits on subresource rules. You cannot use resource name lists on create verbs for subresources to restrict specific pods. Once an engineer can create an exec session in that namespace, they can reach any pod there. They also get full access to the storage admin key mounted inside the toolbox pod. From that shell, a user can delete pools, change replication rules, or read stored secrets.

Kubernetes RBAC: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

What native roles solve and what keeps the session active

Native roles and bindings decide who can talk to the cluster API server. They keep regular app developers out of the storage namespace and protect node resources. They also let you separate read-only log viewing from interactive pod shell access. This setup handles initial entry control for both service accounts and human engineers.

Still, native RBAC stops checking once the pod exec session starts. The API server knows a user opened a shell, but not which storage commands ran. It cannot block a destructive pool deletion command while allowing a safe health status check. It also cannot mask sensitive data like emails or card numbers returned in command output. Teams that need command-level control and full audit trails must inspect actions in flight.

Check Kubernetes API requests and responses on the wire

Kubernetes RBAC decides who may call the API server. RBAC does not check response payloads or split listing Secret names from reading Secret values when a role grants get access. With hoop.dev, you run an open-source HTTP proxy, a sidecar for your Kubernetes API. The kubectl client changes only the server address it talks to while RBAC stays in place.

On each call, hoop.dev guardrails read the method, normalized path, and headers to refuse bad deletes or full Secret reads with a 403 error. On the way back, hoop.dev data masking walks JSON responses by key and masks matched values in Secrets, ConfigMaps, and pod logs. Every request and verdict is logged without storing bearer tokens. View the project on GitHub.

Frequently asked questions

How do apps use storage from the cluster?

Apps request storage by creating standard persistent volume claims in their own namespaces. The storage driver provisions the volume and mounts it into the app pod.

Why do engineers use a toolbox pod for storage checks?

The toolbox pod holds the command tools and keys needed to check cluster health. Engineers open a shell in this pod to run status checks and fix disk issues.

Can Kubernetes RBAC block specific storage commands inside a pod?

No, Kubernetes RBAC only controls whether a user can start a pod exec session. Once the shell opens, RBAC cannot filter or log individual commands run inside the pod.

The query runs. The emails stay hidden.