How to Configure Haystack Kubernetes Read Only Access

Published
Length
3 min read

Luis Felipe Dias

All posts

You set up haystack kubernetes read only access by binding a read role to your pipeline. This role lets the AI agent read cluster state and logs while blocking changes. Many teams start by giving their Haystack app a broad cluster admin role to test things. Broad permissions create big risks when an AI pipeline runs unexpected requests against live clusters. If a prompt causes the agent to delete a pod, the cluster will allow it. How haystack kubernetes read only access works in native RBAC T

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You set up haystack kubernetes read only access by binding a read role to your pipeline. This role lets the AI agent read cluster state and logs while blocking changes. Many teams start by giving their Haystack app a broad cluster admin role to test things. Broad permissions create big risks when an AI pipeline runs unexpected requests against live clusters. If a prompt causes the agent to delete a pod, the cluster will allow it.

How haystack kubernetes read only access works in native RBAC

To lock down haystack kubernetes read only access, you must use native role-based rules. Kubernetes RBAC uses roles and bindings to control what an app or user can do. Every rule in this system is additive, which means there are no deny rules. You start with zero permissions and add only the exact read actions your app needs.

First, you choose between a Role and a ClusterRole for your Haystack service account. A Role sets permissions inside one single namespace. A ClusterRole is not tied to one namespace. It can cover cluster-wide objects like nodes, or resources across all namespaces. If your Haystack agent only checks one app team, use a standard Role. If it needs to check health across the whole cluster, use a ClusterRole.

Next, you define the exact resources and verbs in that role. For read access, you allow verbs like get, list, and watch on pods and deployments. Your Haystack pipeline will also need to read pod logs to help debug crashes. In Kubernetes RBAC, pod logs are a subresource written with a slash after the pod resource. You must grant read verbs to both the pod resource and the pod log subresource.

Finally, you link the role to your Haystack identity with a binding. A RoleBinding grants permissions inside one namespace and can point to a Role or ClusterRole. A ClusterRoleBinding grants those permissions across the entire cluster. Once bound, the API server checks every Haystack request against these additive rules.

Where native Kubernetes read roles fall short

Native RBAC is great at stopping a Haystack agent from changing or deleting cluster objects. It decides who can log in and which API paths they can read. Still, native RBAC does not inspect the actual data inside the response payload. Once a role grants access to read pod logs or secrets, the API returns everything.

Pod logs often hold sensitive user data that an AI pipeline should not ingest. A crash log might print customer emails, phone numbers, card numbers, or national IDs. When Haystack pulls those logs, that raw data goes straight into your language model prompts. Native RBAC cannot rewrite matched fields in results before they reach the AI agent.

Auditor Read-Only Access + Kubernetes API Server Access: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

Another gap appears if you ever grant access to the pod exec subresource for debugging. The API server checks if the role allows creating an exec session on a pod. It does not check the specific shell commands that run inside that open stream. Also, RBAC rules cannot require a human to approve a single request in real time. Teams end up managing dozens of static roles and bindings across every single cluster.

Scoping service accounts for AI pipelines

When you run Haystack outside the cluster, you should avoid sharing user credential files. Create a dedicated service account for the Haystack app inside a specific namespace. Bind that account only to the read role you built earlier. Do not add the secret resource to that role unless the pipeline truly needs it.

Keep the pod exec subresource out of your read role as well. Even if an agent only runs a read command inside a container, exec allows writes. Stick to get, list, and watch verbs on workloads, events, and the pod log subresource. Review your bindings often to make sure test permissions do not stay in production.

Check Kubernetes API requests and responses on the wire

Kubernetes RBAC decides who may call the API server. RBAC does not check response payloads or split listing Secret names from reading Secret values when a role grants get access. With hoop.dev, you run an open-source HTTP proxy, a sidecar for your Kubernetes API. The kubectl client changes only the server address it talks to while RBAC stays in place.

On each call, hoop.dev guardrails read the method, normalized path, and headers to refuse bad deletes or full Secret reads with a 403 error. On the way back, hoop.dev data masking walks JSON responses by key and masks matched values in Secrets, ConfigMaps, and pod logs. Every request and verdict is logged without storing bearer tokens. View the project on GitHub.

Frequently asked questions

Can I add a deny rule to block Haystack from reading secrets?

No, Kubernetes RBAC rules are purely additive and do not support deny rules. To block secret access, you must leave the secret resource out of every bound role.

How do I let Haystack read logs across all namespaces?

Create a ClusterRole that grants get and list verbs on pods and the log subresource. Then attach it to the Haystack service account using a ClusterRoleBinding.

Does read access to pods prevent sensitive data from reaching the LLM?

No, pod logs often contain emails, phone numbers, card numbers, or national IDs. You need data masking in flight to rewrite matched fields in results before model ingestion.

The query runs. The emails stay hidden.