All posts

Audit logging for AI agents on Kubernetes

Start with the requirement, because it decides everything else. Audit logging for AI agents on Kubernetes is only worth keeping if the record is produced by something the agent cannot reach. If the agent writes its own log of which pods it touched and which commands it ran, a compromised or confused agent can edit that record before anyone reads it. The log then describes what the agent wants you to believe, not what happened. So the design question is not which fields to capture. It is where t

Free White Paper

K8s Audit Logging + AI Audit Trails: The Complete Guide

Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.

Free. No spam. Unsubscribe anytime.

Start with the requirement, because it decides everything else. Audit logging for AI agents on Kubernetes is only worth keeping if the record is produced by something the agent cannot reach. If the agent writes its own log of which pods it touched and which commands it ran, a compromised or confused agent can edit that record before anyone reads it. The log then describes what the agent wants you to believe, not what happened.

So the design question is not which fields to capture. It is where the record gets written, relative to the thing being recorded.

The requirement, stated plainly

An agent that operates a cluster reaches the Kubernetes API server: it runs kubectl, opens an exec into a pod, reads secrets, scales a deployment. The Kubernetes API server has its own audit log, and it is valuable. But it sits inside the same cluster the agent is acting on, and it records that a request arrived from a credential, not which task drove the agent to make it. For agent audit logging the requirement is stricter: the record must be written at the point access is brokered, outside the agent process, and it must carry the identity and the task, not just the verb and the resource.

Three properties fall out of that requirement directly.

  • Out of reach. The log lives on the far side of the access boundary, where the agent has no write path. A log the agent can rewrite is a claim, not evidence.
  • Attributed. Every action ties to a distinct agent identity and the task that requested it, so "who and why" survive even when many agents share a service account inside the cluster.
  • Complete. Denials are recorded next to successes. An agent that tried to delete a namespace and was blocked is exactly the event an investigation needs, and most logs drop it.

Why the Kubernetes API audit log is necessary but not sufficient

The cluster audit log answers "what API activity happened here." It does not answer "which agent identity requested this access, for which task, and what was it denied." It also assumes the credential maps cleanly to an actor, which breaks the moment agents share a service account or a kubeconfig. And it lives where the workload runs, so the blast radius of a compromised agent can include the very record you would use to investigate it. Keep the API audit log. Do not make it the only thing standing between you and an unanswerable question.

Continue reading? Get the full guide.

K8s Audit Logging + AI Audit Trails: Architecture Patterns & Best Practices

Free. No spam. Unsubscribe anytime.

Meeting the requirement

Put an access boundary in front of the cluster and make the agent connect through it. That is the model hoop.dev is built to. The agent reaches the Kubernetes API through hoop.dev, a Layer 7 access gateway and identity-aware proxy that sits in front of the cluster's access path. Access is brokered per task, every command the agent issues passes through the gateway, and each session is recorded on the gateway side, outside the agent. Because the boundary is where authorization happens, it is also the natural place to write audit logging that the agent cannot touch. The learn pages on how access and recording fit together describe the model, and the getting-started docs for fronting a Kubernetes cluster cover registering the connection.

The point is not that hoop.dev adds another log. It is that audit logging belongs at the boundary that already brokers the access, so the record and the decision are produced by the same layer the agent does not control.

A concrete trace

An agent needs to restart a failing deployment. Through the boundary, the sequence is recorded as: agent identity billing-reconciler requested access for a stated task, policy matched it to a scope covering one namespace, the gateway brokered the connection, and the session recording shows the kubectl rollout restart and the pods that cycled. A second action in the same run tried to read a secret in kube-system and was denied. The audit log holds both. None of it depends on the agent reporting honestly, because the agent never wrote any of it.

A quick test of your current setup

Ask one question. If an agent were fully compromised right now, could it alter or delete the record of what it did in the cluster? If the only audit logging lives where the agent's credentials can reach, the answer is yes, and the rest of the schema barely matters. Move the record outside the agent first.

FAQ

Is the Kubernetes API server audit log enough for AI agents?

It is a strong cluster-level source, but on its own it logs API requests by credential, not the agent identity, the task, and the access decision behind them. Pair it with audit logging written at the access boundary, outside the agent.

What is the single most important property of an agent audit log?

That the agent cannot alter it. A log inside the agent's reach fails in precisely the scenario you keep logs for.

hoop.dev is open source. To put audit logging at the access boundary in front of your Kubernetes cluster, start with the repository on GitHub.

Open source

Save the open-source gateway for agent data access

Hoop is MIT-licensed infrastructure for controlling how AI agents reach production data. Star hoophq/hoop so you can inspect it, deploy it, or share it when your team starts governing agent access.

Star and save the repo →More posts