Auditing Kubectl: A Practical Guide for Better Kubernetes Security

Kubectl is an essential command-line tool for interacting with Kubernetes clusters. While kubectl gives you powerful control over your infrastructure, it also introduces potential risks if not audited properly. Every kubectl command reflects an action affecting your cluster, whether it’s deploying a new pod or scaling a service. Without auditing, it’s hard to track who did what, when, and why.

This post explains how to conduct a robust audit of kubectl usage. You’ll learn why it’s critical, how to set it up, and the steps to ensure your Kubernetes environment is secure and manageable.

Why Auditing Kubectl Matters

Kubernetes is designed to manage containers at scale, but with great power comes complexity. When multiple users and automations interact with a cluster, a kubectl command executed recklessly can lead to security vulnerabilities, data loss, or broken applications. Auditing kubectl allows you to:

1. Track Changes: Know who modified what resources and when.
2. Enhance Security: Detect and mitigate unauthorized actions.
3. Ensure Compliance: Maintain records to meet security policies or regulatory standards.
4. Debug Faster: Troubleshoot issues by accessing a clear sequence of prior commands.

Enabling Audit Logs in Kubernetes

To audit kubectl commands effectively, you first need to configure Kubernetes to log API server requests. Every kubectl action is essentially an API request from the CLI to the cluster. Here’s how to enable audit logging:

Step 1: Configure an Audit Policy

Create an audit policy file, audit-policy.yaml, to control what gets logged. For example:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 - level: Metadata
 resources:
 - group: ""
 resources: ["pods", "services", "deployments"]
 - level: RequestResponse
 verbs: ["create", "update", "delete"]
 resources:
 - group: ""
 resources: ["configmaps"]

This policy focuses on logging metadata for common resources like pods, and includes request/response data for critical operations like changing ConfigMaps.

Step 2: Apply the Audit Policy to the API Server

Add the following flags to your Kubernetes API server configuration to enable the policy:

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log

Restart your API server to apply the changes.

Step 3: Review the Audit Logs

Audit logs will now be saved to the file path you specified. A typical log entry looks like this:

{
 "kind": "Event",
 "verb": "update",
 "user": {
 "username": "admin",
 "groups": ["system:masters"]
 },
 "objectRef": {
 "resource": "configmaps",
 "namespace": "default",
 "name": "app-config"
 },
 "responseStatus": {
 "code": 200
 },
 "timestamp": "2023-10-15T00:00:00Z"
}

You can parse these logs manually or forward them to monitoring tools for aggregation.

Tracking Historical Usage of Kubectl Commands

While API server audit logs are valuable, they don’t make it easy to focus specifically on kubectl usage or separate them from other sources, like automated controllers. To track kubectl usage more directly:

1. Leverage kubectl Plugins: Tools like kubectl-trace integrate with kubectl’s plugin framework to trace command usage.
2. Use RBAC Audits: Review role-based access control (RBAC) bindings to see who has rights to execute specific commands.
3. Analyze Shell Histories: On cluster management nodes or laptops, review ~/.kube/config and shell command histories to map user behavior.

While these methods offer supplemental insights, they can’t replace full API audits.

Automating and Visualizing Audits

Auditing is time-consuming unless you automate and visualize it. Tools like Prometheus with Loki or Elasticsearch with Kibana can ingest audit logs and create dashboards to track kubectl activity over time. You can filter by user, resource type, or time range to home in on suspicious or important actions.

For faster results, tools like Hoop.dev streamline this process by unifying your auditing and monitoring needs. Instead of starting from scratch, Hoop.dev lets you get audit-based insights into kubectl actions in just a few minutes.

Conclusion

Ensuring the security and reliability of your Kubernetes clusters starts with auditing kubectl. A clear audit trail helps you track changes, troubleshoot issues, and maintain compliance. By enabling Kubernetes audit logs, tracking commands, and automating analysis, you can make kubectl auditing manageable and impactful.

If you want seamless insights into kubectl activity, try Hoop.dev. It turns cumbersome audit log parsing into actionable outcomes—live in minutes.