Kubernetes Workload Monitoring Without Datadog

Published
Length
3 min read

Luis Felipe Dias

All posts

You can set up kubernetes workload monitoring without datadog using native cluster logs and metrics. Teams collect pod status, resource usage, and app events through built-in API endpoints. Many teams start by giving engineers broad cluster roles to read pod logs and check workloads. When those permissions are too wide, anyone can run commands inside live containers or read sensitive keys. How kubernetes workload monitoring without datadog works Kubernetes tracks workload health and container

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You can set up kubernetes workload monitoring without datadog using native cluster logs and metrics. Teams collect pod status, resource usage, and app events through built-in API endpoints. Many teams start by giving engineers broad cluster roles to read pod logs and check workloads. When those permissions are too wide, anyone can run commands inside live containers or read sensitive keys.

How kubernetes workload monitoring without datadog works

Kubernetes tracks workload health and container output through its own control plane API server. Teams do not need a paid vendor agent to read pod state or container logs. Instead, open collectors or engineers query the API server for pod status and log streams. Access to these endpoints relies on role-based access control, which uses purely additive permission rules.

In this model, a Role sets permissions inside a single namespace for specific team members. A ClusterRole sets permissions across the whole cluster, including nodes and all namespaces at once. Teams link these rules to users or service accounts with a RoleBinding or a ClusterRoleBinding. Because there are no deny rules, every binding adds more power to a user account.

Reading workload output requires access to specific API subresources rather than just top-level pod objects. For example, reading container output requires permission on the pod log subresource inside that namespace. When a collector or engineer needs to check a failing app, they request that subresource. If they need to run a check inside a container, that uses the pod exec subresource.

Why native roles leave gaps during live checks

Native roles work well for automated metric collectors that only read pod status and logs. You can bind a read-only role to a service account and collect cluster data safely. The trouble starts when an alert fires and a human engineer must check a broken pod. Logs alone often do not show why a process hangs or why a network call fails.

At that point, the engineer needs interactive access to run diagnostic commands inside the workload. In native Kubernetes, granting the pod exec subresource is an all-or-nothing choice for that namespace. Once a user has exec rights, the API server allows any command inside those containers. The native role checks who logs in, not what commands they run inside the shell.

Even read-only log access brings risk when apps print raw customer records to standard output. A pod log stream might include emails, phone numbers, card numbers, or national IDs in plain text. Native RBAC cannot filter or rewrite those fields before the log reaches the engineer's screen. It only decides whether the user can open the log stream for that pod.

Datadog Security Monitoring + Kubernetes RBAC: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

Separating automated metrics from human workload checks

A clean monitoring setup splits background metric collection from human interactive checks on live workloads. Your open source collectors should use narrow service accounts that only read metrics and pod states. They never need permission to run commands or read secret objects stored in the cluster. This keeps your daily monitoring pipeline simple, cheap, and safe from broad credential leaks.

For human engineers, teams should avoid permanent cluster-wide bindings that allow unrestricted interactive container shells. Granting a ClusterRoleBinding with exec rights gives one account control over every workload in production. Instead, scope human roles to single namespaces and review what subresources each role actually includes. Still, native RBAC alone cannot inspect single commands or mask sensitive fields in log output.

Check Kubernetes API requests and responses on the wire

Kubernetes RBAC decides who may call the API server. RBAC does not check response payloads or split listing Secret names from reading Secret values when a role grants get access. With hoop.dev, you run an open-source proxy, a sidecar for your Kubernetes API, inline on the API connection. On every kubectl or API call an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.

On each call, hoop.dev guardrails read the method, normalized path, and headers to refuse bad deletes or full Secret reads with a 403 error, or hold risky calls for approval. On the way back, hoop.dev data masking walks JSON responses by key and masks matched values in Secrets, ConfigMaps, and pod logs in memory. View the project on GitHub.

Frequently asked questions

Can you read pod logs without giving full pod access?

Yes, Kubernetes treats pod logs as a separate API subresource from the main pod object. You can create a Role that allows reading the log subresource without allowing pod edits or deletions.

Why is the pod exec subresource risky for monitoring?

The exec subresource lets a user run any command inside a running container without limits. Native Kubernetes permissions are purely additive and cannot block risky commands once the shell opens.

Should monitoring tools use a Role or a ClusterRole?

Use a Role with a RoleBinding when a team only monitors workloads in one namespace. Use a ClusterRole when a central collector must read node metrics or pods across all namespaces.

Short role checks help teams work fast. Simple rules make each review clear. Good logs keep every team safe. Scoped keys stop bad changes early.

Teams test new role rules in staging first. Clear owner lists speed up each audit. Small role scopes cut down on risk. Regular log checks catch drift right away.

The query runs. The emails stay hidden.