Microk8s Observability Addon

Published
Length
3 min read

Luis Felipe Dias

All posts

You turn on the microk8s observability addon with one built-in cluster command. This action starts metric, log, and trace tools across your nodes right away. Most teams set up this tool to check pod health and fix broken apps. To view dashboards, engineers often share static cluster keys or broad admin roles. When roles are too broad, anyone can read secret data or delete running pods. How the microk8s observability addon works The microk8s observability addon bundles popular open source moni

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You turn on the microk8s observability addon with one built-in cluster command. This action starts metric, log, and trace tools across your nodes right away. Most teams set up this tool to check pod health and fix broken apps. To view dashboards, engineers often share static cluster keys or broad admin roles. When roles are too broad, anyone can read secret data or delete running pods.

How the microk8s observability addon works

The microk8s observability addon bundles popular open source monitoring tools into one package. When you turn it on, the cluster creates a new space for monitoring pods. It sets up Prometheus to collect node and pod metrics every few seconds. It also starts Grafana so your team can view charts in a browser. Plus, it runs Loki and Tempo to store app logs and request traces.

All of these tools run inside your local cluster as standard workloads. They use internal service accounts to read state data from the Kubernetes API. To reach the dashboards from a laptop, engineers usually forward a local network port. An engineer must hold a valid cluster key to start that port forward. Teams also set up network rules to expose the web dashboards on internal networks.

Once inside the dashboard, a user logs in with a shared admin password. You can change this password by editing the cluster secret in the monitoring space. You can also tie the dashboard login to an external identity tool for single sign-on. Meanwhile, the Kubernetes role system controls who can touch the underlying monitoring pods.

What native cluster roles solve and what they miss

Native Kubernetes roles let you decide who can view or change cluster resources. You can give an engineer a read-only role for the monitoring space. This role stops them from deleting the metric store or changing alert rules. It also keeps unauthorized users from starting a port forward to the dashboard pod. Native roles work well for basic gatekeeping at the cluster entry point.

Still, a significant security gap keeps the session active during everyday cluster maintenance. Native roles only check who logs in and which general resource they touch. They do not inspect what individual commands or queries do in flight. For example, an engineer with port forward rights can query any log stream. Those app logs often hold raw customer emails, phone numbers, or card numbers.

AI Observability: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

Also, engineers often need shell access to fix a broken collector pod. Once Kubernetes grants shell access, the native role cannot limit specific shell commands. A user could run a bad script or read mounted secrets inside the pod. Standard cluster logs only show that a shell session started on that pod. They do not record the exact commands ran or the data returned.

Securing cluster commands and sensitive log data

To keep production data safe, teams must look beyond basic cluster roles. You need a way to check every command and query before it runs. Teams should tie every cluster action to a real person from their identity provider. Removing shared admin keys stops people from hiding behind a generic team account.

When engineers inspect pod logs, automated filters should protect private user records. Data masking rewrites matched fields like emails and card numbers in query results. This step lets developers fix app bugs without exposing real customer data. You can also require a quick peer approval before someone runs a pod shell. Recording every command in a central log makes security audits fast and clear.

Add request guardrails and response masking beside Kubernetes

Cluster roles and service accounts govern who reaches the Kubernetes API. Still, a broad role or agent can delete workloads or print sensitive values from ConfigMaps and logs. In practice, hoop.dev runs beside the API server as a sidecar for your Kubernetes API and checks every request and response alongside RBAC.

With hoop.dev Kubernetes controls, guardrails can allow listing Secret names while refusing reads of Secret data before execution. Data masking redacts matched keys in returned JSON responses and log streams in memory. Furthermore, hoop.dev holds SOC 2 Type II and is MIT licensed on GitHub.

Frequently asked questions

What tools come with the microk8s observability addon?

The addon installs Prometheus for metrics, Grafana for charts, and Loki for pod logs. It also sets up Tempo to track request traces across your cluster apps.

How do engineers reach the monitoring dashboards?

Engineers usually run a port forward command using their local cluster key. Some teams also set up a network rule to open the dashboard on a local network.

Why are Kubernetes roles not enough for safe cluster debugging?

Kubernetes roles only check who starts a session, not what commands run inside it. They cannot rewrite sensitive emails or card numbers inside pod log output.

The query runs. The emails stay hidden.