Kafka Metrics Prometheus
You set up a kafka metrics prometheus pipeline by attaching a JMX exporter to each broker. Prometheus then scrapes those endpoints on a set schedule to collect broker and topic stats. Teams often build this pipeline fast so they can watch lag and disk usage. Still, they often pair this setup with broad cluster admin roles for the on-call team. When an alert fires, engineers log in with shared keys that can alter or delete topics. How the native kafka metrics prometheus workflow operates Apach
Security Metrics & KPIs + Prometheus Security: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up a kafka metrics prometheus pipeline by attaching a JMX exporter to each broker. Prometheus then scrapes those endpoints on a set schedule to collect broker and topic stats. Teams often build this pipeline fast so they can watch lag and disk usage. Still, they often pair this setup with broad cluster admin roles for the on-call team. When an alert fires, engineers log in with shared keys that can alter or delete topics.
How the native kafka metrics prometheus workflow operates
Apache Kafka runs on the Java Virtual Machine and exposes internal stats through JMX beans. Prometheus cannot read JMX data directly out of the box. To bridge this gap, teams run a JMX exporter agent inside the broker process. You can also run an independent exporter alongside the cluster to track consumer group lag. The exporter reads the local JMX beans and translates them into plain text counters and gauges.
Prometheus uses a pull model to gather these numbers over HTTP. You define the broker network targets in your scrape rules or use service discovery. At each scrape interval, Prometheus sends a read request to the exporter port on every broker. The server stores these time series values on disk so teams can query trends and trigger alerts.
Which Kafka metrics matter most for cluster health
A healthy cluster requires tracking three main groups of data across brokers, producers, and consumers. First, broker health depends on under-replicated partitions and active controller counts. If a broker drops out, under-replicated partitions rise right away, which warns you of data loss risks. Offline partition counts are also critical because they show that clients cannot read or write to a topic.
Second, throughput metrics track bytes coming in and going out per second across your topics. Spikes in network processor idle ratios or request handler usage show when brokers hit CPU limits. Disk space metrics also help you spot brokers that need faster storage or shorter retention rules.
Third, consumer lag measures how far behind a reader app is from the latest written offset. High lag means your consumer group cannot keep up with incoming messages. Tracking request latency alongside lag tells you if the slowdown is in the broker or the app.
Security Metrics & KPIs + Prometheus Security: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What the native setup solves and where the security gap hides
A native metrics setup gives your team clear charts and fast alerts when a broker slows down. You can protect the scrape ports with network rules and basic login keys so only Prometheus reads them. That part of the architecture works well and keeps health stats safe from outside eyes.
The real operational gap starts right after an alert wakes up an engineer. Charts only show that a partition is stuck or consumer lag is growing. To fix the problem, an engineer must log in to the cluster or broker host to run admin tools. Native broker roles decide who can log in, not what individual commands do in flight. Teams often grant full cluster admin rights so engineers can inspect topics and reset offsets during an outage. If an engineer runs a bad command or queries live topic payloads, raw customer data such as emails or card numbers can spill into local terminals without a clean user log.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
Should I use the JMX agent or a standalone lag exporter?
Most teams run both tools together for complete coverage. The in-process JMX agent is best for internal broker stats like memory, disk, and replication health. A separate lag exporter is better for reading consumer group offsets stored in internal topics.
How often should Prometheus scrape Kafka brokers?
A scrape interval between fifteen and sixty seconds works well for most production clusters. Scraping too often adds extra CPU load to the JMX exporter when you have thousands of partitions. Longer intervals reduce broker load while still catching replication drops and lag spikes on time.
Why do high partition counts slow down metrics scraping?
Every topic and partition creates new JMX beans inside the broker process. When a cluster has thousands of partitions, the exporter must read and format millions of data points. You can fix this by filtering out per-partition metrics that your alerts do not use.