Streaming pipelines that trust unmanaged service accounts are a prime vector for credential leakage, especially when non-human identity is not enforced.
Most teams wire a producer or consumer directly to Kafka, Kinesis, or Pulsar using a static key that lives in a config file or environment variable. The key grants unrestricted read or write rights across the entire topic namespace, and anyone who can read the container can extract it. Because the connection goes straight from the application to the broker, there is no central audit point, no ability to mask sensitive payloads, and no workflow to require human approval before a new consumer group is created.
What organizations really need is a way to represent services, CI jobs, or AI agents with cryptographically verifiable identities – the so‑called non‑human identity – while still limiting each identity to the exact streams it should touch. Providing a non‑human identity fixes the problem of shared secrets, but the request still reaches the broker directly. Without a gateway in the data path there is no place to enforce per‑topic access, no session recording, and no inline masking of payload fields that might contain PII.
Understanding non-human identity in streaming
A non‑human identity is an OIDC or SAML token issued to a service account, CI runner, or automated agent. The token conveys who the caller is, what groups it belongs to, and any additional attributes that can drive policy decisions. In the streaming context this identity can be mapped to a narrow set of topics, partitions, or consumer groups.
Setup steps – creating the service account in your identity provider, assigning it to a group that matches the allowed streams – decide who the request is and whether it may start. Those steps are necessary, but they do not enforce anything on the actual data flow.
Why a gateway is required
The only place to enforce the policy is the data path that carries the stream traffic. By inserting a Layer 7 gateway between the client and the broker you gain a single control surface that can:
- Validate the non‑human identity on every connection attempt.
- Apply per‑topic allow‑lists derived from group membership.
- Mask fields such as credit‑card numbers or social security numbers in messages before they are written to the topic.
- Require a just‑in‑time approval workflow when a new consumer group is requested.
- Record the full session – every publish and consume operation – for replay and audit.
hoop.dev sits in that data path. It proxies the streaming protocol, inspects each frame, and applies the controls listed above. Because the gateway holds the broker credentials, the client never sees them, eliminating the risk of credential leakage from container images.