When streaming pipelines run without any hidden model calls, teams can trust that data flowing through Kafka, Flink, or Spark is processed exactly as designed, compliance reports match reality, and accidental leakage of proprietary prompts never happens. Shadow AI, the practice of embedding undocumented LLM calls in streaming jobs, disappears entirely.
In practice, many organizations embed third‑party LLM APIs directly into their streaming jobs. Engineers often copy a curl command or a client library call into a Flink operator, stash the API key in a config file, and push the code to production. The model call travels straight from the worker node to the external provider, bypassing any internal policy layer. Because the request is not mediated, there is no audit trail, no real‑time inspection of the payload, and no way to enforce data‑masking or just‑in‑time approval. The result is a “shadow AI” – an invisible, undocumented use of generative models that can exfiltrate PII, violate licensing terms, or introduce unpredictable behavior into the data stream.
Why detecting shadow AI alone is not enough
The first step toward solving the problem is to make the existence of hidden model calls observable. Teams can instrument their code to log every HTTP request, but those logs live on the same host that runs the streaming job. If an attacker compromises the host, the logs can be altered or deleted, and the very act of logging does not stop the request from reaching the model. Moreover, simply knowing that a call happened does not prevent the call from sending sensitive data or from executing a high‑risk operation. What remains missing is a control point where the request can be inspected, approved, or blocked before it ever leaves the network.
That control point must sit in the data path – the only place where enforcement can reliably happen. It must be independent of the streaming runtime, independent of the host’s operating system, and capable of applying policy decisions in real time. Only a gateway that proxies the connection can provide the necessary visibility and guardrails while still allowing the existing streaming code to use its standard client libraries.
hoop.dev as the identity‑aware gateway for streaming workloads
hoop.dev fulfills the architectural requirement by acting as a Layer 7 gateway that sits between the streaming job and the external LLM endpoint. The gateway is deployed as a network‑resident agent inside the same VPC or data‑center segment as the streaming workers. Identity is handled via OIDC or SAML, so each request carries a token that hoop.dev validates before any traffic is allowed through. Because the gateway is the only component that can see the full request and response, it is the sole source of enforcement outcomes.
Setup – Engineers provision an OIDC client for hoop.dev, assign groups that represent different risk levels, and configure the streaming job to point at the gateway’s address instead of the raw LLM URL. The gateway stores the provider credential, so the job never sees the secret.
The data path – All HTTP traffic destined for the model API is forced through hoop.dev. The gateway parses the request body, extracts any fields that match a configured mask pattern (for example, credit‑card numbers or customer IDs), and either redacts them or replaces them with a token before forwarding the call. The response undergoes the same inspection, allowing the system to hide generated PII before it reaches downstream operators.
Enforcement outcomes – hoop.dev records each session, creating an audit record that includes who initiated the call, which streaming job made the request, and the exact payload after masking. If a request contains a high‑risk prompt (such as a request to generate code that could modify infrastructure), the gateway can trigger a just‑in‑time approval workflow, pausing the call until a designated reviewer approves it. For calls that violate policy, hoop.dev can block the request outright, ensuring that no disallowed data ever reaches the model.
Because the gateway runs outside the streaming runtime, the enforcement cannot be bypassed by simply restarting a worker or editing a container image. The policy engine lives in the data path, and every network packet that carries a model call must pass through it.
Key benefits for streaming teams
- Visibility – Every shadow AI call is logged and searchable, turning an invisible risk into an auditable event.
- Data protection – Inline masking prevents accidental leakage of PII in prompts or generated text.
- Risk control – Just‑in‑time approvals let security teams gate high‑impact model usage without breaking CI pipelines.
- Forensics – Session recordings can be replayed to understand exactly what data was sent to the model and what was returned.
- Open source – The gateway is MIT licensed, so teams can inspect the code, extend policies, or run it in air‑gapped environments.
Getting started is straightforward. Follow the getting‑started guide to deploy the gateway, then use the learn section to define masking rules and approval policies that match your streaming use case.
FAQ
What is shadow AI?
Shadow AI refers to the use of generative‑AI services that are not documented, not governed by central policy, and therefore operate invisibly within an organization’s workloads. In streaming, this often appears as undocumented HTTP calls from a data‑processing job to an external LLM.
How does hoop.dev detect hidden model calls?
hoop.dev inspects every Layer 7 request that passes through it. By matching request patterns, URL paths, and payload signatures, it can flag calls that target known LLM endpoints even if the client code tries to hide the destination. The detection happens before any data leaves the network, and the result is recorded in the session log.
Will routing traffic through hoop.dev add noticeable latency?
Because hoop.dev operates at the protocol layer and performs only lightweight parsing, masking, and policy checks, the added latency is typically measured in low‑single‑digit milliseconds. For most streaming workloads, this overhead is negligible compared to the network round‑trip to the external model provider.
To explore the code or contribute, visit the GitHub repository. The project includes example configurations for streaming environments and detailed documentation on extending the gateway’s policy engine.
