How can you make sure that every inference request obeys the same security and compliance rules you write as policy as code? Today most data‑science teams run model servers behind a load balancer or expose them directly to internal networks. Engineers hand out static API keys, service accounts, or shared credentials to the model endpoint. The policy that governs which data may be sent, which responses may be returned, or which users may trigger a prediction is often documented in a repository, but the enforcement point lives somewhere else – if it exists at all. In practice, the request travels straight to the model, the server logs who called it, and the policy is only consulted after the fact, if someone remembers to check the logs.
Why policy as code matters for inference
Policy as code lets you express guardrails in a version‑controlled language: block personally identifiable information (PII) from being sent to a model, require human approval before a high‑risk prediction, or enforce rate limits per tenant. By treating the policy like any other piece of software, you gain review workflows, automated testing, and a single source of truth for compliance.
However, simply storing those rules in a Git repository does not protect the inference path. The request still reaches the model server with the original payload, and the server may execute the operation before any rule is examined. The setup – OIDC authentication, service‑account provisioning, and network segmentation – tells the system who is calling, but it does not guarantee that the request complies with the declared policy.
What remains open after defining policy as code
Even with a well‑structured policy repository, three gaps typically persist:
- Missing enforcement point. The model endpoint receives raw traffic; there is no proxy that can intercept and evaluate the request against the policy.
- No real‑time masking. Sensitive fields that slip through the client side remain visible in the model’s response, creating data‑leak risk.
- Insufficient audit. Without a session recorder, you cannot replay a prediction, verify who approved it, or prove compliance during an audit.
These gaps are not solved by identity configuration alone. The gateway that sits on the data path must be the place where the policy is actually applied.
hoop.dev as the data‑path enforcement layer
Enter hoop.dev. It is a Layer 7 gateway that sits between the inference client and the model server. All traffic flows through the gateway, giving it a unique position to enforce the policy you have codified.
When a request arrives, hoop.dev validates the caller’s OIDC token, extracts group membership, and then evaluates the request against the policy you have stored as code. If the payload contains a prohibited field, hoop.dev masks that field before it reaches the model. If the request exceeds a risk threshold, hoop.dev routes it to a human approver and only forwards it after explicit consent. For every prediction, hoop.dev records the full session – request, response, and approval decision – so you can replay the exact interaction later.
Because hoop.dev holds the credentials for the model server, the client never sees them. This eliminates credential sprawl and ensures that the only entity capable of issuing a prediction is the gateway, which is itself governed by the policy you authored.
Key watch points when applying policy as code to inference
While hoop.dev provides the enforcement surface, you still need to consider the following when designing your policy:
- Data classification. Clearly label which fields are PII, PHI, or other regulated data. The policy engine can then reference those labels for masking decisions.
- Risk scoring. Define a scoring model that quantifies the sensitivity of a request. Use that score to trigger just‑in‑time approvals.
- Version alignment. Keep the policy repository in sync with the gateway’s configuration. A mismatch can cause false‑positives or gaps in protection.
- Performance impact. Real‑time inspection adds latency. Test the policy rules under realistic load to ensure they do not degrade service‑level objectives.
By addressing these points, you ensure that the policy you write as code is both enforceable and practical.
Getting started
To try this approach, deploy hoop.dev using the quick‑start guide. The guide walks you through installing the gateway, connecting it to a model server, and loading your policy definitions. Once the gateway is running, you can observe how every inference request is inspected, masked, approved, and recorded without changing any client code.
For detailed steps, see the getting‑started documentation. The repository on GitHub contains the full source code and example policies you can adapt to your environment.
Explore the implementation and contribute on GitHub: https://github.com/hoophq/hoop.
FAQ
Does hoop.dev replace my existing authentication system?No. hoop.dev trusts the OIDC token issued by your identity provider. It uses that token to identify the caller, but the authentication flow remains unchanged.Can I use hoop.dev with any model serving framework?Yes. hoop.dev proxies any TCP‑based protocol, so it works with TensorFlow Serving, TorchServe, custom Flask APIs, or any HTTP‑based inference endpoint.How does hoop.dev help with audits?hoop.dev records every session, including the request payload, the response, and any approval steps. Those logs can be exported for compliance reporting, giving you concrete evidence that policy as code was enforced.
