Zero Trust for Inference

Many assume that simply exposing an inference API behind a firewall is enough to meet zero trust, but that ignores the need for continuous verification and fine‑grained control.

Zero trust considerations for inference

Inference services, whether they serve language models, image classifiers, or recommendation engines, are usually accessed over HTTP or gRPC. In practice, teams often bake a single API key or service‑account token into the client code, push that binary to production, and leave the endpoint open to any caller that knows the secret. The result is a monolithic trust boundary: anyone who possesses the credential can invoke the model, see raw inputs and outputs, and potentially exfiltrate proprietary data. Auditing is an afterthought; logs are either missing or so coarse that they cannot answer “who asked for which prediction at what time?”

What zero trust actually fixes

Zero trust for inference starts by eliminating shared secrets. Each request is authenticated with an identity token issued by an OIDC or SAML provider, and the token’s scopes limit the model’s surface area. This step ensures that only the right principal can call the service and that the call is justified. However, moving the authentication check to the model server does not close the loop. The request still travels directly to the inference engine, bypassing any runtime guardrails. There is no place to mask personally identifiable information in the response, no workflow to pause a risky payload for human approval, and no immutable record of the exact query that was run. In other words, the core zero‑trust premise, verify every request, enforce policy at the point of use, remains unimplemented.

hoop.dev as the data‑path enforcement layer

Enter hoop.dev. It is a Layer 7 gateway that sits between the caller and the inference endpoint. The gateway verifies the OIDC token on each request, evaluates the caller’s group membership and scopes, and then decides whether to allow, mask, or require approval for the payload. Because hoop.dev sits in the data path, every inference call is recorded, every response can be inspected for sensitive fields, and any disallowed command can be blocked before it reaches the model. The gateway holds the service‑account credential that the model needs, so the client never sees it.

In practice, you deploy the gateway close to the inference service, often as a Docker Compose stack for a quick start or as a Kubernetes sidecar for production. An agent runs on the same network segment, holds the model’s credentials, and forwards approved traffic. Clients, whether they are human engineers, automated pipelines, or AI agents, use their normal HTTP client libraries; the only change is the target address, which points at the gateway instead of the model directly.

Key enforcement outcomes

• Per‑request audit: hoop.dev records who invoked the model, the exact input payload, and the timestamp. This creates a searchable audit trail without requiring changes to the model code.
• Inline data masking: Sensitive fields (for example, user identifiers or credit‑card numbers) can be redacted from the response before it leaves the gateway, protecting downstream logs and downstream consumers.
• Just‑in‑time access: Access policies are evaluated at request time, allowing temporary elevation for a specific inference job without granting permanent rights.
• Approval workflows: If a payload matches a high‑risk pattern, such as a request that could trigger a model jailbreak, hoop.dev can route the request to a human reviewer for explicit approval.
• Command blocking: Malicious or malformed inputs can be rejected outright, preventing the model from processing harmful data.

Why this matters for compliance and risk

Regulatory frameworks often require evidence that data‑processing pipelines are accessed on a need‑to‑know basis and that sensitive outputs are protected. By placing enforcement in the data path, hoop.dev generates the audit artifacts that auditors look for, while also reducing the blast radius of a compromised credential. The gateway’s policy engine can be tuned to meet industry‑specific requirements without rewriting the inference service.

Getting started

Because hoop.dev is open source and MIT‑licensed, you can self‑host the gateway in minutes. Follow the getting‑started guide to spin up a Docker Compose instance, configure OIDC authentication, and register your inference endpoint. The learn section provides deeper examples of masking policies and approval workflows.

The full source code, documentation, and contribution guide are available on GitHub. Feel free to explore, raise issues, or submit pull requests.

FAQ

Does hoop.dev replace my model server?

No. hoop.dev acts as a proxy that sits in front of the server. The model continues to run unchanged; only the traffic is inspected and controlled by the gateway.

Can hoop.dev protect any inference framework?

Yes. Because it operates at the HTTP/gRPC protocol layer, any service that exposes a standard API can be proxied, regardless of the underlying ML library or runtime.

How does zero trust get enforced without modifying the model code?

All enforcement happens in the gateway. The gateway validates the identity token, applies masking rules, records the request, and can block or require approval before the payload reaches the model. The model itself remains oblivious to these controls.