Non-Human Identities for Self-Hosted Models

Static secrets embedded in code give every compromised service unlimited access to your self‑hosted models, and they highlight why relying on non-human identity alone is risky.

Current practice and its blind spots

Most teams ship a single API key or a long‑lived service account credential alongside the model binary. The key is checked into source control, copied between environments, and shared among dozens of micro‑services. When a container starts, it connects directly to the model endpoint using that credential. The connection bypasses any central policy engine, so there is no audit trail, no real‑time validation of the request, and no way to prevent a rogue service from issuing a malicious prompt.

This pattern works until it doesn’t. A compromised CI pipeline, an over‑privileged service account, or a leaked token instantly grants an attacker the ability to run arbitrary inference calls, extract proprietary weights, or poison the model with poison‑data payloads. Because the request travels straight to the model, the organization cannot answer questions such as: who issued the call, what data was returned, or whether the prompt violated internal policy.

Why non-human identity alone is not enough

Introducing a non-human identity, OIDC‑issued service tokens, short‑lived JWTs, or federated cloud identities, solves the credential‑leak problem. Tokens can be scoped, rotated, and revoked, and they give the platform a reliable way to say *which* automated component is speaking.

However, the token itself does not enforce runtime guardrails. The request still reaches the model endpoint directly, meaning the platform cannot:

• Record the exact prompt and response for later review.
• Mask sensitive fields in the model’s output, such as personally identifiable information.
• Require a human approver before a high‑risk operation is executed.
• Block commands that match a deny‑list (for example, attempts to export model weights).

In other words, the *setup* of non-human identity decides who may start a request, but it provides no enforcement on the data path.

The gateway that makes enforcement possible

To close the gap, the request must pass through a layer that can inspect, control, and log every interaction before it reaches the model. That layer is a Layer 7 identity‑aware proxy that sits between the service token and the model endpoint. By placing enforcement in the data path, the platform gains a single point where policy can be applied consistently.

hoop.dev fulfills exactly that role. It receives the non-human token, validates it against the configured identity provider, and then proxies the traffic to the self‑hosted model. Because the gateway is the only place the traffic flows, it can implement the enforcement outcomes that otherwise would be impossible.

How hoop.dev adds runtime safeguards

When a service initiates an inference call, hoop.dev performs several actions:

• Session recording. hoop.dev records the full request and response, creating an audit log that can be replayed for investigations.
• Inline masking. Sensitive fields in the model’s output are redacted in real time, preventing accidental data leakage.
• Just‑in‑time approval. For prompts that match a high‑risk policy (for example, requests to export model parameters), hoop.dev routes the request to a human approver before forwarding it.
• Command blocking. Known dangerous patterns, such as attempts to load arbitrary code into the model, are blocked outright.

All of these outcomes depend on the gateway being in the data path; without hoop.dev, the same non-human identity would have no way to enforce them.

Putting the pieces together

The overall architecture looks like this:

1. Setup. Create a service account in your identity provider, configure short‑lived JWTs, and grant the minimal scopes needed for inference.
2. The data path. Deploy hoop.dev as a network‑resident gateway near the model. The gateway holds the model’s service credentials, so the calling service never sees them.
3. Enforcement outcomes. hoop.dev records each session, masks outputs, requires approvals for risky prompts, and blocks disallowed commands.

This separation makes it clear why the gateway is essential: the setup gives you identity, the data path gives you control, and hoop.dev provides the concrete security benefits.

Getting started

To try the approach, follow the getting‑started guide and explore the feature documentation on the learn page. The repository contains the full open‑source implementation and example configurations.

FAQ

Q: Do I still need to rotate service tokens if I use hoop.dev?
A: Yes. Token rotation limits the window of exposure if a token is leaked, while hoop.dev provides the runtime guardrails that protect the model even during that window.

Q: Can hoop.dev mask data for any model output?
A: hoop.dev can apply pattern‑based redaction to any text flowing through the gateway, so it works for language model responses, embeddings, or structured JSON.

Q: Does hoop.dev store the model’s credentials?
A: The gateway holds the credentials internally, ensuring that calling services never see them. This design prevents credential sprawl and supports just‑in‑time access.

Explore the code, contribute improvements, and see how the community is building stronger runtime governance for AI workloads.

Explore the repository on GitHub