How to Implement Human-in-the-Loop Approval for Self-Hosted Models

An engineering team hands a self‑hosted LLM over to a newly onboarded contractor. The contractor receives a static API token that works for weeks, and the model begins serving requests that unintentionally expose internal data. The breach is discovered only after the damage is done, because no one saw who ran the query, what the query asked, or whether the response should have been allowed.

Why human-in-the-loop approval matters for self‑hosted models

Self‑hosted models sit behind the same network perimeter as any other internal service, but they often expose powerful generation capabilities that can leak secrets, violate policy, or cause regulatory risk. A human-in-the-loop approval workflow forces every high‑risk request to pause for a brief manual review before the model runs. The approach gives teams:

• Visibility into who is asking the model to generate content.
• Control over which prompts are allowed to reach the model.
• Audit records that satisfy compliance auditors and internal governance.
• The ability to block or mask responses that contain sensitive data.

Without a dedicated approval step, any token that can reach the model becomes a de‑facto backdoor.

Current practice and its blind spots

Most organizations grant a service account or static API key to every consumer of a self‑hosted model. The key is often stored in CI pipelines, scripts, or local developer environments. This “direct‑connect” pattern satisfies the need to get the model up quickly, but it leaves three critical gaps:

1. There is no runtime gate that can inspect the request before it reaches the model.
2. All calls are recorded only in the client’s logs, which can be altered or deleted.
3. Any high‑risk prompt can be sent without a human ever seeing it.

Introducing a non‑human identity (OIDC token, service account) and least‑privilege scopes reduces the blast radius, yet the request still travels straight to the model endpoint with no audit, no masking, and no approval point.

Architectural pattern with hoop.dev

hoop.dev provides the missing data‑path component. It sits as a Layer 7 gateway between the identity system and the self‑hosted model. The flow looks like this:

1. Setup: Engineers configure an OIDC or SAML identity provider (Okta, Azure AD, Google Workspace). The provider issues short‑lived tokens that identify the caller and their group membership.
2. The data path: The token is presented to hoop.dev. The gateway validates the token, extracts the caller’s attributes, and then decides whether the request may proceed.
3. Enforcement outcomes: If the request matches a policy that requires review, hoop.dev pauses the request and routes it to an approver. The approver sees the prompt, can edit or reject it, and then authorizes the request to continue. While the request is in flight, hoop.dev records the full session, masks any fields that match configured patterns, and stores a persistent audit record.

Because hoop.dev is the only point that can see both the request and the response, every human-in-the-loop approval decision is enforced by the gateway, not by the model or the client. If hoop.dev were removed, the request would flow directly to the model again, and none of the approval, masking, or recording would occur.

Implementing the workflow

To add human‑in‑the‑loop approval to a self‑hosted model, follow these high‑level steps:

• Deploy the hoop.dev gateway in the same network segment as the model. The quick‑start Docker Compose file gets you a running instance with OIDC support out of the box.
• Register the model as a connection in hoop.dev, supplying the host, port, and the service‑account credential that the gateway will use to talk to the model.
• Define a policy rule that marks certain prompt patterns (for example, any request containing the words “password”, “API key”, or “customer data”) as requiring approval.
• Configure an approval group – typically a security or data‑privacy team – that will receive the paused request via email, Slack, or the hoop.dev UI.
• Enable session recording and inline masking for the connection so that every interaction is stored for later review.

The policy engine lives inside the gateway, so the model never sees an unapproved request. The approval step is a single, auditable hand‑off that can be integrated with existing ticketing or incident‑response tools.

Getting started

All of the pieces described above are covered in the official documentation. Begin with the getting‑started guide to spin up a gateway, connect your model, and enable approval policies. The learn section dives deeper into policy syntax, masking rules, and session replay features. After deployment, you can monitor the gateway health via its built‑in metrics endpoint.

FAQ

Q: Do I need to modify the model’s code to use hoop.dev?
A: No. hoop.dev operates at the protocol layer, so existing client libraries (curl, HTTP, gRPC, etc.) continue to work unchanged.

Q: Can I enforce approval only for a subset of users?
A: Yes. Policies can reference group membership extracted from the OIDC token, allowing fine‑grained control per team or role.

Q: Where are the audit records stored?
A: hoop.dev writes session logs to a configurable backend (object storage, database, or log aggregation service). The logs are retained and can be reviewed for compliance.

With the gateway in place, every high‑risk request to your self‑hosted model must pass through a human‑in‑the‑loop approval step, ensuring that powerful generation capabilities are used responsibly and auditable.

Ready to try it yourself? The source code and contribution guidelines are available on GitHub.