How can you prove that every prompt sent to a large language model and every response returned complies with internal policy without manually stitching logs together?
Most teams treat a context window as a black box, leaving compliance evidence scattered across logs and making it difficult to prove policy adherence. Engineers embed API keys in code, scripts call the model directly, and the raw HTTP traffic lands in a cloud provider’s logs that are hard to correlate with internal ticketing or approval systems. The result is a patchwork of evidence: a few request IDs in a monitoring dashboard, an occasional screenshot of a prompt, and a vague audit note that “the model was used.” When a regulator asks for proof that sensitive data never left the organization, the team scrambles to produce whatever fragments remain. The underlying problem is that the data path – the actual request‑response stream – is never observed by a control plane that can record, mask, or require approval.
What you need is a continuous, tamper‑evident stream of compliance evidence that lives outside the application that initiates the request. The requirement is simple: every interaction with the model must be captured, any sensitive fields must be redacted in real time, and an approval workflow must be enforceable before high‑risk prompts are sent. Yet, even with strong identity providers and least‑privilege service accounts, the request still reaches the model directly, leaving no place for policy enforcement or audit.
Why compliance evidence matters for context windows
Regulatory frameworks such as SOC 2 or industry‑specific guidelines often require proof that data processing activities are controlled and observable. In the AI space, a context window can contain personally identifiable information, trade secrets, or regulated health data. Without a reliable audit trail, a breach of policy can go unnoticed until after the fact, exposing the organization to fines, reputational damage, and loss of customer trust. Continuous compliance evidence also supports internal governance: security teams can review who accessed the model, what data was included, and whether any escalation was required.
The missing audit layer in current workflows
Today's typical setup looks like this:
- Developers obtain an API token from the LLM provider and store it in a secret manager.
- Application code calls the provider’s endpoint directly, passing the full prompt.
- Network traffic is encrypted end‑to‑end, so intermediate firewalls see only bytes.
- Logs are generated on the client side, often at the application level, and may omit the actual payload for privacy reasons.
In this model, the setup – identity verification and token distribution – decides who can start a request, but it does not enforce any policy on the request itself. The request travels straight to the model without a gate that can inspect or record it. Consequently, there is no guarantee that compliance evidence is being created, and any masking or approval must be built into each application, leading to inconsistency and maintenance overhead.
How hoop.dev provides continuous compliance evidence
hoop.dev inserts a Layer 7 gateway between the caller and the model. The gateway runs a lightweight agent inside the same network as the application, so the request never leaves the controlled environment without passing through the gateway. This placement makes the gateway the data path where enforcement occurs.
When a request arrives, hoop.dev performs three actions that directly generate compliance evidence:
- Session recording: hoop.dev records the full request and response pair, storing the transcript in a log that can be queried later for audits.
- Inline masking: before the response leaves the gateway, hoop.dev can redact fields that match configured patterns, ensuring that sensitive data never appears in downstream logs.
- Just‑in‑time approval: for prompts that match high‑risk criteria, hoop.dev routes the request to a human approver. The approval decision is logged alongside the session, providing a clear audit trail.
All of these outcomes are produced because hoop.dev sits in the data path. The setup – OIDC or SAML authentication, group‑based role mapping, and least‑privilege service accounts – still determines who can initiate a request, but the enforcement and evidence generation happen exclusively within the gateway.
Because hoop.dev is open source, teams can inspect the code that writes the audit records, verify that masking rules are applied correctly, and integrate the log store with their existing SIEM or compliance platform. The gateway does not expose credentials to the caller; the agent holds the provider token, so the caller never sees the secret.
Getting started with continuous evidence for AI workloads
To adopt this approach, follow the general steps outlined in the getting started guide. Deploy the gateway using the provided Docker Compose file or your preferred orchestrator, register the LLM endpoint as a connection, and define masking and approval policies in the configuration. Once the gateway is live, every context‑window interaction will be captured, redacted as needed, and stored as compliance evidence.
For deeper technical details on policy configuration, masking syntax, and audit‑log integration, see the learn section. The documentation provides examples of how to map identity groups to specific approval workflows, ensuring that only authorized users can send high‑risk prompts.
FAQ
Do I need to change my existing application code?No. hoop.dev works as a transparent proxy. Your client can continue to use the standard HTTP endpoint for the LLM provider; the gateway intercepts the traffic.How long are the audit records retained?Retention is configurable in the gateway’s log store. You can align it with your organization’s data‑retention policy.Is the gateway itself a security risk?Because hoop.dev runs inside your trusted network and never exposes the provider credential, it reduces the attack surface. All enforcement happens before the request leaves the network.
Ready to see the implementation? Explore the source code on GitHub and start building a continuous compliance evidence pipeline for your AI context windows today.
