Audit logging for autonomous agents on Datadog

Why audit logging matters for autonomous agents

Autonomous agents that push metrics to Datadog can silently alter observability data without anyone noticing. In many teams the agent runs with a static API key that is checked into source control or shared across dozens of services. The key grants unrestricted write access, and the agent talks directly to Datadog’s HTTP API. There is no per‑action record of who triggered a change, what data was sent, or whether the payload contained accidental secrets. When a bug or a compromised agent sends malformed events, the damage spreads before anyone spots the anomaly.

This unchecked flow is the default reality for many organizations: a single credential, standing access, and no independent audit trail. The risk is amplified when agents act on behalf of multiple owners, because the original intent of each request is lost in a sea of identical API calls.

What audit logging alone does not solve

Enabling Datadog’s built‑in request logs adds some visibility, but the logs are generated after the request has already reached the service. They cannot block a dangerous operation, mask sensitive fields before they are stored, or require a human to approve a high‑risk change. The request still travels straight from the agent to Datadog, bypassing any enforcement point that could intervene.

In other words, audit logging without a control plane leaves three gaps:

• The agent retains the raw API key and can reuse it at any time.
• There is no gate that can stop a request that matches a risky pattern.
• Sensitive data that appears in payloads may be written to logs in clear text.

To close those gaps you need a dedicated data‑path component that sits between the identity that initiates the request and the Datadog endpoint.

How hoop.dev provides the missing control layer

hoop.dev acts as an identity‑aware proxy for HTTP traffic destined for Datadog. The gateway is deployed inside the same network as the agents, and it holds the Datadog API key. Agents authenticate to hoop.dev with OIDC tokens that represent a service account or a machine identity. The setup phase decides who may start a request, but it does not enforce any policy on its own.

Once the token is validated, hoop.dev becomes the only place where the request can be inspected. It records every HTTP call, stores a replay‑able session, and can apply inline masking to redact secrets before they are written to logs. If a request matches a rule that requires approval, such as creating a new dashboard or modifying a monitor, hoop.dev pauses the call and routes it to a designated approver. The agent never sees the Datadog credential, and the organization gains a complete audit trail that ties each action to a verified identity.

Setup: identity and credential provisioning

The first step is to define a non‑human identity for the autonomous agent in your identity provider. The identity receives a short‑lived OIDC token that hoop.dev can verify. The Datadog API key is stored only inside the gateway configuration; it never leaves the host running hoop.dev. This separation ensures that the agent cannot extract or reuse the credential.

The data path: proxying Datadog traffic

All outbound traffic to the Datadog API endpoint is forced through hoop.dev. Because the gateway sits at Layer 7, it can parse the JSON payload, identify fields that contain tokens or personal data, and mask them before they are persisted in the audit store. It also timestamps each request, records the caller’s identity, and logs the full request‑response cycle for later replay.

Enforcement outcomes enabled by hoop.dev

• hoop.dev records each session, providing an audit log that links every Datadog write operation to a verified identity.
• It masks sensitive values in real time, preventing secret leakage in logs or downstream systems.
• It blocks or routes high‑risk API calls for just‑in‑time human approval, reducing the blast radius of a compromised agent.
• hoop.dev stores the audit trail outside the agent process, guaranteeing that the evidence cannot be altered by the same code that generated it.

All of these outcomes exist only because hoop.dev occupies the data path between the agent and Datadog.

Getting started

To implement this architecture, follow the getting‑started guide to deploy the gateway and register a Datadog connection. The learn section provides deeper detail on configuring masking rules, approval workflows, and session replay.

FAQ

Do I still need to rotate the Datadog API key?

Yes. hoop.dev stores the current key, but regular rotation remains a best practice. When you rotate, update the gateway configuration and the new key is automatically used for all subsequent requests.

Can hoop.dev forward requests to multiple Datadog accounts?

Yes. Each account can be defined as a separate connection with its own stored credential. The same identity can be scoped to one or more connections based on group membership.

Will enabling hoop.dev add latency to my agents?

The proxy introduces a small, predictable overhead because it parses and logs each request. In most environments the added latency is measured in milliseconds and is outweighed by the security and audit benefits.

Next steps

Review the source code, contribute improvements, or fork the repository to tailor the gateway to your environment. Explore the open‑source repository on GitHub for the full implementation details and community support.