> ## Documentation Index > Fetch the complete documentation index at: https://mintlify.hoop.dev/docs/llms.txt > Use this file to discover all available pages before exploring further. # Workflow Observability > See exactly what AI agents, scripts, and CI/CD jobs did in Hoop, attributed to named identities and grouped by workflow run export const WorkflowObservabilityAnimation = () => { const SCENARIOS = [{ caller: "deploy-pipeline", keyHash: "hpk_xR9a", icon: "ci", correlationId: "deploy-2026-04-28-17a", sessions: [{ connection: "postgres-migrate", command: "ALTER TABLE users ADD COLUMN locale TEXT", duration: 340 }, { connection: "postgres-migrate", command: "INSERT INTO schema_versions VALUES ('2026_04_24_a')", duration: 89 }, { connection: "internal-api", command: "POST /healthcheck", duration: 120 }, { connection: "postgres-analytics", command: "SELECT COUNT(*) FROM active_users", duration: 74 }] }, { caller: "nightly-report", keyHash: "hpk_m4Tk", icon: "cron", correlationId: "cron-2026-04-28T02:00", sessions: [{ connection: "warehouse-read", command: "SELECT * FROM fact_sales WHERE ds = CURRENT_DATE - 1", duration: 612 }, { connection: "warehouse-read", command: "SELECT * FROM dim_customer", duration: 418 }, { connection: "s3-exports", command: "PUT s3://reports/2026-04-28/daily.parquet", duration: 204 }, { connection: "slack-notify", command: "POST /notify #data-reports", duration: 93 }] }, { caller: "ticket-agent", keyHash: "hpk_qB7w", icon: "agent", correlationId: "task-12345", sessions: [{ connection: "tickets-db", command: "SELECT * FROM automation_tasks WHERE id = 12345", duration: 128 }, { connection: "docs-api", command: "GET /documents/47219", duration: 208 }, { connection: "third-party-api", command: "POST /enrich", duration: 416 }, { connection: "tickets-db", command: "UPDATE automation_tasks SET status='done' WHERE id = 12345", duration: 94 }] }]; const FONT_URL = "https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;500&family=Inter:wght@400;500;600;700;800&display=swap"; const animationStyles = ` @keyframes workflowSlideIn { from { opacity: 0; transform: translateX(-20px); } to { opacity: 1; transform: translateX(0); } } @keyframes workflowFadeIn { from { opacity: 0; transform: translateY(8px); } to { opacity: 1; transform: translateY(0); } } @keyframes workflowPulse { 0%, 100% { transform: scale(1); opacity: 1; } 50% { transform: scale(1.25); opacity: 0.5; } } @keyframes workflowScanLine { 0% { left: 0%; opacity: 0; } 15% { opacity: 1; } 85% { opacity: 1; } 100% { left: 100%; opacity: 0; } } @keyframes workflowKeyShine { 0%, 100% { filter: drop-shadow(0 0 0px rgba(var(--warm-gold-rgb),0)); } 50% { filter: drop-shadow(0 0 8px rgba(var(--warm-gold-rgb),0.45)); } } `; const CallerIcon = ({kind, size = 22}) => { const color = "var(--warm-gold)"; if (kind === "ci") { return ; } if (kind === "cron") { return ; } return ; }; const [scenarioIdx, setScenarioIdx] = useState(0); const [visibleCount, setVisibleCount] = useState(0); const [runningIdx, setRunningIdx] = useState(-1); const timerRef = useRef(null); const scenario = SCENARIOS[scenarioIdx]; useEffect(() => { setVisibleCount(0); setRunningIdx(-1); let step = 0; const stepThrough = () => { if (step < scenario.sessions.length) { setRunningIdx(step); setVisibleCount(step + 1); step += 1; timerRef.current = setTimeout(() => { setRunningIdx(-1); timerRef.current = setTimeout(stepThrough, 350); }, 900); } else { timerRef.current = setTimeout(() => { setScenarioIdx(prev => (prev + 1) % SCENARIOS.length); }, 2200); } }; timerRef.current = setTimeout(stepThrough, 500); return () => clearTimeout(timerRef.current); }, [scenarioIdx]); const totalDuration = scenario.sessions.slice(0, visibleCount).reduce((acc, s) => acc + s.duration, 0); return
{}
{}
{scenario.caller}
{scenario.keyHash} ****
API Key
{}
correlation-id {scenario.correlationId} {runningIdx >= 0 &&
}
{}
{scenario.sessions.map((s, i) => { const visible = i < visibleCount; const running = i === runningIdx; const done = i < visibleCount && !running; return
{} {} {s.connection} {} {s.command} {} {running ? "running…" : done ? `${s.duration}ms` : "—"}
; })}
{}
{visibleCount} / {scenario.sessions.length} sessions 1 workflow run {totalDuration}ms total
; };
## What You'll Accomplish Workflow Observability lets you see exactly what any headless caller (AI agent, script, CI/CD job, or scheduled batch) did in Hoop, with the same identity, audit, and policy story as an interactive user. You can: * Authenticate automation with **named identities** instead of shared admin credentials * **Scope** each caller's permissions to only the resource roles and groups it needs * **Group** the sessions of the same logical run into one workflow, even across multiple commands or resource roles * Keep a complete **audit trail** of who ran what, when, and as part of which task *** ## The Problem Workflow Observability Solves Much of the work that touches production is not interactive: ad-hoc scripts against the warehouse, deploy pipelines running migrations and smoke tests, scheduled export jobs, AI agents chaining tool calls to finish each queued task. To an audit log, all of that typically looks the same: a string of isolated, anonymous-looking sessions with no way to answer the questions operators actually ask: * Who (or which system) ran this command? * What else did it do before and after? * Which of these fifty sessions belong to the ticket we're investigating? Without a named identity for the caller and a way to correlate its sessions, every invocation is disconnected. Workflow Observability closes that gap so audit, masking, and guardrail features work for automation the same way they do for humans. *** ## How Workflow Observability Works Two building blocks make this possible. ### 1. API Keys: named identities for headless callers Automated callers authenticate using an **API Key** (prefixed with `hpk_`) instead of a browser login. Keys are: * **Named**: sessions are attributed to `nightly-report`, `deploy-pipeline`, or `ticket-agent` rather than an anonymous administrator. * **Scoped**: the key inherits only its assigned groups' permissions, so the blast radius matches the job. * **Revocable**: deactivate a compromised or retired key instantly, without redeploying the Gateway. * **Auditable**: the Gateway tracks who created each key, who deactivated it, and when it was last used. Keys are managed in the Web App under **Settings → API Keys**. ### 2. Correlation IDs: group related sessions into a workflow Most automated work is a chain of steps: a deploy migration then smoke test then health check, a pipeline querying a source and writing to a destination, an agent reading a task and calling several APIs. Tag each call with a shared correlation value so the sessions of the same run link together. The behavior is identical on the CLI and the REST API. Pass `--correlation-id` on every `hoop exec` invocation: ```sh theme={"dark"} hoop exec postgres-demo \ --correlation-id deploy-2026-04-24-17a \ -i "SELECT COUNT(*) FROM schema_migrations" hoop exec internal-api \ --correlation-id deploy-2026-04-24-17a \ -i "POST /health HTTP/1.1 ..." ``` Send the value in the `X-Hoop-Correlation-Id` header on every request that opens a session: ```sh theme={"dark"} curl https://yourgateway-domain.tld/ \ -H "Authorization: Bearer hpk_your_key_value" \ -H "X-Hoop-Correlation-Id: deploy-2026-04-24-17a" \ -d '...' ``` Sessions carrying the same value are stored against one workflow run, so you can trace the full path: which commands ran, against which resource roles, in what order, with what outcome. The correlation ID is a free-form printable ASCII string up to 255 characters. Use whatever identifier the caller already has: a CI build number, cron run ID, ticket/task ID, agent workflow ID, or a UUID generated at the start of the run. *** ## Common Scenarios A data scientist runs a multi-step script against production. Each step is executed through `hoop exec` with a shared correlation ID, so the whole run shows up as one traceable workflow instead of scattered queries. A deploy pipeline uses a scoped API Key to run migrations, seed data, and health checks. Tagging every step with the build ID answers "what did build #842 actually do in prod?" from the audit log. A nightly export uses an API Key scoped to a single read-only resource role. Tagging each run with its cron run ID lets you trace and replay a failed export without guessing which sessions belong to it. An agent picks up tasks from a queue and issues many tool calls per task. With a named API Key and task-ID tagging, operators can open any task and see exactly what the agent did, under the same masking and guardrails as any other caller. *** ## Quick Start ## Prerequisites To get the most out of this guide, you will need to: * Either [create an account in our managed instance](https://use.hoop.dev) or [deploy your own hoop.dev instance](/setup/deployment/overview) * You must be your account administrator to perform the following actions - Admin access to create API Keys - A script, CI job, scheduled task, or agent that calls Hoop through the CLI (`hoop exec`) or the REST API ### Step 1: Create an API Key for the workflow Go to **Settings → API Keys** in the Web App and click **Create new API key**. Name the key after the caller (e.g. `nightly-report`, `deploy-pipeline`, `ticket-agent`). Assign groups that grant only the resource roles this workflow needs; avoid `admin` unless genuinely required. Copy the `hpk_…` value at creation (it is shown only once) and load it into your workflow's secret store (CI secret, Vault, Kubernetes Secret, etc.). See the [API Keys setup guide](/setup/apis/api-key) for the full reference on key management, rotation, and REST endpoints. ### Step 2: Authenticate with the API key Persist the key in the local config on the workflow's host: ```sh theme={"dark"} hoop config create \ --api-key hpk_your_key_value \ --api-url https://yourgateway-domain.tld ``` The CLI then authenticates as the key's identity for every subsequent command. Send the key as a bearer token in the `Authorization` header on every request: ```sh theme={"dark"} curl https://yourgateway-domain.tld/ \ -H "Authorization: Bearer hpk_your_key_value" ``` Any REST endpoint that accepts an authenticated caller accepts an `hpk_` bearer token, subject to the key's group permissions. ### Step 3: Tag each call with a correlation ID Pass a shared identifier on every call in the same logical run: Use the `--correlation-id` flag on `hoop exec`: ```sh theme={"dark"} CORRELATION_ID="${CI_BUILD_ID:-$(uuidgen)}" hoop exec postgres-demo \ --correlation-id "$CORRELATION_ID" \ -i "SELECT * FROM customers WHERE id = 42" ``` Send the value in the `X-Hoop-Correlation-Id` header on every request that opens a session: ```sh theme={"dark"} curl https://yourgateway-domain.tld/ \ -H "Authorization: Bearer hpk_your_key_value" \ -H "X-Hoop-Correlation-Id: $CORRELATION_ID" \ -d '...' ``` The header is session-scoped: read once when the session opens, then inherited by every subsequent request that reuses the session. All sessions sharing that value are linked as one workflow run in audit logs and session history. To fetch every session in a run, filter the sessions endpoint by the same identifier: ```sh theme={"dark"} curl "https://yourgateway-domain.tld/api/sessions?correlation_id=deploy-2026-04-24-17a" \ -H "Authorization: Bearer hpk_your_key_value" ``` *** ## Related Features Full replay of every session, including automated ones. See exactly what a script, pipeline, or agent executed and what it got back. Block destructive operations regardless of the caller. The same rules apply to humans, scripts, and agents. Mask automated query results just like interactive sessions, so sensitive data never leaves the Gateway. Technical reference for creating, rotating, and revoking API keys via the Web App and REST API.