How to Configure Haystack Bigquery Audit Log

Published
Length
3 min read

Luis Felipe Dias

All posts

You set up a haystack bigquery audit log workflow by linking pipeline service accounts to cloud log sinks. This setup records every query that your search pipeline runs against your data tables. Teams often give the pipeline one broad role to make setup fast. Broad roles let the app read sensitive rows across all tables without clear limits. How the haystack bigquery audit log workflow runs Haystack pipelines use components called retrievers to fetch text and numbers from cloud data stores. W

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You set up a haystack bigquery audit log workflow by linking pipeline service accounts to cloud log sinks. This setup records every query that your search pipeline runs against your data tables. Teams often give the pipeline one broad role to make setup fast. Broad roles let the app read sensitive rows across all tables without clear limits.

How the haystack bigquery audit log workflow runs

Haystack pipelines use components called retrievers to fetch text and numbers from cloud data stores. When a user asks a question, the pipeline builds a query and sends it out. The cloud data warehouse runs that query and sends the matching rows back to the app. At the same time, the cloud platform writes an audit entry for that job.

This log entry shows which service account ran the query and when it happened. It also records the exact query text and the number of bytes read. Security teams route these audit records into a central table for review and alerts. They check these records to track cost spikes and spot strange query patterns.

To make this work, admins assign an identity role to the pipeline host. The host uses a key or a workload identity to log in to the project. Once logged in, every table read triggers a data read log event in the cloud console. Admins must turn on data read logs by hand because the cloud platform keeps them off to save space.

Managing cloud roles for pipeline queries

A clean setup depends on tight cloud roles for the pipeline identity. Many builders start by granting project-wide read roles so the pipeline never fails on missing permissions. That choice creates a risk when the pipeline serves many different users at once. Every user prompt runs under the same shared service account inside the data warehouse.

When you check the audit records later, every single query shows the exact same bot identity. You can see the SQL text in the log, but you cannot see which user asked the question. If a prompt pulls sensitive rows, the audit trail stops at the pipeline host. Teams try to fix this by adding user tags to query jobs or splitting pipelines up.

Splitting pipelines means making a new service account for each team or app feature. That approach increases setup work and makes role reviews hard to track over time. Admins spend hours checking which bot key belongs to which search pipeline. Even with clean tags, cloud roles only control access at the table or dataset level.

Audit Log Integrity + BigQuery IAM: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

What native audit logs solve and what they miss

Native audit logs are great for past reviews and compliance checks after a query finishes. They give you a full history of which service account touched which table over the last month. Security engineers can query these logs to build reports and check total data use. Still, an audit log only records an event after the data has already left the table.

Native roles decide who can log in, not what a single query does in flight. If a pipeline has read rights on a customer table, it can pull emails, phone numbers, and card numbers. The audit log will record the read, but the sensitive text still enters the model prompt. Masking tools that rewrite matched fields in results help stop raw emails and national IDs from spreading.

Teams need a way to inspect queries before they run and clean results before they return. Relying only on past logs leaves a gap between when a bad query runs and when an admin spots it. Real control means pairing cloud audit history with active checks on every pipeline request.

Check database and API queries on the wire

Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.

Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.

Frequently asked questions

Why are my pipeline queries missing from the cloud audit trail?

Cloud platforms turn off data read audit logs by default to keep storage costs low. You must turn on data read logging in the cloud console to capture pipeline select queries.

How can I tell which user triggered a pipeline query in the logs?

Standard service accounts group all pipeline queries under one shared machine identity in the logs. You must pass user metadata in the job labels or use an identity layer that tracks each caller.

Do audit logs stop a pipeline from reading sensitive columns?

No, audit logs only write down what happened after the query finishes running. To hide emails, phone numbers, or card numbers, you must mask matched fields in results before the app reads them.

The query runs. The emails stay hidden.