Dbt Cloud Data Observability Features
You set up dbt cloud data observability features by defining schema tests and freshness rules. These built-in checks run during scheduled jobs to catch broken tables and late data. Data teams often assign broad warehouse roles so these automated checks can read every schema. When those roles hold too many permissions, a routine check or developer run can expose sensitive records. How dbt cloud data observability features work in practice Data teams rely on built-in checks to keep warehouse ta
Observability Data Classification + Linkerd Security Features: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up dbt cloud data observability features by defining schema tests and freshness rules. These built-in checks run during scheduled jobs to catch broken tables and late data. Data teams often assign broad warehouse roles so these automated checks can read every schema. When those roles hold too many permissions, a routine check or developer run can expose sensitive records.
How dbt cloud data observability features work in practice
Data teams rely on built-in checks to keep warehouse tables clean and accurate. The platform groups these checks into four main areas. These areas are schema tests, source freshness checks, lineage graphs, and job alerts. Each tool reads metadata or table rows from your cloud warehouse during a job run.
Schema tests check if your data meets basic rules. You can check that a column has no null values or holds only unique keys. You can also check that values match a set list or link to another table. When a job runs, the platform compiles these rules into select queries. The warehouse runs the queries and returns the count of failing rows.
Source freshness checks track how old your raw tables are. You tell the tool which timestamp column marks the latest load time. During a check, the platform queries the maximum timestamp in that column. If the time gap exceeds your warning threshold, the job flags a stale source.
Lineage graphs map how data flows from raw sources to final reports. The platform builds this map by reading model links in your project files. Teams use the graph to see which dashboards break when a source test fails. Job alerts then send messages to chat channels or email when a run fails.
How project roles and warehouse credentials execute checks
To run these checks, the platform needs a warehouse user role and login credentials. Teams usually set up two main profiles in the cloud workspace. One profile runs scheduled production jobs with a shared service account. The second profile lets human analysts run tests in their own scratch schemas.
The service account needs read permissions across raw sources and write permissions in target schemas. In many workspaces, admins grant a single broad role to this service account. They do this so new source tables do not break nightly test runs. Developers also use personal credentials or shared roles to test models before merging code.
Observability Data Classification + Linkerd Security Features: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native checks solve and where gaps remain
Native warehouse roles and workspace settings solve the core scheduling problem. They let your team run automated data tests on a set timer. They also help analysts trace broken columns back to raw source tables. Workspace roles control who can edit job schedules or view run logs in the browser.
Still, this native setup leaves a major operational gap during query execution. Warehouse roles decide who can log in and which schemas they can read. They do not inspect what individual select statements or test queries do in flight. When an analyst runs an ad-hoc test query, the warehouse returns full raw rows.
If a source table holds emails, phone numbers, or card numbers, those fields appear in clear text. A developer debugging a failed test can query raw customer tables directly from the workspace. Standard warehouse roles also group many actions under one static login credential. If a shared service credential leaks, anyone with that key can read the whole warehouse. Native run logs show that a job ran, but do not rewrite sensitive fields in results.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
What is the difference between schema tests and source freshness checks?
Schema tests check data values inside rows, such as unique IDs or non-null columns. Source freshness checks only measure the time since the last table update occurred.
Do data tests read raw rows from the warehouse?
Yes, the platform compiles each test into a SQL select query for the warehouse. The warehouse scans the target columns and returns failing row counts or sample records.
Why do development runs create data exposure risks?
Developers often query production sources to test new logic and debug failing models. Without field masking, those test queries return real emails, phone numbers, and card numbers.