Crawl Compass
You set up a crawl compass by mapping data sources and assigning strict read roles. This guide helps your team track where automated scripts discover and read records. Most teams start by giving a discovery service account broad read rights across all databases. That broad role lets the crawler map tables fast, but it creates a big risk. Anyone who gets that key can read every table in your system. How a crawl compass architecture works in practice A crawl compass acts as the directional guid
Compass: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up a crawl compass by mapping data sources and assigning strict read roles. This guide helps your team track where automated scripts discover and read records. Most teams start by giving a discovery service account broad read rights across all databases. That broad role lets the crawler map tables fast, but it creates a big risk. Anyone who gets that key can read every table in your system.
How a crawl compass architecture works in practice
A crawl compass acts as the directional guide for your automated data discovery jobs. It tells your scanning tools which hosts, schemas, and tables to check on a schedule. To make this work, engineers create a dedicated service role in each target database. They grant this role permission to read system views and sample rows from user tables.
Next, the team stores the database key in a central secret store. The scanning job fetches this key when a scheduled run starts. It logs in to each data store and reads the table structure. Then it writes those findings back to a central metadata catalog for the team. Engineers use this catalog to see what data exists and where sensitive fields live.
Good setups split these roles by data zone. A script that checks public analytics tables should not use the same key for billing. By splitting keys and roles, teams limit what a single job can reach. They also set network rules so only the scanner host can reach the database port.
What native roles solve and where gaps remain
Native database roles and network rules solve the first part of access control. They decide which service account can log in from a specific host address. If an unknown server tries to log in, the database blocks the request right away. This stops outside attackers from using stolen keys from random locations on the internet.
Still, native setup rules only check identity at the start of a session. Once the crawler or an engineer logs in, the database trusts every valid query. Native roles cannot tell if a query is a safe schema check or a bulk export. If a script has table read rights, it can pull every row without warning.
Also, static database roles cannot rewrite sensitive fields inside query results on the fly. When a discovery job samples a customer table, it reads raw emails and phone numbers. Those raw values often end up in scan logs or test files by mistake. Masking rules that rewrite card numbers or national IDs in results fix this exposure.
Compass: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Running discovery jobs with safer query controls
To close these gaps, teams need controls that inspect individual commands in flight. Instead of trusting an entire session, you should check each query before it runs. This lets you allow metadata reads while blocking large table dumps on the same role.
You should also apply field masking to every sample query that your discovery tools run. When a script reads a user table, masking rewrites matched emails and phone numbers. The scanner still sees the column types and formats, so the catalog stays accurate. At the same time, raw card numbers and national IDs stay out of your logs.
Finally, teams should log every single query statement alongside the user or service identity. Native database logs often scatter across many hosts and lack clear user context. Central command logs let security teams review exact queries after a scan finishes. If a job runs an unexpected command, your team can spot and fix it fast.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
What is the main purpose of a crawl compass setup?
It guides automated discovery jobs across your data stores using scoped roles and clear rules. This helps teams map their tables and columns without losing track of access rights.
Why are read-only database roles still a risk for scanning tools?
A read-only role on a user table still allows full data exports of sensitive rows. It also pulls raw emails, phone numbers, and card numbers into scanner memory and logs.
How does data masking help during automated schema discovery?
Masking rewrites matched fields like national IDs or phone numbers in the query results. The discovery tool still learns the table structure while raw values stay protected.