Airbyte Mappings
You set up airbyte mappings to rename, hash, or filter fields during a data sync. These rules change your records after reading the source and before writing the output. Teams often use this feature to clean up tables or hide private text. Still, broad database roles still let users read raw source tables outside the pipeline. How airbyte mappings work in your pipeline Data teams move millions of rows from apps into central storage every day. Raw records often hold private fields like emails,
Mappings: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up airbyte mappings to rename, hash, or filter fields during a data sync. These rules change your records after reading the source and before writing the output. Teams often use this feature to clean up tables or hide private text. Still, broad database roles still let users read raw source tables outside the pipeline.
How airbyte mappings work in your pipeline
Data teams move millions of rows from apps into central storage every day. Raw records often hold private fields like emails, phone numbers, or card numbers. You can apply airbyte mappings inside each sync job to alter those fields.
The sync engine reads a batch of records from the source database first. Then the engine runs your rules on each row in memory. You can hash a column, rename a field, or drop a whole row. After the changes finish, the worker writes the clean batch to the destination.
This workflow keeps you from writing extra transformation scripts after the load step. It also stops raw values from landing in your analytics tables by mistake. Teams set up these rules per stream inside their sync tool.
How to set up field rules for your streams
Start by picking the specific data stream you want to change. Check the schema list to find columns that hold private customer details. Decide if you need to remove a column or mask its text.
If a downstream team never uses card numbers, drop that field from the stream. If analysts need to join records by email, hash the email field instead. Hashing turns the raw email into a fixed string of letters and numbers. Analysts can still count unique users, and the rule rewrites the matched email field.
You can also filter out rows that match a specific text value. For example, you might skip test accounts or records from a certain region. Save your stream rules and run a test sync to check the output.
Mappings: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native pipeline rules solve and what they miss
In-flight stream rules work well for scheduled data movement between two systems. They stop private strings from piling up in your downstream warehouse tables. They also reduce the number of cleanup jobs your analysts must maintain.
Still, pipeline rules only cover data moving through that single sync path. Engineers, support staff, and scripts still log in to the source database directly. A database role decides who logs in, not what individual queries do in flight.
When a developer runs a select query to fix a bug, stream rules do not run. The database returns raw emails, phone numbers, and national IDs in the query results. Also, shared service keys used by the sync tool often carry broad permissions. If someone copies that key, they can read or change any table directly.
Add runtime guardrails and masking beside your databases and APIs
Login rules, role grants, and CI checks govern who connects to your stack. Yet any allowed script, engineer, or AI agent still runs commands with the full scope of its role. For workloads on Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes, SSH, or internal HTTP services, hoop.dev sits beside the service as an open-source proxy, a sidecar for your database or API.
Before a query runs, hoop.dev guardrails check the statement against your rules. An AI Analyzer you configure can score intent or hold risky calls for human review. Next, hoop.dev data masking rewrites sensitive fields in responses, and each verdict is logged per listener. Explore the MIT-licensed project on GitHub.
Frequently asked questions
Do airbyte mappings change data in the source database?
No, the rules only change records in memory while the sync job runs. Your source database keeps the original raw values untouched.
When should a team hash a field instead of dropping it?
Drop a field when nobody needs it for reports or joins. Hash a field when analysts must link records without reading the raw text.
Do sync rules protect direct database queries from engineers?
No, sync rules only apply to data moving through the pipeline job. Direct queries against the source or warehouse bypass those stream rules completely.