Airbyte Without Add Field
You can run airbyte without add field rules by turning off custom schema changes. This keeps your synced tables matching the exact structure of your source data. Data teams often add extra fields during sync runs to tag or track records. Still, doing this with broad database roles lets sync jobs change destination schemas at will. When permissions are too wide, one bad sync rule can break downstream reports. How to set up airbyte without add field transformations Data pipelines move records f
Without Add Field: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You can run airbyte without add field rules by turning off custom schema changes. This keeps your synced tables matching the exact structure of your source data. Data teams often add extra fields during sync runs to tag or track records. Still, doing this with broad database roles lets sync jobs change destination schemas at will. When permissions are too wide, one bad sync rule can break downstream reports.
How to set up airbyte without add field transformations
Data pipelines move records from source apps into destination databases on a schedule. During this move, the platform reads the source schema and maps it to target tables. If you use an add field step, the tool injects new columns into the stream. Running airbyte without add field steps means the platform transfers only the original columns. You set this up by leaving the custom field transformation list empty in your stream. The sync worker then reads the raw data and writes it straight to the destination.
This approach simplifies how you manage database roles for your data pipelines. When a sync worker adds fields, it needs database permissions to alter target tables. Those alter permissions allow the worker to change column types or add new columns anytime. If you skip custom field additions, you can lock down the destination table schema. You create the target tables ahead of time using a separate admin role. Then, you give the sync worker a role that only inserts and updates rows. The worker no longer needs the power to alter table structures on the fly.
What native schema controls solve and what they miss
Turning off extra field transformations keeps your data warehouse clean and predictable. Analytics teams know the target tables will match the source app schema every day. It also stops pipeline builders from putting business logic inside the ingestion layer. Instead, teams can handle column changes later inside the warehouse using SQL models. Restricting the sync role from altering tables also reduces the risk of broken schemas.
Still, native pipeline settings and database roles leave an operational gap open. A database role decides who can log in and which tables they can touch. Once the sync worker or an engineer logs in, the database trusts every command. Native setup rules cannot inspect individual SQL statements or API requests in flight. If a source table holds sensitive fields like emails or phone numbers, the sync copies them. The database role cannot rewrite card numbers or national IDs inside a query result. Also, shared service roles make it hard to see which person triggered a manual query. Teams need a way to check commands in flight and mask sensitive fields dynamically.
Without Add Field: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Managing schema updates and live data queries safely
When you stop pipelines from adding fields, you must plan for real source changes. Source apps still add new columns when developers ship new features to users. You can set your pipeline to ignore new source columns until a human reviews them. This stops sudden source updates from breaking your target tables or exposing private data. An engineer checks the new column first to see if it holds emails or card numbers.
After the review, an admin updates the target table schema in a controlled release. Meanwhile, engineers often log in to the destination database to check sync results and fix bugs. Giving those engineers the same broad role used by the pipeline creates a security risk. Anyone with that key can read full tables or run slow queries on production data. Separating automated sync roles from human query sessions keeps your data safe during daily checks.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
What happens if the source database adds a new column during a sync?
The sync tool detects the schema change during the next scheduled run. Depending on your stream rule, it can ignore the column or pause the sync. This lets your team update the target table before loading the new data.
Can we still track when a row was loaded without adding custom fields?
Yes, the platform includes built-in metadata columns for sync time and record IDs. These default system columns track load times without needing custom add field rules. Your target table just needs matching columns ready to receive that metadata.
How do we hide sensitive columns if we do not transform fields in the pipeline?
You can deselect whole columns in the stream setup so they never copy over. If you need the column in the warehouse, data masking rewrites matched fields in results. This protects emails, phone numbers, and card numbers when engineers query the target table.