Airbyte Self Host
You set up an airbyte self host deployment by running its core services on your own servers. You then add source and destination database keys inside the web app to move data. Data teams often run this setup on virtual machines or container clusters. During setup, engineers often use broad database roles to make pipelines work fast. These wide permissions let sync jobs read every table, even ones with private user details. If a worker node or state database leaks, attackers gain those same broad
Self-Service Access Portals: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up an airbyte self host deployment by running its core services on your own servers. You then add source and destination database keys inside the web app to move data. Data teams often run this setup on virtual machines or container clusters. During setup, engineers often use broad database roles to make pipelines work fast. These wide permissions let sync jobs read every table, even ones with private user details. If a worker node or state database leaks, attackers gain those same broad database keys.
How an airbyte self host architecture works
A self-hosted deployment splits the platform into a few core services. The web app and server handle user requests and store sync schedules. A job scheduler tracks when each pipeline needs to run. When a timer fires, the scheduler spins up temporary worker pods or containers. One worker reads rows from the source, and another writes rows to the destination. An internal database holds job state, sync history, and stored target keys.
To run a sync, you create a user role inside each source and target database. You then paste those static passwords or API keys into the web interface. The platform saves those secrets in its local storage volume or an external secret store. Every time a worker starts a sync, it pulls the saved key and logs in. The database checks the role at login time and trusts all queries in that session.
What native setup solves and where gaps remain
Running your own data movement platform keeps raw records inside your private network. You control the compute size, sync timing, and network firewall rules between subnets. Native database roles also let you limit which schemas the sync user can read. This setup solves basic network isolation and stops public internet exposure for internal stores.
Still, native setup only checks who logs in at the start of a job. It does not inspect or filter what individual queries do while data moves in flight. Also, engineers who maintain the host servers need shell or database access to fix broken jobs. When admins debug worker nodes or inspect the state store, they can read stored keys. Static database roles also cannot rewrite sensitive fields like emails or card numbers during a query. Once a user or worker connects with a valid role, every matching row flows through untouched.
Self-Service Access Portals: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Securing admin sessions and database roles on your hosts
A strong setup separates pipeline worker roles from human debug sessions. First, give each source connector a read-only database role scoped to specific tables. Never reuse an admin user account for automated sync jobs. Second, lock down the internal state database so developers cannot query secret tables directly.
Third, monitor how engineers log into the host machines and backing databases during incidents. When a sync fails, engineers often open a shell on the host or query the source. If they share static admin keys for these checks, audit logs lose track of who ran what. Teams need a way to link every human command and query to an individual identity. They also need controls that rewrite sensitive fields like phone numbers or national IDs in query results.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
How should we scope database roles for an airbyte self host deployment?
Create a dedicated user for the platform in each source and target database. Grant read-only permissions on the exact source tables you plan to sync, and block everything else.
Why are static database keys a risk on self-hosted sync servers?
The platform stores source and target keys so background workers can run on a schedule. Anyone who gains admin access to the host or state database can extract those saved credentials.
How can engineers debug failed syncs without sharing root passwords?
Tie every server shell and database login to your central identity provider instead of shared keys. This links every debug command to a specific person and removes static passwords from laptops.