Airbyte Self Hosted
You set up an airbyte self hosted instance by running its server and worker apps on your own cloud hosts. This setup lets your team move data between sources and targets inside your private network. Most teams start by giving the sync workers broad database roles to make setup fast. Broad roles create big risks when a worker or user runs a bad query. How an airbyte self hosted architecture works The platform splits its work across a few core parts. A web app and an API server take user reques
Self-Service Access Portals: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up an airbyte self hosted instance by running its server and worker apps on your own cloud hosts. This setup lets your team move data between sources and targets inside your private network. Most teams start by giving the sync workers broad database roles to make setup fast. Broad roles create big risks when a worker or user runs a bad query.
How an airbyte self hosted architecture works
The platform splits its work across a few core parts. A web app and an API server take user requests and schedule jobs. A workload manager spins up short-lived worker pods for each data sync. One pod reads rows from the source database. Another pod writes those rows into the target database.
A central database holds the job state, schedules, and saved keys. You can also store database keys in a cloud secret store. When a sync starts, the worker fetches the needed key and logs in. The worker keeps that database session open until the batch finishes. Network rules must let these worker pods reach every source and target host.
Admins often put the web app behind an internal login check. Once an engineer logs into the web app, they can view all saved sources. They can also trigger manual sync runs or edit target settings. The native role system in the open edition gives every logged-in user wide control.
What native setup solves and where gaps remain
Running the tool on your own hosts keeps raw data off third-party clouds. Your team controls the compute size, network paths, and storage disks. You can place the worker nodes in the same virtual network as your databases. This layout cuts data transfer fees and keeps traffic off the public web.
Still, the native setup leaves a major gap during live database sessions. A database role only checks who logs in at the start of a job. It does not check individual queries or commands while they run in flight. If a connector has write rights, a bug or bad edit can drop tables.
Self-Service Access Portals: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Also, engineers often need to check the backing database or host nodes to fix failed jobs. Giving engineers direct SSH keys or database passwords creates shared, standing access. Shared keys make it impossible to see which person ran a specific query. Static read roles also return raw sensitive fields like emails, phone numbers, and card numbers. Without field-level controls, sensitive user records flow straight into lower test databases or worker logs.
Managing day two operations and worker access
Day two work requires tight rules around both sync workers and human operators. Start by giving each source connector a read-only database user limited to required tables. Give each target connector write rights only on the landing schema it owns. Do not reuse the same database user across different sync pipelines or apps.
Next, separate how the app runs from how your engineers fix broken jobs. Engineers will need to check job state tables or inspect stuck worker pods. If they log in with a shared admin role, your audit trail breaks down. You need a way to tie every command and query to a single human identity. You should also mask sensitive fields like national IDs in query results during debug sessions.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
How should we scope database roles for sync workers?
Create a dedicated database role for each source and target in your pipelines. Grant read-only rights on specific source tables and write rights only on target landing schemas.
Where does an airbyte self hosted instance store database keys?
By default, the platform stores keys inside its own internal database. For better safety, you should link the deployment to an external cloud secret manager.
Why is SSH access to worker hosts a security risk?
Worker hosts hold active memory and keys for every database in your data pipelines. Direct SSH access lets an engineer read those keys or run untracked commands on the host.