Airbyte Self-hosted
You set up an airbyte self hosted instance by running its containers on your servers. Then you configure source and destination roles so workers can move data between databases. Most engineering teams deploy the platform on virtual machines or clusters inside a private network. They create database roles for each source and store those keys inside the platform workspace. When teams grant broad permissions to speed up setup, one bad sync can harm production. Engineers who log into the host server
Self-Service Access Portals: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up an airbyte self hosted instance by running its containers on your servers. Then you configure source and destination roles so workers can move data between databases. Most engineering teams deploy the platform on virtual machines or clusters inside a private network. They create database roles for each source and store those keys inside the platform workspace. When teams grant broad permissions to speed up setup, one bad sync can harm production. Engineers who log into the host server to check jobs also gain wide database permissions.
How an airbyte self hosted architecture works
The platform splits its work across several core services that run in containers. A web app gives your team a place to set up sources and schedules. A central server stores job states and secret keys in an internal database. A scheduler tracks time rules and starts sync jobs when they are due. Worker nodes do the heavy lifting by running isolated tasks for every data transfer. Each worker starts a source container to read records and a destination container to write them. You must open network paths from these worker nodes to every source and destination database. Teams usually place the whole deployment inside a private network and restrict external traffic. Admins then create dedicated database users for the workers to read tables and write rows.
What native setup solves and where gaps remain
Running the platform on your own servers keeps data pipelines inside your private network. You control the compute size, upgrade timing, and network rules for every worker node. Using dedicated database roles for each source stops workers from reading unrelated schemas. Still, this native setup only controls how the app logs into your data stores. It does not check what individual queries or commands do while a session runs. Moreover, engineers still need direct access to the host server and internal database for maintenance. When a sync fails, an engineer logs into the server or database to fix it. Native database roles cannot block an engineer from running a bad drop command by mistake. They also fail to mask sensitive fields like emails, phone numbers, or card numbers during checks. Teams lack a clear audit log that ties each manual query to a specific person. Shared worker keys also stay on disk for months without automatic rotation.
Securing engineer and worker database sessions
To close these gaps, teams must separate automated worker tasks from human maintenance sessions. Workers should use strict, read-only database roles that only touch the exact tables needed. You should never reuse these automated worker keys when an engineer needs to inspect data. Instead, require every person to log in with their own identity before running queries. Good operations also require checking every command in flight before it hits the database. Data masking rules should rewrite emails, phone numbers, and national IDs in query results. This approach keeps sensitive records safe while engineers fix broken data pipelines in production.
Add runtime guardrails and masking beside your databases and APIs
Login rules, role grants, and CI checks govern who connects to your stack. Yet any allowed script, engineer, or AI agent still runs commands with the full scope of its role. For workloads on Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes, SSH, or internal HTTP services, hoop.dev sits beside the service as an open-source proxy, a sidecar for your database or API.
Self-Service Access Portals: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Before a query runs, hoop.dev guardrails check the statement against your rules. An AI Analyzer you configure can score intent or hold risky calls for human review. Next, hoop.dev data masking rewrites sensitive fields in responses, and each verdict is logged per listener. Explore the MIT-licensed project on GitHub.
Frequently asked questions
How should we size servers for a new deployment?
Start with a modest virtual machine that has enough memory for concurrent worker containers. Watch memory use during peak sync hours and add compute nodes as jobs grow.
What database permissions do sync workers actually need?
Source workers only need read permissions on the specific schemas and tables you sync. Destination workers need permissions to create tables, write rows, and update schemas in target zones.
How do we protect sensitive records during pipeline checks?
Avoid giving engineers raw database keys when they check failed sync jobs. Use a setup that logs every query and rewrites card numbers and emails in results.
Short role checks help teams work fast. Simple rules make each review clear. Good logs keep every team safe. Scoped keys stop bad changes early.