Airbyte Scaling
Teams handle airbyte scaling by moving from a single server to a cluster. You add worker nodes and split sync jobs across separate pods. Many teams start by giving the platform one broad database role. That role can read and write every table in the system. When sync volume grows, this broad role creates serious data risks. How airbyte scaling works in production A basic setup runs all tasks on one virtual machine. This works fine when you only run a few small syncs. As data grows, single mac
Scaling: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
Teams handle airbyte scaling by moving from a single server to a cluster. You add worker nodes and split sync jobs across separate pods. Many teams start by giving the platform one broad database role. That role can read and write every table in the system. When sync volume grows, this broad role creates serious data risks.
How airbyte scaling works in production
A basic setup runs all tasks on one virtual machine. This works fine when you only run a few small syncs. As data grows, single machines run out of memory and CPU. To grow further, teams move the platform onto a container cluster.
In a cluster, a main scheduler service watches the job queue. When a sync starts, the scheduler spins up new worker pods. One pod reads from the source, and another writes to the target. Each pod runs its task and shuts down when the job finishes. You can run many syncs at the same time this way. Teams also move the internal state database to a managed service. External storage holds the job logs so pods stay light and fast.
Managing worker resource limits
Cluster nodes still need clear limits to run well under load. Heavy database syncs can eat up all the memory on a node. If one pod takes too much memory, other jobs will crash. Platform teams set strict CPU and memory limits for each worker type.
You can give more memory to pods that read big tables. Smaller syncs can run in tiny pods to save cluster costs. Teams also set a cap on how many jobs run at once. This cap stops a rush of syncs from breaking the source database. Slow and steady job queues keep both the cluster and databases safe.
Scaling: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native scaling solves and what it misses
Native cluster setups solve the problem of slow and stuck data pipelines. You can move billions of rows without crashing your main servers. Role settings in the app also control who can trigger a sync. These settings decide which team members can log in and edit pipelines.
Still, cluster scaling does not control what queries do in flight. Worker pods still use static keys stored inside the app database. Engineers also need direct database roles to check failed sync jobs. Once a person or pod logs in, the database trusts every query. A bad query can read raw emails, phone numbers, or card numbers. Native scaling moves data fast, but it leaves raw records exposed during fixing. Teams need a way to check queries and mask fields in flight.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
When should a team move from one server to a cluster?
Move to a cluster when sync jobs wait in long queues. Frequent out-of-memory crashes also mean you need separate worker pods.
Why do sync jobs fail when we add more workers?
More workers open more database sessions at the same time. If you pass the database session limit, new sync jobs will fail.
How do we protect sensitive fields during pipeline checks?
Use dynamic data masking to rewrite emails and card numbers in results. This lets engineers fix broken pipelines without reading raw customer records.