Airbyte Medium
You set up an airbyte medium pipeline by linking the source connector to a warehouse. First, generate an API token in the publishing app and enter it in Airbyte. Data teams often run this setup to pull post stats and reader numbers. Many engineers use a broad admin role to get the first sync working fast. That wide role creates risk when the job pulls sensitive records into staging tables. How an airbyte medium sync works The source connector reads data from the publishing API on a set timer.
Medium: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up an airbyte medium pipeline by linking the source connector to a warehouse. First, generate an API token in the publishing app and enter it in Airbyte. Data teams often run this setup to pull post stats and reader numbers. Many engineers use a broad admin role to get the first sync working fast. That wide role creates risk when the job pulls sensitive records into staging tables.
How an airbyte medium sync works
The source connector reads data from the publishing API on a set timer. You pick a start date and choose which data streams to copy over. The worker container then pulls articles, tags, and user metrics in small batches. Next, the destination connector writes those records as raw tables in your database. Each sync run logs row counts and job status in the main dashboard.
Teams also need to set up database roles for the destination writer. A good setup gives the writer role permission to create tables in one schema. You should keep this role separate from the roles that analysts use daily. This separation stops a broken sync job from changing core production tables. Clear role rules make the pipeline much easier to maintain over time.
What native sync permissions solve and what keeps the session active
Native source tokens and database roles control which service can log in and write. They stop outside users from reading your publishing stats or dropping warehouse tables. Workspace roles in the sync tool also limit who can edit pipeline settings. These built-in checks handle basic login rules for your data stack.
Still, login checks do not inspect what individual queries do during a live session. Once an engineer or worker logs in, the database trusts every command they send. A user with read permissions can pull full tables with emails and phone numbers. Static roles cannot rewrite card numbers or national IDs in query results by default. Teams also lack a clear log of every SQL statement run by human users.
Medium: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Managing database sessions after data lands
After the sync finishes, analysts and apps query the new tables for reports. Teams often share one database login across many scripts and notebooks to save time. When a bad query locks a table, nobody knows which person ran it. Shared keys also stay valid for months and are hard to rotate safely.
A safer workflow ties every database session to a real person on your team. You should check each query before it runs and log who sent it. Masking rules can also rewrite emails and card numbers in the query output. This approach keeps daily work fast while protecting raw fields in your warehouse.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
How do I limit what the sync worker can change in my database?
Create a dedicated database role just for the sync worker to use. Grant that role write permissions on one raw landing schema and nothing else.
Can native database roles hide emails and phone numbers in query results?
Standard table permissions only allow or block a user from reading a table. To rewrite matched fields in results, you need dynamic data masking rules in place.
Why should teams avoid sharing one database key for reporting tools?
Shared keys hide who ran a specific query in your database audit logs. If that single key leaks, you must update every app at the same time.