Data Factory Data Flow
A data factory data flow lets you build visual data jobs without writing code. It runs these transformations on managed cloud clusters that scale up and down. Teams often set up these pipelines using one shared service account. That shared account usually holds broad read and write roles across many databases. When a developer opens a live debug session, they use those same wide permissions. A single bad query or test run can alter or expose production tables. How a data factory data flow work
Data Flow Diagrams (Security): the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
A data factory data flow lets you build visual data jobs without writing code. It runs these transformations on managed cloud clusters that scale up and down. Teams often set up these pipelines using one shared service account. That shared account usually holds broad read and write roles across many databases. When a developer opens a live debug session, they use those same wide permissions. A single bad query or test run can alter or expose production tables.
How a data factory data flow works in practice
First, an engineer sets up a linked service to reach a source database. This link stores a secret key or uses a managed cloud identity. Next, the engineer adds source blocks, filter steps, and sink blocks on a canvas. The service turns this visual graph into a job script behind the scenes.
When the pipeline runs, the service spins up a Spark cluster to do the work. The cluster uses the linked service role to log in to the database. It pulls raw rows into memory, applies the rules, and writes out clean tables. During build time, developers turn on a debug mode to preview live data rows. This preview mode uses the exact same linked service credentials to read source tables.
What native roles solve and what they leave open
Cloud role rules control who can edit pipelines or trigger a scheduled run. Managed identities also help teams avoid pasting raw database passwords into job settings. These cloud tools decide which workspace or service can log in to a database. They do a good job of locking down the outer cloud control plane.
Still, native cloud roles stop at the login step. Once the cluster logs in, the database trusts every command on that link. The cloud role cannot see or filter individual queries in flight. If a source table holds emails, phone numbers, or card numbers, the preview shows them. Masking rewrites matched fields in results, yet linked services return raw rows by default. Also, database logs only show the shared workspace identity, not the individual developer.
Data Flow Diagrams (Security): architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Managing live debug queries and sensitive data
Data teams need a way to inspect queries after the login step happens. Many teams try to fix this by building separate test databases with fake rows. Keeping those test tables in sync with production takes hours of manual work every week. Other teams grant read roles directly to engineers so they can check source issues. Direct roles create even more places where raw card numbers or national IDs leak out.
To reduce risk, teams must check every command and result row in real time. Each query needs a clear link back to the person who ran the test. Sensitive fields like emails and phone numbers need automatic masking before they reach a screen. High-risk write statements also need a quick peer review before they touch production tables.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev sits inline in the protocol beside each supported service as an open-source proxy, a sidecar for your database or API. For every query or command an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human approval.
First, hoop.dev guardrails block banned queries before they run or hold risky ones for review, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
What is the difference between a pipeline and a data flow?
A pipeline moves data and controls the order of steps in a job. A data flow sits inside a pipeline to clean, join, and reshape rows in memory.
How do linked services handle database keys during a run?
A linked service stores a secret key in a vault or uses a cloud identity. The worker cluster reads this identity at runtime to log in to the database.
Why do audit logs miss which developer ran a data preview?
The database only sees the shared identity attached to the cloud workspace. It cannot tell which logged-in user clicked the preview button inside the visual tool.