Synapse Pipeline
A synapse pipeline is a cloud workflow that moves and changes data across systems. You set one up by linking data sources, adding tasks, and setting a schedule. Data teams build these workflows to pull raw files and load clean tables. Many teams start by giving the workspace broad permissions to read and write data. When roles are too wide, one bad task can change or delete key tables. How a synapse pipeline works in your workspace A synapse pipeline groups data tasks into one logical unit of
DevSecOps Pipeline Design: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
A synapse pipeline is a cloud workflow that moves and changes data across systems. You set one up by linking data sources, adding tasks, and setting a schedule. Data teams build these workflows to pull raw files and load clean tables. Many teams start by giving the workspace broad permissions to read and write data. When roles are too wide, one bad task can change or delete key tables.
How a synapse pipeline works in your workspace
A synapse pipeline groups data tasks into one logical unit of work. Each task does one job, like copying a file or running a SQL script. You chain these tasks together so one step finishes before the next step starts. You can set rules for what happens if a step passes or fails.
Linked services store the details needed to reach external storage accounts and databases. Datasets point to the specific folders, files, or tables inside those linked systems. Compute engines run the actual work when the workflow starts. You can start a run by hand, on a timer, or when files arrive.
To run these jobs, the workspace uses a managed identity or a service principal. Admins grant this identity a role on the target database or storage account. The role lets the workflow log in and move data without human help. Engineers also get workspace roles so they can build, test, and fix these workflows.
What native workspace roles solve and what they miss
Native workspace roles control who can edit workflows and who can start a run. Role assignments on storage and databases control which systems the workspace can reach. This setup stops outside users from reading your data or changing your jobs. It also lets automated jobs run on a schedule without storing plain text keys.
Still, workspace roles only check permissions when a user or job logs in. Once a role grants entry, every SQL command or script inside that session runs. A developer debugging a failed run often uses the same wide database permissions. That person can run an ad hoc query that drops a table by mistake.
DevSecOps Pipeline Design: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Standard roles also do not inspect the data returning from a query in flight. If a table holds emails, phone numbers, or card numbers, the query returns them. Masking rules that rewrite matched fields in results help protect those sensitive values. Teams also need clear logs tied to the exact person who ran each command.
Managing live database sessions during workflow debugging
When a scheduled data job breaks, engineers must inspect the source and target databases. They log in to check table schemas, row counts, and bad data formats. In many setups, teams share one powerful admin role to fix issues fast. Shared roles make it hard to see which person ran a specific change.
Splitting automated job roles from human debug access lowers the risk of mistakes. Automated workflows should only have the exact permissions needed for their normal tasks. Human engineers should run checks with individual identities and scoped rules for each query. Adding an approval step before high-risk SQL commands also prevents accidental data loss.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
What is the difference between a pipeline and a data flow?
A synapse pipeline manages the overall order, schedule, and control rules for your tasks. A data flow is one specific task inside that workflow that transforms data visually.
How do you trigger a synapse pipeline run?
You can start a run on demand during testing or set a clock schedule. You can also use a storage event trigger that starts when new files land.
Why do failed workflow runs create security risks?
Engineers often use broad admin roles on live databases to fix broken runs fast. Without query checks or data masking, a manual fix can expose or delete records.