Google Dataflow Architecture
You build a google dataflow architecture by linking a pipeline runner with managed worker nodes. The control plane reads your job graph and scales compute nodes to process data. Most engineering teams set up this flow with a single shared service account. They give that account broad project roles so jobs do not fail during runs. Broad roles let any job or developer read or alter every table and bucket. How a google dataflow architecture runs pipelines A google dataflow architecture splits pi
Zero Trust Architecture: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You build a google dataflow architecture by linking a pipeline runner with managed worker nodes. The control plane reads your job graph and scales compute nodes to process data. Most engineering teams set up this flow with a single shared service account. They give that account broad project roles so jobs do not fail during runs. Broad roles let any job or developer read or alter every table and bucket.
How a google dataflow architecture runs pipelines
A google dataflow architecture splits pipeline logic from the servers that run the work. Developers write code using the Apache Beam model to define data steps and transforms. When you submit a job, the service reads the code and builds an execution graph. It sends this graph to the control plane, which schedules tasks across worker nodes.
Two main service roles make this workflow operate from start to finish. The service agent manages the cloud resources, starts virtual machines, and sets up network paths. The worker service account lives on the compute nodes and reads or writes your data. Workers pull input records from message queues, object buckets, or SQL databases. They run the transforms in memory and push results to the final storage sink.
The service also handles scaling and state storage without manual steps from your team. For stream jobs, the backend tracks watermarks and holds state data in fast storage disks. If input traffic spikes, the control plane adds more worker nodes to clear the backlog. When the load drops, the system removes idle nodes to save compute costs.
How to set up worker roles and network rules
Start by creating a dedicated service account for each pipeline instead of sharing one key. Grant the user submitting the job permission to act as that specific worker account. Give the worker account read roles only on the exact source buckets or tables needed. Add write roles only on the output sink and the temporary staging bucket.
Next, lock down the network paths where your worker nodes run their tasks. Turn off public IP addresses on worker nodes so they stay inside your private network. Use private service paths to let workers reach cloud storage and database APIs. Set firewall rules so worker nodes can talk to each other on required ports. This setup keeps raw data traffic off the public internet during job execution.
Zero Trust Architecture: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native roles solve and what they miss
Cloud identity roles control which user or service account can start a pipeline job. They also decide which storage buckets and databases the worker nodes can reach. Network rules and private subnets keep outside traffic away from your active compute nodes. Together, these native controls stop unapproved users from launching jobs or reading storage buckets.
Still, native cloud roles only check who logs in when a session starts. Once a worker node connects to a database, it can run any query allowed by its role. The native setup does not check individual SQL statements or commands in flight. If an engineer inspects a database during a failed run, broad roles expose raw records. Native roles also cannot rewrite sensitive fields like emails, phone numbers, or card numbers in query results.
Add runtime guardrails and masking beside your databases and APIs
Login rules, role grants, and CI checks govern who connects to your stack. Yet any allowed script, engineer, or AI agent still runs commands with the full scope of its role. For workloads on Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes, SSH, or internal HTTP services, hoop.dev sits beside the service as an open-source proxy, a sidecar for your database or API.
Before a query runs, hoop.dev guardrails check the statement against your rules. An AI Analyzer you configure can score intent or hold risky calls for human review. Next, hoop.dev data masking rewrites sensitive fields in responses, and each verdict is logged per listener. Explore the MIT-licensed project on GitHub.
Frequently asked questions
What is the difference between the service agent and the worker account?
The service agent is a managed identity that creates virtual machines and disks for your job. The worker account is the identity on those machines that reads and writes your pipeline data.
Why should teams avoid using the default compute service account for jobs?
The default compute account often holds broad editor permissions across the entire cloud project. Using it lets a single pipeline job read or delete unrelated databases and storage buckets.
How does the system handle temporary files during a pipeline run?
The runner writes code packages and intermediate shuffle data to a cloud storage staging bucket. The worker service account needs read and write permissions on that bucket to finish the job.