Dataflow Machine Types
You set up dataflow machine types by matching worker CPU and RAM to your job workload. Pick standard instances for balanced jobs, high-memory workers for joins, and high-CPU workers for math tasks. Data teams often assign broad cloud roles to worker accounts just to make jobs run fast. Those broad roles let workers read or change data stores they should not touch. How to select dataflow machine types for your pipeline Google Cloud Dataflow runs Apache Beam pipelines on managed virtual machine
Machine Identity: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up dataflow machine types by matching worker CPU and RAM to your job workload. Pick standard instances for balanced jobs, high-memory workers for joins, and high-CPU workers for math tasks. Data teams often assign broad cloud roles to worker accounts just to make jobs run fast. Those broad roles let workers read or change data stores they should not touch.
How to select dataflow machine types for your pipeline
Google Cloud Dataflow runs Apache Beam pipelines on managed virtual machines called workers. When you submit a job, the service boots these workers to read, change, and write data. The machine type you pick sets the virtual CPU count and memory size for each worker. It also caps the network speed and persistent disk speed available to that node.
Standard machine families give a steady ratio of memory to each virtual processor core. They work well for simple extract and load pipelines that move rows between storage buckets. High-memory machine families give more gigabytes of RAM for each processor core on the host. Teams pick high-memory nodes when jobs hold large lookup tables or group wide records in memory.
Compute-optimized families focus on fast clock speeds and give less memory per processor core. These nodes fit streaming pipelines that parse complex text, run heavy math, or compress files. If your job uses the Shuffle service or Streaming Engine, heavy sorting moves off the workers. That shift lets you pick smaller worker nodes and save money on your cloud bill.
How autoscaling and worker permissions interact
Once you pick a worker family, the autoscaler adjusts the number of running nodes during the job. For batch jobs, it watches the backlog of unread data and adds workers to finish fast. For streaming jobs, it watches processor usage and backlog time to keep latency low. You can set a hard cap on worker count so a spike does not break your budget.
Every worker node boots with a cloud service account that holds identity and role bindings. That service account decides which storage buckets, message queues, and databases the worker can reach. Many teams share one default service account across all pipelines to save setup time. If a developer runs test code on a worker, that code inherits every permission on that account.
Machine Identity: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native worker setup solves and what it misses
Picking the right machine size stops out-of-memory crashes and keeps streaming lag within your targets. Setting a custom service account per pipeline limits which cloud resources that specific job can reach. Native cloud roles verify who can start a job and which databases the workers can reach. This setup handles basic network isolation and resource billing well for most data teams.
Still, cloud role bindings only check identity when a worker or engineer first opens a session. They do not inspect the actual queries or commands sent inside that active session. A user or worker with database write rights can still drop a table by mistake. Native machine settings also cannot mask sensitive fields like emails or card numbers in query results.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
When should I switch from standard to high-memory workers?
Switch to high-memory workers when job logs show out-of-memory errors during large joins or side inputs. Extra RAM per core stops workers from crashing when they hold big state tables in memory.
Does using Streaming Engine change the machine size I need?
Yes, Streaming Engine moves state storage and shuffle work off the worker virtual machines. You can often pick smaller standard machine types and use less persistent disk space per worker.
Why should each pipeline use a separate worker service account?
A shared service account gives every job the union of all database and storage permissions. Separate accounts ensure a test pipeline cannot read or overwrite production tables from another team.