Temporal Scalability
You achieve temporal scalability by adding history shards, scaling worker nodes, and tuning database persistence. This setup lets your workflow engine run more tasks per second as traffic grows. Teams often set up broad cluster roles to speed up this scaling work. Those wide permissions create risk when engineers run admin commands on live workflow state. How temporal scalability works in practice A workflow engine splits work across four main services. The frontend service takes requests fro
Scalability: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You achieve temporal scalability by adding history shards, scaling worker nodes, and tuning database persistence. This setup lets your workflow engine run more tasks per second as traffic grows. Teams often set up broad cluster roles to speed up this scaling work. Those wide permissions create risk when engineers run admin commands on live workflow state.
How temporal scalability works in practice
A workflow engine splits work across four main services. The frontend service takes requests from your client apps and worker nodes. It checks rate limits and sends calls to the right internal service. The matching service holds task queues in memory and hands out work to pollers. The history service tracks state changes and writes each step to the persistence store. Internal worker processes run background jobs like archival and system cleanups.
To scale the cluster, you first pick a high history shard count at setup time. You cannot change this shard count later without a full cluster migration. Each shard manages a slice of all running workflows in the system. As load grows, you add more history nodes so each host holds fewer shards. You also scale matching nodes and split hot task queues into more partitions. Finally, you check your database metrics to make sure disk writes stay fast.
Setting up worker roles and database permissions
Scaling workers requires clean separation between namespaces and service roles. Each app team gets its own namespace to isolate workflow runs and task queues. Worker nodes use service keys or certificates to talk to the frontend service. You set up namespace permissions so workers only poll queues they own. The engine itself uses a separate database role to read and write state tables.
During high load events, engineers often need to inspect stuck workflows or reset bad state. Teams grant admin roles so operators can run cluster tools and query system tables. In many setups, these admin roles share the same high permissions across all namespaces. Engineers log in with shared keys or broad cloud roles to fix urgent scale issues. This makes fast fixes easy, but it gives too much power to every user session.
Scalability: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native scaling solves and what keeps the session active
Native cluster tuning solves the hard math of routing and storing millions of workflow steps. Adding shards and database nodes stops queue lag and keeps task latency low. Namespace roles also do a good job of deciding which app can start a workflow. They stop a billing worker from polling a shipping queue by mistake.
Still, native setup rules only check who logs in at the start of a session. They do not inspect what individual admin commands or database queries do in flight. An engineer with cluster admin rights can terminate active workflows or read sensitive payload data. A broad database role lets someone run a heavy query that locks a busy shard table. Static roles also stay active long after an incident ends, leaving production systems exposed. Also, workflow payloads often hold user emails, phone numbers, card numbers, or national IDs. Without runtime checks, anyone querying workflow history or database tables sees those raw fields in results.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
Can I increase history shards on a running cluster?
No, the history shard count is fixed when you first set up the cluster. To raise it later, you must build a new cluster and migrate your workflows over.
Why do my workers sit idle when task queues are full?
Idle workers usually mean your matching service partitions or worker pollers are too low. Increase the poller count on your workers and check if the database is slow.
How do I protect sensitive data inside workflow payloads?
You can encrypt payloads in your worker code before sending them to the cluster. You can also use data masking to rewrite emails and card numbers in query results.