Openmetadata Argo Workflows Connector
You set up an openmetadata argo workflows connector by pointing the catalog at your server endpoint. Then you provide a service account token with read permissions for workflow runs. Most teams start by granting broad cluster roles to make the sync job work fast. Those wide permissions create risk if a user or script misuses the shared token. How the openmetadata argo workflows connector works Data teams run complex jobs inside Kubernetes clusters every single day. Tracking what each pipeline
Access Request Workflows: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up an openmetadata argo workflows connector by pointing the catalog at your server endpoint. Then you provide a service account token with read permissions for workflow runs. Most teams start by granting broad cluster roles to make the sync job work fast. Those wide permissions create risk if a user or script misuses the shared token.
How the openmetadata argo workflows connector works
Data teams run complex jobs inside Kubernetes clusters every single day. Tracking what each pipeline does is hard without a central catalog. This connector links your workflow engine to your metadata store. It runs on a set schedule or as an external job. During each run, the ingestion task talks to the workflow server API. It reads template definitions, task steps, and past run states. Then it sends those details back to the central catalog.
To make this work, you must create a Kubernetes service account. That account needs a role that can read workflow objects across namespaces. You generate a bearer token for that service account. Next, you give the host address and token to the catalog ingestion service. The ingestion worker uses these credentials to pull metadata over HTTPS. Once the run finishes, engineers can see pipeline lineage and task status in one place.
Scoping service roles for metadata ingestion
A common mistake is attaching cluster admin rights to the ingestion service account. Engineers do this to avoid permission errors when adding new namespaces. Still, an admin token can delete pods, read secrets, or start new jobs. The catalog worker only needs to read workflow templates and execution results. It never needs to change cluster state or read app secrets.
You should build a custom Kubernetes role with read-only verbs. Limit this role to the specific API groups for your workflow engine. Bind that role only to the namespaces that run data pipelines. You should also rotate the service account token on a regular schedule. Storing the token in a secret manager keeps it out of plain text files. This basic step shrinks the blast radius if someone steals the credential.
Access Request Workflows: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
What native setup solves and what it misses
Native roles and tokens solve the basic problem of who can log in. They verify the identity of the catalog worker before allowing API calls. They also block the worker from touching unrelated parts of the cluster. For automated background sync jobs, scoped read roles work quite well.
The gap appears when human engineers need to check or fix broken workflows. A catalog shows that a pipeline failed, so an engineer logs in to fix it. Native cluster roles decide who gets in, not what specific commands do in flight. If an engineer has edit rights, they can run any command in that namespace. Static roles cannot inspect individual requests or mask sensitive fields like emails and card numbers. Teams also struggle to tie shared role actions back to a single person during audits.
Add runtime guardrails and masking beside your databases and APIs
Login rules, role grants, and CI checks govern who connects to your stack. Yet any allowed script, engineer, or AI agent still runs commands with the full scope of its role. For workloads on Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes, SSH, or internal HTTP services, hoop.dev sits beside the service as an open-source proxy, a sidecar for your database or API.
Before a query runs, hoop.dev guardrails check the statement against your rules. An AI Analyzer you configure can score intent or hold risky calls for human review. Next, hoop.dev data masking rewrites sensitive fields in responses, and each verdict is logged per listener. Explore the MIT-licensed project on GitHub.
Frequently asked questions
What permissions does the ingestion token actually need?
The token only needs read access to workflow templates, runs, and task logs. It does not need write permissions or access to Kubernetes secrets.
Does the connector run inside the same Kubernetes cluster?
It can run inside the same cluster or from an external orchestrator. Either way, it needs network routing to reach the workflow server API endpoint.
How often should the metadata sync job run?
Most teams run the sync job every few hours or once a day. Running it too often can put extra load on your workflow server API.