Luigi Pipelines
You build luigi pipelines by defining Python task classes with inputs, logic, and output files. A central scheduler checks which output files exist and runs only the missing tasks. Data teams often give the worker host one broad database role to run every job. That broad role lets any broken task drop tables or read private user rows. How luigi pipelines work from task to output Every workflow in this Python framework starts with a basic task unit. Each task has three main parts that control
Bitbucket Pipelines Security: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You build luigi pipelines by defining Python task classes with inputs, logic, and output files. A central scheduler checks which output files exist and runs only the missing tasks. Data teams often give the worker host one broad database role to run every job. That broad role lets any broken task drop tables or read private user rows.
How luigi pipelines work from task to output
Every workflow in this Python framework starts with a basic task unit. Each task has three main parts that control how it runs. First, a task lists the upstream tasks that must finish before it starts. Second, it points to a target, such as a file or database marker. Third, it holds the Python code that reads data and writes the new output.
When you start a batch job, the worker checks the target of the final task. If that output file or table row already exists, the job stops right there. If the output is missing, the worker checks the upstream tasks in the chain. It walks backward through the dependency graph until it finds finished outputs. Then it runs the remaining tasks in order from start to finish.
Teams can run jobs with a local scheduler for quick tests on one machine. For shared production jobs, teams run a central scheduler daemon on a server. The central planner makes sure two workers never run the exact same task at once. It also gives engineers a simple web view to check failed or waiting tasks.
Why worker roles need tighter database rules
Native task planning solves dependency order and stops duplicate batch runs across workers. Still, the scheduler does not manage how tasks log in to your data stores. Most teams store static database keys on the worker host or in host files. Every task on that worker uses the same broad role to read and write data.
Native roles decide who can log in, not what each query does in flight. Once a worker logs in, the database trusts every command from that session. A bug in a Python script can delete a production table by mistake. A task that only needs counts might pull raw emails, phone numbers, or card numbers. Static keys on worker hosts also stay valid long after a job finishes.
Bitbucket Pipelines Security: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Steps to keep batch worker runs safe
Start by splitting worker permissions so one task does not hold every database key. Give read-only tasks a separate role from tasks that write or update tables. Keep secrets out of task code and off shared worker disk drives. Check query logs often so you know which task ran each database command.
Teams also need controls that inspect queries and commands at the moment they run. Blocking drop commands or rewriting sensitive fields like national IDs in results cuts risk. When you pair clean task graphs with statement checks, batch jobs stay safe and predictable.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
How does a task know if it already ran?
Each task checks its target output before it runs any job logic. If the target file or database marker exists, the worker marks the task complete.
What happens when an upstream task fails during a run?
The worker stops that branch of the graph and skips all dependent downstream tasks. Since the failed task writes no final target, the next run retries it.
Why is a single shared worker database key risky?
A shared key gives every Python task the exact same database permissions. Any bad script or team member with worker access can run destructive queries.