Buildkite Self-hosted Agents
You set up buildkite self hosted agents by running the agent binary on your own servers. The binary polls the control plane for work and runs build steps locally. Engineering teams often deploy these runners inside private cloud networks to reach internal databases and clusters. When teams attach broad cloud roles to these machines, every pipeline job inherits those wide permissions. A single bad script can then read sensitive data or alter production systems. How buildkite self hosted agents
Self-Service Access Portals: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up buildkite self hosted agents by running the agent binary on your own servers. The binary polls the control plane for work and runs build steps locally. Engineering teams often deploy these runners inside private cloud networks to reach internal databases and clusters. When teams attach broad cloud roles to these machines, every pipeline job inherits those wide permissions. A single bad script can then read sensitive data or alter production systems.
How buildkite self hosted agents work in your network
The architecture splits orchestration from execution. The vendor manages the web dashboard, schedules jobs, and stores build logs in the cloud. Your team manages the virtual machines or containers that execute the actual work. Each runner registers with the central API using a registration token over outbound HTTPS. You do not need to open inbound firewall ports on your network.
Teams group runners into queues using tags. A pipeline requests a specific queue, and a matching runner claims the job. The runner pulls the source code from your version control system. Next, it executes the local hook scripts and the pipeline steps defined by developers. When the step finishes, the runner streams the console output back to the central dashboard.
Runners also use local hooks to manage the job lifecycle. Admins place scripts on the host machine to run before or after every job. These hooks often fetch secrets, configure local tools, or clean up temporary files. Because hooks live on the host, developers cannot override them from a code repo.
What native agent roles solve and what they leave open
Running jobs inside your own network solves major connectivity problems. Your build machines can talk directly to internal artifact stores, private Kubernetes clusters, and staging databases. You also control the machine size, operating system packages, and local disk cache. Cloud identity bindings let the host machine assume an IAM role without storing static keys on disk.
Still, host-level identity creates a major operational gap. The cloud role decides what the machine can log in to, not what individual commands do. Every job scheduled on that queue shares the same underlying machine identity and network reach. If a runner can reach a production database for migrations, any job on that queue can query it.
Self-Service Access Portals: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Pipeline configs often live inside code repos where many developers can edit them. A developer can alter a build step in a pull request and run arbitrary commands on the runner. Native queue isolation forces you to build separate runner pools for every single permission boundary. Maintaining dozens of small runner pools increases operational toil and wastes compute budget.
Managing secrets and host state across pipeline runs
Shared runners also retain disk state and memory between jobs unless you wipe them. If one build writes a secret key to a temporary folder, the next build might read it. Many teams run ephemeral containers or single-use virtual machines to prevent cross-job leaks. Ephemeral hosts start clean for one job and terminate right after the job ends.
Even with ephemeral hosts, secret handling during the job remains a high risk. Hooks pull database passwords or API tokens into environment variables for the build step to use. Any sub-process or third-party package in that build step can read those variables and print them. Redacting known strings in build logs helps, but encoded outputs can still bypass simple text filters. Teams need fine-grained controls over the exact commands and queries that jobs execute in flight.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
Do buildkite self hosted agents require inbound firewall ports?
No, the runners only make outbound HTTPS requests to the central control plane. They poll for work and send logs out without needing any open inbound ports.
How do teams isolate untrusted pull request builds?
Teams route pull request jobs to a separate queue with restricted network rules and no production roles. They also use ephemeral hosts that destroy themselves after each run.
Where should secret keys live during a build step?
You should never store plain keys in pipeline files or on persistent runner disks. Fetch secrets into memory only for the specific step that needs them, then clear them.