Buildkite Elastic CI Stack
You set up the buildkite elastic ci stack by deploying a cloud template in your account. This template starts and stops build machines based on your job queue size. Engineering teams use this setup to run tests inside their own network. Still, teams often attach broad cloud roles to these build machines. When a machine has wide permissions, any script in the pipeline can use them. A bad script or a stray command can read or change live data. How the buildkite elastic ci stack works The buildk
GitLab CI Security + Elastic Security: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up the buildkite elastic ci stack by deploying a cloud template in your account. This template starts and stops build machines based on your job queue size. Engineering teams use this setup to run tests inside their own network. Still, teams often attach broad cloud roles to these build machines. When a machine has wide permissions, any script in the pipeline can use them. A bad script or a stray command can read or change live data.
How the buildkite elastic ci stack works
The buildkite elastic ci stack runs agents inside your own cloud account. You keep the control dashboard in the cloud service, while your servers run the jobs. A central scaler watches your build queue for new work. When developers push code, the queue grows and triggers the scaler. The scaler then boots up virtual machines inside an auto-scaling group.
Each machine starts up, registers with the main service, and pulls a job. The agent runs your build steps, unit tests, and deploy scripts. After the job finishes, the machine waits for new tasks or shuts down. This design keeps cloud bills low when your team is not pushing code.
To do real work, these build agents need permissions in your cloud account. The template attaches an instance role to every virtual machine in the pool. Agents also pull secrets from cloud storage to log in to databases and internal apps. Every job that lands on a machine shares that machine's network and role.
Why agent roles need tighter controls
This native setup solves the problem of scaling build servers up and down. It also keeps your source code and build tools inside your private network. You do not need to expose internal servers to the public internet for deploys. Your cloud role decides which services the build agent can reach.
Still, cloud instance roles leave a big operational gap during the build run. A role only checks who logs in at the start of a connection. It does not check what individual SQL queries or shell commands do in flight. Once a build step logs in to a database, it has full role rights.
GitLab CI Security + Elastic Security: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Also, cloud logs only show that the shared machine role ran an action. They do not tie a specific database query back to the person who triggered the build. If a deploy script runs a bad delete statement, the database just runs it. If a test queries production tables, raw emails, phone numbers, and card numbers appear in build logs. Masking rewrites matched fields in results so sensitive data stays out of logs.
Managing secrets and network rules on build agents
Teams often try to fix this gap by creating many separate agent queues. They set up one queue for testing and another queue for production deploys. Each queue gets its own cloud stack and its own instance role. This limits which jobs can touch live databases or production servers.
Yet this approach multiplies the number of cloud stacks your team must maintain. You end up patching, updating, and paying for idle machines across many small pools. Even inside a locked-down deploy queue, the agent still holds full credentials during the job. A single mistake in a deploy script can still drop a table or leak keys. Teams need a way to inspect every command and mask data in real time.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
Where do build logs and test artifacts go in this stack?
Agents send build logs back to the main dashboard by default. You can also set up the stack to save artifacts in your own cloud storage bucket.
How do build agents get database keys and API tokens?
Agents use their cloud instance role to read secrets from your cloud secret store. A startup hook fetches the keys and puts them into the job session.
Why should teams avoid putting production keys on shared build queues?
Any developer who can edit the pipeline file can print or use those keys. Separating permissions and checking commands in flight prevents accidental damage to live systems.