Elastic Push Integrations
You set up elastic push integrations by giving a source app an API key. The app then sends log and event records straight to your search cluster. Most teams start this setup by creating a broad write role for the sender. That quick choice saves time at first, but it creates big security risks later. If a shared key leaks, anyone can push bad records or change index settings. How elastic push integrations work In a pull model, an agent reaches out to read data from your apps. A push model turn
Push-Based Authentication + Elastic Security: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up elastic push integrations by giving a source app an API key. The app then sends log and event records straight to your search cluster. Most teams start this setup by creating a broad write role for the sender. That quick choice saves time at first, but it creates big security risks later. If a shared key leaks, anyone can push bad records or change index settings.
How elastic push integrations work
In a pull model, an agent reaches out to read data from your apps. A push model turns this flow around so the source app starts the call. The app formats events as JSON and sends them over HTTPS to your cluster. Your cluster checks the API key or user role on every single incoming batch. If the role matches the target index, the cluster writes the new records down.
To build this well, you need to set up strict role rules first. Create one role per source app instead of sharing one key across the team. Limit that role so it can only write new documents to one specific index. Never give a push worker the power to delete indexes or read old data. Next, set up an ingest pipeline inside the cluster to check the incoming fields. The pipeline parses text, drops blank fields, and puts timestamps in the right format. This keeps your search index clean even when an app sends messy log lines.
What native cluster roles solve and miss
Native cluster roles and API keys do a good job of stopping strangers. They check who is calling the cluster and which index they want to touch. You can revoke one key when you retire an app without breaking other senders. You can also separate write keys used by apps from read keys used by engineers.
Still, native setup only checks who logs in at the start of a request. It does not inspect what individual statements or queries do while they run. An engineer who troubleshoots a broken push pipeline often needs read access to production. Once that person logs in, a native role lets them pull entire raw documents. Those documents often hold sensitive fields like emails, phone numbers, card numbers, or national IDs. A static role cannot rewrite those fields in query results on the fly. It also cannot require a quick peer check before an engineer runs a delete call.
Push-Based Authentication + Elastic Security: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Managing day two operations for push pipelines
Keeping push pipelines healthy takes ongoing work from both app and platform teams. Apps change their log formats often, which can break strict index mappings during deploys. When a field type clashes, the cluster rejects the document and drops the event. Engineers then have to log into the cluster to read failed payloads and fix mappings.
This fix loop is where most security gaps show up in daily work. Teams hand out wide admin roles so developers can inspect dead letter queues fast. Those wide roles stay active long after the bug is fixed and closed. To keep data safe, you must control both the automated push keys and human sessions. Keep app keys scoped to write actions only, and rotate them on a schedule. For human sessions, inspect every request in flight and mask sensitive fields in results.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
How do push setups differ from pull collectors?
In a push setup, your app sends events straight to the cluster over HTTP. In a pull setup, a separate worker reaches out to scrape logs from your app.
What permissions should a push API key have?
Give the key permission to add new items to one specific data stream only. Do not allow the key to read data, delete indexes, or change cluster settings.
Why do sensitive fields end up in search indexes?
Apps often dump full request payloads into logs when an error happens in production. Those payloads carry user emails, phone numbers, and card numbers straight into your cluster.