How to Configure Elasticsearch Audit Log

Published
Length
3 min read

Luis Felipe Dias

All posts

You set up an elasticsearch audit log by turning on security audit settings on each cluster node. Then you pick which user actions and login events the node writes to disk. Most teams start by logging every single read and write request across the cluster. When roles are too broad, shared accounts flood these files with unnamed queries. Teams soon face huge storage bills and slow nodes without knowing who ran a bad request. How the native elasticsearch audit log works Each node in your cluste

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You set up an elasticsearch audit log by turning on security audit settings on each cluster node. Then you pick which user actions and login events the node writes to disk. Most teams start by logging every single read and write request across the cluster. When roles are too broad, shared accounts flood these files with unnamed queries. Teams soon face huge storage bills and slow nodes without knowing who ran a bad request.

How the native elasticsearch audit log works

Each node in your cluster runs its own local audit engine. When a user or app sends a request, the node checks their role first. The security module then records the outcome of that check in a local file. You must turn this setting on inside the setup file on every single node. If you miss one node, any request hitting that host goes unrecorded.

By default, the engine tracks both transport traffic and web traffic. Transport traffic includes internal messages passed between nodes in the cluster. Web traffic covers external search queries and data updates from users and apps. Recording internal node messages creates massive log volume very fast. Most admins turn off transport logging and only record external web requests.

You can also tune which event types the cluster writes down. Teams often record failed logins and blocked requests first to spot intruders. If you need to track data reads, you can log granted permissions too. Be careful when you turn on request body logging for search queries. That setting writes the full text of every search and document update to disk. It can expose raw emails or card numbers inside your log files.

What native audit logs solve and what they miss

Native audit files help you prove that role checks are working on your cluster. They give security teams a record of failed logins and permission changes. If someone tries to guess a password, the log captures their source address. You can ship these files to a central index to build dashboards and alerts.

Still, node logs only record events after the cluster processes them. They decide who logs in, not what individual queries do in flight. Once an engineer logs in with a broad role, they can run any command. The log will record that they deleted an index, but it cannot stop them. It also cannot require a second person to approve a bulk delete before it runs.

Audit Log Integrity + Elasticsearch Security: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

Shared roles make these logs even harder to use during an incident. Many teams give one read role or API key to an entire group. When someone runs a query that pulls sensitive phone numbers, the log shows the shared role. You see the query text, but you do not see which person typed it. Native logs also do not rewrite sensitive fields like national IDs in query results. If a user has read rights on an index, they get every field back in plain text.

How to keep audit volume low and useful

To keep your cluster fast, filter out noisy service accounts right away. Monitoring tools and health checks hit the cluster every few seconds. You can add exclusion rules that skip audit logging for specific system users. You can also ignore read requests on internal system indices. This keeps your disk usage predictable and makes real user actions easier to find.

Make sure you test your exclusion rules in a staging cluster first. A broad filter rule can hide real admin actions by mistake. Check your disk space often when you change what the audit engine records. Store the output files on a separate disk so full logs never crash the node.

Govern statements and responses across databases and APIs

Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev sits inline in the protocol beside each supported service as an open-source proxy, a sidecar for your database or API. For every query or command an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human approval.

First, hoop.dev guardrails block banned queries before they run or hold risky ones for review, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.

Frequently asked questions

Does turning on the audit log slow down my cluster?

Yes, logging every read and write event adds extra disk and CPU work. You can reduce this load by logging only failed logins and admin changes.

Do I need to configure audit settings on every node?

Yes, audit settings are local to each host in the cluster. You must apply the exact same rules to every node so you do not miss events.

Should I turn on request body logging in production?

Only turn it on if you have strict rules for who reads the log files. Request bodies contain raw data, which can put emails or card numbers into plain text logs.

The query runs. The emails stay hidden.