How to Configure Elasticsearch Column Level Security
You set up elasticsearch column level security by adding field rules to index roles. These rules tell the cluster which JSON fields a user role can read. Most engineering teams start by giving developers broad read roles on staging and production indices. Broad roles expose every field in a document, including emails, phone numbers, and card numbers. To fix this, admins build custom roles that list allowed or blocked fields. How elasticsearch column level security works In a document store, f
Your coding agents run with your access.
Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.
You set up elasticsearch column level security by adding field rules to index roles. These rules tell the cluster which JSON fields a user role can read. Most engineering teams start by giving developers broad read roles on staging and production indices. Broad roles expose every field in a document, including emails, phone numbers, and card numbers. To fix this, admins build custom roles that list allowed or blocked fields.
How elasticsearch column level security works
In a document store, fields act like columns in a relational table. The search engine calls this feature field level security inside its role settings. Admins define a role and attach it to a specific index pattern. Inside that role, you list the exact field names to grant or deny. When a user runs a search, the cluster checks their assigned roles first.
The cluster then strips blocked fields from the JSON document before sending the response. You can use wildcard patterns to match groups of related fields at once. For example, a rule can allow all metric fields while blocking billing fields. If a user belongs to two roles, the engine merges their allowed field lists. This means an overly broad second role can undo your strict field rules.
Why teams use field rules for data privacy
Search indices often hold app logs, user profiles, and payment records in one place. Support engineers and analysts need to search these indices to fix bugs fast. Still, they do not need raw national IDs or card numbers to do that work. Field rules let teams share index read rights without exposing every private field. Auditors also ask teams to prove that only approved roles read private fields.
Setting up these rules requires a clear map of every index schema. In a fast-moving company, developers add new JSON fields to logs every week. If a developer adds a new field, grant lists hide it until an admin updates roles. Conversely, deny lists expose the new field until an admin adds it to the block list. Keeping role definitions in sync with changing log schemas takes constant manual work.
Column-Level Encryption + Elasticsearch Security: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Where native index role permissions fall short
Native field rules control which fields vanish altogether from a search result. They work well when a role never needs to see a field exist. Yet many debug tasks require seeing that a field holds data without reading raw values. Native field security also depends on the license tier running on your cluster. Many teams run open distributions or basic tiers that lack built-in field rules.
Also, static roles decide who logs in and which index fields they can read. They do not inspect individual queries in flight or mask values inside unstructured text fields. If an app writes an email or phone number inside a free-text message field, field rules miss it. The user still reads the whole message field if their role allows it. Teams also have to manage separate role mappings across every single cluster they run.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev sits inline in the protocol beside each supported service as an open-source proxy, a sidecar for your database or API. For every query or command an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human approval.
First, hoop.dev guardrails block banned queries before they run or hold risky ones for review, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
How does Elasticsearch handle column rules on nested JSON objects?
You reference nested fields using dot paths inside the role definition. The cluster evaluates the path and removes matching sub-fields from the returned document.
What happens when a user runs an aggregation on a restricted field?
If a role denies read rights to a field, aggregations on that field return empty results. The engine treats the restricted field as if it does not exist for that user.
Can field rules protect sensitive values inside free-text log messages?
No, native field rules only hide or show entire JSON keys by name. They do not scan or rewrite emails or card numbers buried inside a text string.
Related guides
In this post