How to Configure Opensearch PII

Published
Length
3 min read

Luis Felipe Dias

All posts

You set up opensearch pii protection by adding role rules and field masking to your cluster. You also filter out sensitive values during data ingestion before the engine indexes them. Many teams start by giving developers broad read roles across all search indexes. When roles are too broad, anyone running a query can read raw emails and card numbers. How to set up native opensearch pii controls Search clusters store large volumes of app logs, user events, and text records. These JSON document

Your coding agents run with your access.

Claude Code reaches prod with a developer's credentials, and approval prompts get clicked through. hoop.dev checks every query, masks sensitive data, and holds risky changes for a human.

You set up opensearch pii protection by adding role rules and field masking to your cluster. You also filter out sensitive values during data ingestion before the engine indexes them. Many teams start by giving developers broad read roles across all search indexes. When roles are too broad, anyone running a query can read raw emails and card numbers.

How to set up native opensearch pii controls

Search clusters store large volumes of app logs, user events, and text records. These JSON documents often hold emails, phone numbers, card numbers, and national IDs. To protect this data natively, teams work in two main areas. They clean data during ingestion, and they restrict fields when users read data.

First, you can set up an ingest pipeline inside the search engine. The pipeline checks incoming text before the cluster writes it to disk. It uses text patterns to find emails or phone numbers in log lines. When a pattern matches, the pipeline replaces the raw text with a placeholder string. This step keeps sensitive values out of the stored index from day one.

Second, you can use the built-in security plugin to manage read roles. The security plugin lets admins set up field-level security and field masking rules. With field-level security, you pick which JSON fields a specific role can read. If an index holds a card number field, you hide it from basic roles.

Native field masking works in a similar way for structured fields. Instead of hiding the whole field, the cluster hashes or replaces the text value. You link these rules to user roles in the security setup. When a user logs in, the cluster checks their role and rewrites matched fields in results.

Where native search roles fall short in daily work

Native security controls work well when your data sits in neat, predictable fields. Field-level security stops basic users from reading known columns like email or phone number. Ingest pipelines catch standard text patterns before logs land in storage. These tools help teams manage dashboard users who only need summary charts.

Still, native role rules decide who logs in, not what ad-hoc queries do in flight. During an outage, engineers often need broad read roles to check raw error logs. A shared admin role lets them query any index and run any search request. If an app dumps a full payload into a free-text message field, field security misses it.

End-to-End Encryption + PII in Logs Prevention: architecture patterns and best practices

Continue reading? Get the full guide.

Free. No spam. Unsubscribe anytime.

Maintaining text patterns across dozens of native roles and indexes also takes constant work. Every time a team adds a new index, admins must update the role rules. If someone forgets a rule, raw national IDs or card numbers show up in search hits. Also, engineers who connect with local API tools often share static cluster keys. Static keys make it hard to track which person ran a specific search query.

Why teams need query and result checks in flight

To close this gap, teams need controls that check each search request and response live. Static role mapping cannot predict every unstructured log message an engineer might query. Checking data in flight lets you catch sensitive patterns across all fields at read time. This approach rewrites matched emails and phone numbers in results regardless of the index layout.

Live request checks also give security teams clear audit logs tied to real user identities. Instead of seeing a shared admin key, you see who ran each search command. You can require peer approval for sensitive queries while keeping daily log checks fast.

Check database and API queries on the wire

Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev sits inline in the wire protocol beside those services as an open-source proxy, a sidecar for your Postgres or API. On every query an AI agent or engineer runs against production, hoop.dev decides to pass, mask, block, or hold it for human review.

Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run, and hoop.dev data masking rewrites matched sensitive values in responses in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.

Frequently asked questions

How does native field masking work in a search cluster?

Native field masking attaches to a security role and targets specific JSON field names. When a user with that role runs a search, the engine hashes or replaces those fields.

Why do sensitive values still appear when field security is active?

Field security only protects named fields that you list in the role setup. If an app writes emails or card numbers inside an unstructured log string, the text still shows up.

What is the difference between ingest filtering and read-time masking?

Ingest filtering removes sensitive values before the cluster saves the document to disk. Read-time masking keeps the stored record intact and rewrites matched fields in results when users query it.

The query runs. The emails stay hidden.