logo

Integrate AI Data Masking with Apache Kafka (Kubernetes)

Ensuring the protection of sensitive data while enabling seamless Apache Kafka access for development and analytics is a critical challenge for organizations. Hoop.dev introduces an innovative data masking solution that leverages OpenID connect for authentication and utilizes advanced machine learning models for real-time data redaction. This guide will take you through the process of integrating Hoop.dev's data masking capabilities with your Apache Kafka to enhance security and ensure compliance with privacy regulations, all without disrupting your service.

How to integrate AI Data Masking with Apache Kafka + Hoop.dev

  1. Create a Hoop.dev account and get your authentication token
  1. Create a connection to access with AI Data Masking using the Hoop CLI
  1. Invite your team to access the connection with AI Data Masking using the Hoop CLI
If you don't need a tutorial and just want to get started, please consider reading the quickstart docs instead.

Apache Kafka support in Hoop.dev

You can use Hoop.dev with
as shown in the examples below, or alternatively use the
if your prefer.Hoop.dev gets out of the way and users experience the interface of any service as they are. The only changes are to make things more secure and easier to use. For instance: users don't have to run 10 steps in 3 systems to get a temporary credentials anymore, instead they use a single command, and security is more robust.

Why Hoop.dev AI Data Masking?

Enhanced Security

Hoop.dev's use of OpenID for authentication eliminates the reliance on password-based access, reducing the risk of credential leaks. The proxy-based approach to access management further ensures the protection of sensitive information from unauthorized exposure.

Real-Time PII Data Masking with AI

With its advanced machine learning algorithms, Hoop.dev provides real-time analysis and redaction of sensitive data within query outputs. This capability allows teams to access necessary data without the risk of exposing PII, thereby simplifying compliance with data privacy regulations.

Simplified Compliance

Implementing Hoop.dev's data masking solution aids organizations in adhering to stringent data protection regulations by automating the redaction of PII, thereby negating the need for extensive manual data handling or complex policy configurations.

Seamless Integration

Designed for easy integration with your existing systems, Hoop.dev requires minimal adjustments to current data access and utilization practices. The provision of read-only replicas ensures that service continuity is preserved.

Integrating AI Data Masking

Hoop.dev offers a CLI and Web interface for interacting with the connections. This guide will show how to use them.You can also refer to
for all the information needed to start using Hoop.dev

Main concepts

Hoop.dev has three important entities you will be interacting with:
  • connection: these are the services you'll manage. They have have credentials, attached policies, and a few other properties.
  • policy: policies are the rules you set for how users interact with a connection. Each connection can have multiple policies and each session for that connection will have a set of policies applied on the begging and during the session time.
  • session: these are the interactions of your users with the connections. They can be behave differently depending on the policies configured for the connection.

Install dependencies

Install the Hoop CLI to easily connect to your service via Hoop.
MacOS: Top this brew formula and install the CLI
shell
brew tap hoophq/hoopcli https://github.com/hoophq/hoopcli brew install hoop
Linux: Run this shell script to install the CLI (check the code here)
shell
curl -s -L https://releases.hoop.dev/release/install-cli.sh | sh

Step 1: Sign In and Authenticate

Begin by using the Hoop.dev Command Line Interface (CLI), which offers seamless integration with OpenID Connect for authentication. This process grants users access and eliminates the need for temporary credentials and significantly improving your security posture.
Authenticate your CLI with your Hoop.dev account. You can signup for an account here.
shell
hoop login

Step 2: Get your authentication token

Before you can start the Hoop proxy agent you need to get an authentication token. You can get it running the following command:
shell
hoop admin create agent demo
The authentication token will look something like this: grpcs://demo:xagt-SV9N48TQiD2TozyHFCAE4XsoKa21YF1GKLsvflAWSnV@use.hoop.dev:8443?mode=standard

Step 3: Run the Hoop.dev agent

To start the Hoop.dev process on your local computer, export your authentication token as an environment variable named HOOP_DSN and then use the CLI start the long-running process. The agent can run on any platform, from Linux VMs to container platforms like Apache Kafka. Check out the installation guides for details on how to deploy to other platforms.
shell
export HOOP_DSN=grpcs://demo:xagt-SV9N48TQiD2TozyHFCAE4XsoKa21YF1GKLsvflAWSnV@use.hoop.dev:8443?mode=standard hoop start agent

Step 4: Create a connection per profile

Connections are what users connect to. A connection represents one of the profiles, like or , for a given service. Each of your users will have access to one or more connections. The easiest way to create a new connection is when a new service is setup.
You'll need the connection name when using a connection. You can either find it in the web dashboard where all the connections you can access are easily visible, or list them with the CLI.
To create a connection you'll provide the connection name, the agent name (created in the previous step), and the credentials Hoop will use to access the service on the backend. Remember: these credentials are kept in the vault and are never exposed to the user.
Setup the connection with the Kafka CLI operation you want to perform
shell
hoop admin create conn my-kafka-cluster -a demo \ -- bin/kafka-console-consumer.sh --bootstrap-server HOST1:PORT1,HOST2:PORT2 --consumer.config config.properties --topic testTopic --property "print.key=true"

Connect with SSO (sessions)

We will now use this Kubernetes cluster with Hoop.dev CLI. You previously signed in with Google or Github, and this connection will use the same authentication. You can customize the default Google or Github federation with your own IDP with this guide.
Execute the operation with SSO
shell
hoop connect my-kafka-cluster

Helping your team access with SSO

In addition to securing your own connections to a service, Hoop.dev provides several resources to make it easy to manage your team when accessing these services. On top of that they'll get a great developer experience while using it.
We offer an intuitive and easy to use internal developer platform to easily connect to any service which you can share with internal user directly. Simply invite them in the settings and they'll be able to use it.
We also offer easy to use docs for how to easily connect to any service which you can share with internal user directly.
Finally, the Web editor provides users with a consistent interface on the web with modern IDE experience without the need to interact with CLI tools.

Further reading

That's it! We covered everything you need to know in order to integrate Hoop.dev with your service. There are many more features that you can use to improve the experience of your developers. Three such examples are on-the-fly data masking with AIsessions recording, and Git-ops Runbooks. For the most up to date information please refer to the Hoop.dev docs.
If you have any questions, or you just want to chat, please reach out using the chat bubble menu.

Related articles

Integrate AI Data Masking with Apache Kafka (AWS MKS)Integrate AI Data Masking with Apache Kafka (Kubernetes)Integrate AI Data Masking with Apache Kafka (Confluent Cloud)Integrate AI Data Masking with Apache Kafka (Heroku)Integrate AI Data Masking with Apache Kafka (Google Cloud)Integrate AI Data Masking with Apache Kafka (Azure)