Ensuring the protection of sensitive data while enabling seamless Apache Kafka access for development and analytics is a critical challenge for organizations. Hoop.dev introduces an innovative data masking solution that leverages OpenID connect for authentication and utilizes advanced machine learning models for real-time data redaction. This guide will take you through the process of integrating Hoop.dev's data masking capabilities with your Apache Kafka to enhance security and ensure compliance with privacy regulations, all without disrupting your service.
How to integrate AI Data Masking with Apache Kafka + Hoop.dev
- Create a Hoop.dev account and get your authentication token
- Create a connection to access with AI Data Masking using the Hoop CLI
- Invite your team to access the connection with AI Data Masking using the Hoop CLI
If you don't need a tutorial and just want to get started, please consider reading the quickstart docs instead.
Apache Kafka support in Hoop.dev
You can use Hoop.dev with
as shown in the examples below, or alternatively use the
if your prefer.Hoop.dev gets out of the way and users experience the interface of any service as they are. The only changes are to make things more secure and easier to use. For instance: users don't have to run 10 steps in 3 systems to get a temporary credentials anymore, instead they use a single command, and security is more robust.
Why Hoop.dev AI Data Masking?
Enhanced Security
Hoop.dev's use of OpenID for authentication eliminates the reliance on password-based access, reducing the risk of credential leaks. The proxy-based approach to access management further ensures the protection of sensitive information from unauthorized exposure.
Real-Time PII Data Masking with AI
With its advanced machine learning algorithms, Hoop.dev provides real-time analysis and redaction of sensitive data within query outputs. This capability allows teams to access necessary data without the risk of exposing PII, thereby simplifying compliance with data privacy regulations.
Simplified Compliance
Implementing Hoop.dev's data masking solution aids organizations in adhering to stringent data protection regulations by automating the redaction of PII, thereby negating the need for extensive manual data handling or complex policy configurations.
Seamless Integration
Designed for easy integration with your existing systems, Hoop.dev requires minimal adjustments to current data access and utilization practices. The provision of read-only replicas ensures that service continuity is preserved.
Integrating AI Data Masking
Hoop.dev offers a CLI and Web interface for interacting with the connections. This guide will show how to use them.You can also refer to
for all the information needed to start using Hoop.dev
Main concepts
Hoop.dev has three important entities you will be interacting with:
- connection: these are the services you'll manage. They have have credentials, attached policies, and a few other properties.
- policy: policies are the rules you set for how users interact with a connection. Each connection can have multiple policies and each session for that connection will have a set of policies applied on the begging and during the session time.
- session: these are the interactions of your users with the connections. They can be behave differently depending on the policies configured for the connection.
Install dependencies
Install the Hoop CLI to easily connect to your service via Hoop.
MacOS: Top this brew formula and install the CLI
shellbrew tap hoophq/hoopcli https://github.com/hoophq/hoopcli brew install hoop
Linux: Run this shell script to install the CLI (check the code here)
shellcurl -s -L https://releases.hoop.dev/release/install-cli.sh | sh
Step 1: Sign In and Authenticate
Begin by using the Hoop.dev Command Line Interface (CLI), which offers seamless integration with OpenID Connect for authentication. This process grants users access and eliminates the need for temporary credentials and significantly improving your security posture.
Authenticate your CLI with your Hoop.dev account. You can signup for an account here.
shellhoop login
Step 2: Get your authentication token
Before you can start the Hoop proxy agent you need to get an authentication token. You can get it running the following command:
shellhoop admin create agent demo
The authentication token will look something like this:
grpcs://demo:xagt-SV9N48TQiD2TozyHFCAE4XsoKa21YF1GKLsvflAWSnV@use.hoop.dev:8443?mode=standardStep 3: Run the Hoop.dev agent
To start the Hoop.dev process on your local computer, export your authentication token as an environment variable named HOOP_DSN and then use the CLI start the long-running process. The agent can run on any platform, from Linux VMs to container platforms like Apache Kafka. Check out the installation guides for details on how to deploy to other platforms.
shellexport HOOP_DSN=grpcs://demo:xagt-SV9N48TQiD2TozyHFCAE4XsoKa21YF1GKLsvflAWSnV@use.hoop.dev:8443?mode=standard hoop start agent
Step 4: Create a connection per profile
Connections are what users connect to. A connection represents one of the profiles, like or , for a given service. Each of your users will have access to one or more connections. The easiest way to create a new connection is when a new service is setup.
You'll need the connection name when using a connection. You can either find it in the web dashboard where all the connections you can access are easily visible, or list them with the CLI.
To create a connection you'll provide the connection name, the agent name (created in the previous step), and the credentials Hoop will use to access the service on the backend. Remember: these credentials are kept in the vault and are never exposed to the user.
Setup the connection with the Kafka CLI operation you want to performshellhoop admin create conn my-kafka-cluster -a demo \ -- bin/kafka-console-consumer.sh --bootstrap-server HOST1:PORT1,HOST2:PORT2 --consumer.config config.properties --topic testTopic --property "print.key=true"
Connect with SSO (sessions)
We will now use this Kubernetes cluster with Hoop.dev CLI. You previously signed in with Google or Github, and this connection will use the same authentication. You can customize the default Google or Github federation with your own IDP with this guide.
Execute the operation with SSOshellhoop connect my-kafka-cluster
Helping your team access with SSO
In addition to securing your own connections to a service, Hoop.dev provides several resources to make it easy to manage your team when accessing these services. On top of that they'll get a great developer experience while using it.
We offer an intuitive and easy to use internal developer platform to easily connect to any service which you can share with internal user directly. Simply invite them in the settings and they'll be able to use it.
We also offer easy to use docs for how to easily connect to any service which you can share with internal user directly.
Finally, the Web editor provides users with a consistent interface on the web with modern IDE experience without the need to interact with CLI tools.
Further reading
That's it! We covered everything you need to know in order to integrate Hoop.dev with your service. There are many more features that you can use to improve the experience of your developers. Three such examples are on-the-fly data masking with AI, sessions recording, and Git-ops Runbooks. For the most up to date information please refer to the Hoop.dev docs.
If you have any questions, or you just want to chat, please reach out using the chat bubble menu.