Databricks Instance Profile
You set up a databricks instance profile by linking a cloud role to a compute cluster. This setup lets jobs read and write cloud storage without saving static keys in notebooks. Data teams use this method to avoid hardcoding cloud secrets inside shared workspaces. Still, every user on that cluster shares the exact same cloud role and storage rights. If the role is too broad, any notebook user can read or delete sensitive files. How a databricks instance profile works An instance profile is a
Instance Profile: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You set up a databricks instance profile by linking a cloud role to a compute cluster. This setup lets jobs read and write cloud storage without saving static keys in notebooks.
Data teams use this method to avoid hardcoding cloud secrets inside shared workspaces. Still, every user on that cluster shares the exact same cloud role and storage rights. If the role is too broad, any notebook user can read or delete sensitive files.
How a databricks instance profile works
An instance profile is a container for an identity role in your cloud account. It passes that role directly to the virtual machines that run your compute cluster. When a cluster starts, the cloud provider attaches this profile to every worker node.
First, an admin creates a role in the cloud identity console. This role holds policies that grant read or write rights to specific storage buckets. Next, the admin wraps that role inside an instance profile resource.
The admin must also give the workspace main role permission to pass this profile. After that, the admin registers the profile identifier inside the workspace admin settings. Finally, team members with the right permissions can pick the profile when they build a cluster.
When a job runs on the cluster, it asks the local node metadata for temporary keys. The cloud provider checks the attached profile and returns short-lived keys to the node. The job uses those keys to read files from cloud storage without human input.
What native profiles solve and what they miss
Native profiles solve the major risk of pasting static cloud keys into code. Developers do not need to manage key rotation or hide secret strings in scripts. The cloud provider handles key rotation in the background on every single cluster node.
Still, this setup operates at the machine level rather than the user level. Every person who can attach a notebook to the cluster gets the same storage rights. The cloud storage log only shows that the cluster role read a bucket. It does not show which specific engineer or script ran the query.
Instance Profile: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
This creates a gap when multiple teams share large clusters to save money. If one job needs production data, every other user on that cluster can read it too. Native node roles decide which machine can log in, not what individual queries do in flight.
Also, storage permissions are all-or-nothing at the file or bucket path level. A profile grants full read rights to a file, including any sensitive columns inside it. It cannot rewrite matched fields like emails, phone numbers, card numbers, or national IDs in query results.
Managing cluster roles and least privilege
To reduce risk with native profiles, admins must separate clusters by data sensitivity. You should create one profile per project instead of one big profile for the whole company. Each profile should only point to the exact storage buckets that a specific job needs.
Admins also need to lock down who can attach notebooks to high-privilege clusters. Workspace access control lists let you limit which groups can pick a registered profile. You can also use cluster policies to force specific profiles onto certain job types.
Even with strict policies, managing dozens of cloud roles takes significant time. Each new bucket or team requires changes in both the cloud console and the workspace. Teams often end up granting wider permissions just to keep daily pipelines running on time.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
What is the difference between an IAM role and an instance profile?
An IAM role holds the actual permissions for cloud storage and services. An instance profile is the wrapper that passes that role to a virtual machine.
Why can multiple users share the same cloud permissions on a cluster?
The cloud provider attaches the profile to the underlying virtual machine, not the user. Any code running on that machine inherits the machine role by default.
How do you limit who can assign a profile to a new cluster?
Workspace admins can set role permissions inside the admin console. Only users or groups on that list can select the profile during cluster setup.