Databricks

With Opal’s integration with Databricks:

• Users can request time-bound access to your Databricks groups.
• Where Unity Catalog is enabled, users can request access to Unity Catalog items (catalogs, schemas, volumes, functions, registered models, and AI Gateway services) and choose the privilege they need. You must configure an access policy for these resources. See Connect Unity Catalog resources.
• Auditors can initiate access reviews that assign managers or group admins to periodically review users with long-lived access to Databricks resources.
• Admins can add resources from other Opal integrations to an Databricks group so a Databricks group’s members can automatically gain birthright access to, for example, a GitHub repo, AWS IAM role, etc.
• All access changes are tracked in as events that you can log to Slack or export to your favorite tools.

Supported resources

*You can add Databricks users and service principals as members of groups. Account users sync as users, and they get access through group membership or, for Unity Catalog items, direct grants. †Opal grants and revokes privileges on Databricks UC Catalogs, Schemas, Volumes, Functions, Registered Models, and AI Gateway services. You can grant them directly to users, service principals, and groups, and group members inherit access granted to a group. Access can be time-bound: Opal revokes the grant when it expires. See Connect Unity Catalog resources. The Databricks integration does not support managing identities at the workspace level, based on Databricks’ guidelines for identity federation.

Requirements

To set up the Databricks integration, you must:

• Be an Opal Admin
• Have permission to create a service principal in Databricks
• To also sync Unity Catalog resources, have permission to assign the Metastore Admin role in the Databricks account console (see Connect Unity Catalog resources)

1. Configure fields in Databricks

First, create a service principal in Databricks and create an OAuth secret for it. If you’ll authenticate with OIDC federation instead, skip the OAuth secret (steps 3 and 4) and save only the service principal’s Client ID:

1. Add a Databricks service principal.
2. Assign the service principal the Account admin role.
3. Create an OAuth secret for this service principal. Select Generate secret and specify any lifetime. You have to rotate this when it expires, so you might want to choose a long expiration. By default, secrets refresh every 2 years.
4. Save the Secret and Client ID, which you’ll use in the next step.

You’ll also need the following from Databricks:

• Account Login URL. Use the base URL you use to log in to Databricks. For example, https://accounts.cloud.databricks.com. For Azure Databricks, use https://accounts.azuredatabricks.net.
• Account ID. Retrieve this from your avatar in the top left corner of Databricks.

2. Configure fields in Opal

Go to Inventory > + App and find the Databricks integration. Give the integration a name, admin, description, and specify its visibility. Enter the Account Login URL, Account ID, Client ID, and Client secret fields from the previous step. To authenticate without a client secret, turn on Authenticate with OIDC federation instead of a client secret and follow Authenticate with OIDC federation. To sync Unity Catalog resources, turn on the Enable Unity Catalog switch. You can also turn it on later when you edit the connection. When it’s on, you can also:

• If your Account Login URL is https://accounts.azuredatabricks.net, enter Workspace URLs. See Connect Unity Catalog resources.
• Under Unity Catalog types to import, select which of Volumes, Functions, Models, and Services to import. Opal always imports catalogs and schemas. If you clear a type, Opal removes its resources on the next sync, and you can’t clear a type while Opal still manages access to resources of that type.

Select Save.

3. Import resources to Opal

In the Inventory in the Databricks app, go to the Assets tab and select Import assets to add your Databricks resources to Opal. Databricks connections use the Manual import setting by default. With Manual import, Unity Catalog items appear on the Import page only after you select Sync Items on that page. The page then lists all items in the app, and you select the ones to import. You can now manage access to Databricks resources in Opal.

Authenticate with OIDC federation

With OIDC federation, Opal doesn’t store a client secret for your service principal. Instead, Opal signs a short-lived token with a key that’s unique to the connection, and Databricks exchanges it for an access token using a federation policy on the service principal. There’s no secret to rotate when it expires.

Set up a new connection with OIDC federation

1. In Opal, follow Configure fields in Opal: enter the Account Login URL, Account ID, and Client ID, then turn on Authenticate with OIDC federation instead of a client secret. Opal shows an Issuer, Subject, Audience, and JWKS. Leave this page open.
2. In the Databricks account console, go to User management > Service principals, select the service principal, and open the Credentials & secrets tab. Under Federation policies, select Create policy.
3. Configure the policy with the values from Opal:
   • Issuer URL: the Issuer.
   • Audiences: the Audience, which is your Databricks account ID.
   • Subject: the Subject.
   • Subject claim: leave the default, sub.
   • For token signature validation, paste the JWKS as the JWKS JSON.
4. Select Create policy in Databricks, then select Save in Opal.

Switch an existing connection to OIDC federation

1. In the Databricks app in Opal, go to Setup > Edit and turn on Authenticate with a new OIDC federation key instead of a client secret.
2. Add a federation policy to the service principal with the values Opal shows, as described in Set up a new connection with OIDC federation.
3. Select Save in Opal.

Rotate the federation key

You can replace a connection’s federation key without interrupting syncs:

1. In the Databricks app in Opal, go to Setup > Edit and turn on Authenticate with a new OIDC federation key instead of a client secret. Opal generates a new key and shows two JWKS values: JWKS, which accepts both the current and new keys, and JWKS after saving, which accepts only the new key.
2. In Databricks, update the service principal’s federation policy to use the JWKS value, then select Save in Opal. Opal starts signing with the new key.
3. Update the federation policy again to use the JWKS after saving value. This retires the old key.

The issuer and subject stay the same when you rotate the key, so you only update the JWKS. Databricks accepts up to 5 keys in a JWKS JSON. Each rotation is recorded as an event on the connection, identified by the key’s ID.

Connect Unity Catalog resources

In addition to account users, groups, and service principals, Opal can import Unity Catalog securables — Catalogs, Schemas, Volumes, Functions, Registered Models, and AI Gateway services — so you can review who has access to them and manage that access for groups, users, and service principals. Opal doesn’t support tables and views. Unity Catalog data belongs to a metastore rather than any single workspace, so Opal discovers it by connecting through one workspace assigned to each Unity Catalog metastore in your account. To let Opal see your Unity Catalog resources, in addition to the Account admin role already granted to your service principal in Configure fields in Databricks:

1. Add the service principal to the workspace or workspaces assigned to each Unity Catalog metastore. Opal picks one running workspace per metastore, so add the service principal to each workspace assigned to the metastore. See Databricks’ guide to managing service principals.
2. In the Databricks account console, assign the service principal the Metastore Admin role on each metastore you want Opal to sync. Alternatively, grant the service principal USE_CATALOG on only the specific catalogs you want visible to Opal.

Azure workspace URLs

On AWS and GCP, Opal discovers workspaces itself. On Azure, when your Account Login URL is https://accounts.azuredatabricks.net and Enable Unity Catalog is on, enter Workspace URLs:

• Add one workspace URL per metastore, for example https://adb-1234567890123456.7.azuredatabricks.net.
• Each URL must end in .azuredatabricks.net. Sovereign Azure clouds aren’t supported.
• The service principal must be a member of each workspace.

Troubleshooting

If the service principal isn’t added to a workspace assigned to a metastore, the Unity Catalog sync fails. Connection validation doesn’t catch this. If Opal can reach a metastore but no catalogs come back, double-check that the service principal has the Metastore Admin role (or USE_CATALOG grants) — Opal surfaces this as an import error rather than an empty list, since Databricks’ API can’t otherwise distinguish “no catalogs exist” from “the service principal can’t see any.”