3.1.8 Ensure that data at rest and in transit is encrypted in Azure Databricks using customer managed keys (CMK)

Information

Azure Databricks encrypts data in transit using TLS 1.2+ to secure API, workspace, and cluster communications. By default, data at rest is encrypted using Microsoft-managed keys.

Organizations with stricter needs for control of encryption keys should enable customer-managed keys (CMK) for greater control over data encryption, auditing, and regulatory compliance. Azure Key Vault should be used to store and manage CMKs.

Enforcing encryption at rest and in transit in Azure Databricks:

- Protects sensitive data from unauthorized access.
- Ensures regulatory compliance (ISO 27001, GDPR, HIPAA, SOC 2).
- Allows key revocation and rotation control with customer-managed keys (CMK).
- Mitigates insider threats by preventing unauthorized access to raw storage.

Solution

NOTE: These remediations assume that an Azure KeyVault already exists in the subscription.

Remediate from Azure CLI

- Create a dedicated key:

az keyvault key create --vault-name <keyvault-name> --name <key-name> --protection <"software" or "hsm"> <xhtml:ol start="2"> - Assign permissions to Databricks:

az keyvault set-policy --name <keyvault-name> --resource-group <resource-group-name> --spn <databricks-spn> --key-permissions get wrapKey unwrapKey <xhtml:ol start="3"> - Enable encryption with CMK:

az databricks workspace update --name <databricks-workspace-name> --resource-group <resource-group-name> --key-source "Microsoft.KeyVault" --key-name <key-name> --keyvault-uri <keyvault-uri>

Remediate from PowerShell

$Key = Add-AzKeyVaultKey -VaultName <keyvault-name> -Name <key-name> -Destination <"software" or "hsm">
Set-AzDatabricksWorkspace -ResourceGroupName "<resource-group-name>" -WorkspaceName "<databricks-workspace-name>" -EncryptionKeySource "Microsoft.KeyVault" -KeyVaultUri $Key.Id

Impact:

Enabling CMK encryption requires additional configuration. Key management introduces maintenance overhead (rotation, revocation, lifecycle management). Potential access issues will be encountered if keys are deleted or rotated incorrectly.

See Also

https://workbench.cisecurity.org/benchmarks/19304

Item Details

Category: ACCESS CONTROL, IDENTIFICATION AND AUTHENTICATION, SYSTEM AND COMMUNICATIONS PROTECTION

References: 800-53|AC-17(2), 800-53|IA-5, 800-53|IA-5(1), 800-53|SC-8, 800-53|SC-8(1), 800-53|SC-28, 800-53|SC-28(1), CSCv7|14.4, CSCv7|14.8

Plugin: microsoft_azure

Control ID: fb674ffeb6ce4f302faeafda72a6b74498b4b4f68d4c6b94f22bb871aa46f327