Chapter 2. Preparing HashiCorp Vault for the Record Encryption filter
Chapter 2. Preparing HashiCorp Vault for the Record Encryption filter To use Vault with the Record Encryption filter in an OpenShift cluster, use the following setup for your Vault instance: Enable the Transit Engine as the Record Encryption filter relies on its APIs. Create a Vault policy specifically for the filter with permissions for generating and decrypting Data Encryption Keys (DEKs) for envelope encryption. Obtain a Vault token that includes the filter policy. The deployment configuration for the proxy uses the URL for the Vault Transit Engine service. Vault can be deployed as an existing instance, a cloud instance, or on OpenShift. With accessibility to the proxy, it can either be co-located with the Streams for Apache Kafka Proxy or deployed remotely. For information on installing Vault on OpenShift and setting up access, refer to the HashiCorp Vault product documentation. This procedure outlines two options for preparing Vault: Deploying Vault to the OpenShift cluster using Helm with an example ephemeral deployment configuration provided with Streams for Apache Kafka Proxy. Updating your existing Vault instance. When you have prepared a Vault instance, you must then create a Vault policy and token for the Record Encryption filter. Warning The example deployment configuration is not suitable for production environments. Streams for Apache Kafka includes example installation artifacts in the examples/proxy/record-encryption/vault folder, which contains pre-configured Vault deployment files compatible with the proxy and Record Encryption filter. amqstreams_proxy_encryption_filter_policy.hcl defines a Vault policy for the Record Encryption filter helm-dev-values.yaml specifies the Helm deployment configuration for Vault These installation files offer a quick setup for trying out the proxy. Prerequisites Installation requires an OpenShift user with cluster-admin role, such as system:admin . The oc command-line tool is installed and configured to connect to the OpenShift cluster with admin access. The helm command line tool is installed and configured to connect to the OpenShift cluster with admin access. An OpenShift project namespace called proxy , which is the same namespace where the proxy is installed by default. For information on the oc and helm command line options used in this procedure, check the --help . Deploying Vault using the example Helm deployment configuration Download and extract the Streams for Apache Kafka Proxy installation artifacts. The proxy is available from Streams for Apache Kafka software downloads page . The files contain the deployment configuration required for deploying Vault. Create a root token and make a note of it: cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 > vault.root.token export VAULT_TOKEN=USD(cat vault.root.token) Install Vault using Helm: helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault \ --create-namespace --namespace=vault \ --version <helm_version> \ --values vault/helm-dev-values.yaml \ --set server.dev.devRootToken=USD{VAULT_TOKEN} \ --wait The root token is used for the Vault instance. Check the status of the deployment: oc get pods -n vault Output shows the deployment name and readiness NAME READY STATUS RESTARTS vault-0 1/1 Running 0 A pod ID identifies the pod created. With the default deployment, you install a single proxy pod. READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running. Create a Vault address ( VAULT_ADDR ) environment variable to point to the new Vault instance: export VAULT_ADDR=USD(oc get route -n vault vault --template='https://{{.spec.host}}') Login to Vault as an administrator and enable the Vault Transit secrets engine: vault secrets enable transit If the secrets engine is already enabled, ignore the error. Create an environment variable to point to the Vault Transit address: export VAULT_TRANSIT_URL=USD{VAULT_ADDR}/v1/transit The address is used in the proxy deployment configuration. Create a Vault policy and token . Configuring your own Vault instance If you already have a Kafka instance installed, you can update it to use it with Streams for Apache Kafka Proxy. Create a Vault address environment variable ( VAULT_ADDR and VAULT_NAMESPACE , if using Enterprise) to point to the Vault instance: export VAULT_ADDR=https://<vault server>:8200 export VAULT_NAMESPACE=<namespaces> Login to Vault as an administrator and enable the Vault Transit secrets engine: vault secrets enable transit If the secrets engine is already enabled, ignore the error. Create an environment variable to point to the Vault Transit address: export VAULT_TRANSIT_URL=USD{VAULT_ADDR}/v1/USD{VAULT_NAMESPACE}/transit The address is used in the proxy deployment configuration. Update the proxy deployment configuration to refer to your Vault instance: sed -i "s/\(vaultTransitEngineUrl:\).*USD/\1 USD{VAULT_TRANSIT_URL}/" */proxy/proxy-config.yaml Create a Vault policy and token . Creating a Vault policy and token With the Vault instance set up, create a Vault policy and token for the Record Encryption filter. Create a Vault policy: vault policy write amqstreams_proxy_encryption_filter_policy vault/amqstreams_proxy_encryption_filter_policy.hcl Write the policy to Vault using the HashiCorp policy definition file ( .hcl ) provided with Streams for Apache Kafka Proxy. The policy is named amqstreams_proxy_encryption_filter_policy . Create a Vault token: vault token create \ -display-name "amqstreams-proxy encryption filter" \ -policy=amqstreams_proxy_encryption_filter_policy \ -no-default-policy \ -orphan \ -field=token > vault.encryption.token The command creates a token with the specified policy, and with no associated parent token or default policies. Create a secret containing the token: oc create secret generic proxy-encryption-vault-token \ -n proxy \ --from-file=encryption-vault-token.txt=vault.encryption.token \ --dry-run=client \ -o yaml > base/proxy/proxy-encryption-vault-token-secret.yaml The command stores the Vault token in the secret and creates the secret as a YAML file in the proxy namespace. The proxy-encryption-vault-token-secret.yaml secret is applied to the OpenShift cluster when deploying Streams for Apache Kafka Proxy with the Record Encryption filter. Tip Rotate keys periodically to minimize the impact of compromised keys. When using a Key Management System (KMS), such as HashiCorp Vault, rotate the Key Encryption Key (KEK) stored in the KMS. Streams for Apache Kafka Proxy manages DEK rotation automatically. Occasional restarts may be necessary for the proxy to pick up the new key. Additionally, encrypted messages should include key version metadata to indicate key rotation.
cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 > vault.root.token export VAULT_TOKEN=USD(cat vault.root.token) helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault --create-namespace --namespace=vault --version <helm_version> --values vault/helm-dev-values.yaml --set server.dev.devRootToken=USD{VAULT_TOKEN} --wait get pods -n vault NAME READY STATUS RESTARTS vault-0 1/1 Running 0 export VAULT_ADDR=USD(oc get route -n vault vault --template='https://{{.spec.host}}') vault secrets enable transit export VAULT_TRANSIT_URL=USD{VAULT_ADDR}/v1/transit export VAULT_ADDR=https://<vault server>:8200 export VAULT_NAMESPACE=<namespaces> vault secrets enable transit export VAULT_TRANSIT_URL=USD{VAULT_ADDR}/v1/USD{VAULT_NAMESPACE}/transit sed -i "s/\(vaultTransitEngineUrl:\).*USD/\1 USD{VAULT_TRANSIT_URL}/" */proxy/proxy-config.yaml vault policy write amqstreams_proxy_encryption_filter_policy vault/amqstreams_proxy_encryption_filter_policy.hcl vault token create -display-name "amqstreams-proxy encryption filter" -policy=amqstreams_proxy_encryption_filter_policy -no-default-policy -orphan -field=token > vault.encryption.token create secret generic proxy-encryption-vault-token -n proxy --from-file=encryption-vault-token.txt=vault.encryption.token --dry-run=client -o yaml > base/proxy/proxy-encryption-vault-token-secret.yaml