Keeping secrets secure on Kubernetes is crucial for every organization. However, it’s easier said than done.
Securing secrets in Kubernetes is crucial but challenging due to its inherent limitations. Kubernetes' native mechanism lacks comprehensive security measures, such as encrypted central storage and robust access controls. To address these shortcomings, third-party solutions like HashiCorp Vault and Azure Key Vault offer enhanced security features.
This article compares Azure Key Vault and HashiCorp Vault, focusing on their operational mechanisms, security features, user-friendliness, and integration with Kubernetes.
Keeping secrets secure on Kubernetes is crucial for every organisation. However, it's easier said than done.
Although Kubernetes has its own mechanism to manage secrets, it's not up to current security challenges due to the way Kubernetes handles secrets:
While these are not the only drawbacks of the native Kubernetes mechanism for handling secrets, they are the most critical, as they defy best security practices. There are four basic aspects that should be implemented to handle secrets securely:
The simplest way to fulfil these requirements is through the use of third-party secret managers such as HashiCorp Vault and Azure Key Vault. Both solutions exceed the minimum security features listed above, but they use very different approaches to do so. In this article, you'll take a look at the pros and cons of each tool so that you can decide which one best suits your needs and requirements.
With any comparison, it’s important to set clear parameters for how each tool will be judged. In this article, each solution will be evaluated by taking into account the following:
The following are the main features of Azure Key Vault (AKV):
Key Vault is a service designed to seamlessly integrate with the rest of the Azure ecosystem. In that sense, it's not surprising that you must create an Azure account to access it. Here's a high-level overview of how to use AKV:
The above approach allows secrets to be stored at rest in Azure Key Vault in a secure, encrypted manner. Moreover, the secrets are only stored in the AKS cluster when the pod with the secrets is running. Once the pod is removed, the secrets are also removed.
It should be noted that the Secrets Store CSI driver provides several methods to manage the identities of those who can access your Azure Key Vault. These methods include Azure Active Directory (Azure AD) pod-managed identities, user-assigned managed identities, and system-assigned managed identities. For more information on how to provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver, you can take a look at the documentation.
Azure Key Vault is a fully managed service, which means that the encryption of secrets is handled by Microsoft Azure. The service offers two tiers of protection:
It's important to note that both tiers offer the security features mentioned previously, such as ACLs, secret handling, automated secret rotation, and audit trails. In short, unless your organisation is required to comply with the FIPS 140-2 level 3 standard, the standard tier is sufficient.
That said, there are no known security issues that would make AKV prone to data leaks, which just leaves the number one cause of security flaws: users. Microsoft recommends using separate key vaults for each application, as well as setting strict access controls on secrets. For more information on using Azure Key Vault effectively, you can look at the best practices section of the documentation.
Setting up Azure Key Vault is straightforward. After creating your Azure account, you’ll be able to access the Azure portal:
From there, type "vault" in the search box and click Key vaults to access the AKV dashboard:
The Azure Key Vault dashboard allows you to perform tasks such as creating, managing, and deleting vaults. Click Create key vault to create your first vault:
On the next screen, you’ll be able to assign a name to the vault, select a region and pricing tier, and assign access policies. You can learn more about creating a key vault using the Azure portal in the quickstart guide.
Once your vault is created, you can call it by name to assign permissions in AKS, create SecretProviderClass, and create pods that expose the secrets. For detailed instructions on how to use Azure Key Vault with AKS, read the documentation. Below are examples of what the code looks like.
Using the Azure CLI, you can assign permissions to new identities using code similar to the following:
# set policy to access keys in your key vault
az keyvault set-policy -n <keyvault-name> --key-permissions get --spn <pod-identity-client-id>
# set policy to access secrets in your key vault
az keyvault set-policy -n <keyvault-name> --secret-permissions get --spn <pod-identity-client-id>
# set policy to access certs in your key vault
az keyvault set-policy -n <keyvault-name> --certificate-permissions get --spn <pod-identity-client-id>
The following is an example of a SecretProviderClass that uses the name of the previously created vault:
# This is a SecretProviderClass example using aad-pod-identity to access the key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname-podid
spec:
provider: azure
parameters:
usePodIdentity: "true" # Set to true for using aad-pod-identity to access your key vault
keyvaultName: <key-vault-name> # Set to the name of your key vault
cloudName: "" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
objects: |
array:
- |
objectName: secret1
objectType: secret # object types: secret, key, or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty
- |
objectName: key1
objectType: key
objectVersion: ""
tenantId: <tenant-Id> # The tenant ID of the key vault
Below is an example of a pod that mounts the secrets to AKS:
# This is a sample pod definition for using SecretProviderClass and aad-pod-identity to access the key vault
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline-podid
labels:
aadpodidbinding: <name> # Set the label value to the name of your pod identity
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kvname-podid"
As you can see, the process can be handled programmatically and doesn't hinder the workflow of the administrators at all.
In the previous section, you saw how Azure Key Vault interacts with AKS. As for AKV integration with Kubernetes, although Key Vault is not a Kubernetes-specific solution, it's designed to offer a high degree of integration with the Azure ecosystem. This allows AKV to work hand in hand with Active Directory and AKS, Microsoft's Kubernetes distribution.
The above raises the question of whether AKV can be used outside of AKS. The short answer is yes, but while Key Vault offers built-in integration for Azure services, including AKS, it requires a bit more work to use with other Kubernetes distributions. To use it outside of the Azure environment, you'll probably have to use tools like akv2k8s, which doesn't use the Azure Key Vault Provider for
Secrets Store CSI Driver to inject secrets into Kubernetes. For more about how to install Azure Key Vault to Kubernetes outside of AKS using akv2k8s, you can take a look at the documentation.
The following are the main features of HashiCorp Vault:
Unlike Azure Key Vault, HashiCorp Vault uses an open, flexible model that gives you more control over secrets. To achieve this, Vault uses a highly pluggable architecture that allows you to choose which backends to use for authentication, auditing, and secret storage. For example, you could authenticate using LDAP, GitHub, or Kubernetes, along with using Splunk to handle audit logs, and MySQL, Consul, or Google Spanner to store encrypted secrets at rest. This means that one major advantage of HashiCorp Vault is that it minimises vendor lock.
HashiCorp Vault's flexibility can be interpreted as an advantage or a disadvantage depending on your goals, requirements, and use case. While it offers more control, there's more administrative overhead than AKV. The following is a brief review of how Vault works.
You must set up Vault before you can create secrets. At this point, the differences with AKV begin. In addition to binary packages aimed at local development, Vault can be deployed in multiple ways as listed below. You can:
As you can see, you can even use HashiCorp Vault on AKS if you wish. Moreover, you can go further, injecting secrets into Kubernetes pods via Vault Agent sidecar containers, or mounting Vault secrets through Container Storage Interface (CSI) volumes.
Once Vault is configured, the app, service, or user can be authenticated using almost any agent. Vault can also use a wide variety of secrets engines, and supports different methods for handling audit logs.
HashiCorp Vault uses external components for authentication, audit logs, and secret storage. This is a big difference from AKV, which takes care of everything except authentication.
By design, Vault is set up to handle the entire lifecycle of secrets, assign access control, and keep an audit trail. Moreover, Vault allows you to use dynamic secrets that are served through its API to ensure that encryption is properly implemented. This means that Vault can be used for both injecting secrets into Kubernetes and managing service and user authentication certificates. In either scenario, the dynamic nature of secret generation allows for best security practices, such as short-lived certificates.
Despite the fact that the secrets are encrypted at rest and that Vault is considered a secure platform, there are other risks that need to be considered. These risks have to do with the components used in Vault. It's been made clear that unlike AKV, Vault is not a standalone service. No matter how secure Vault is, a bad implementation of the secrets engine or authentication agent can compromise your data. If you prefer the convenience and safety of a managed service, Vault on the HashiCorp Cloud Platform is the way to go.
One aspect that HashiCorp Vault shares with AKV is its ease of use. You can interact with Vault from either the command line or the UI. To illustrate this, you can deploy the Vault dev server for macOS using the following commands:
brew tap hashicorp/tap
brew install hashicorp/tap/vault
Once installed, you can start the server using the following command:
vault server -dev
The output should be similar to the following:
Now you just need to copy the token and point your browser to http://127.0.0.1:8200 to connect to the Vault UI:
Use the token to enter the Vault dashboard:
One of the first steps to using Vault is to set up a secrets engine, which you can do from the dashboard:
In the image above, you can see most of the secrets engines supported by Vault. For more information on how to get started with Vault, take a look at the documentation.
All in all, Vault's web UI is easy to use, and doesn't introduce any major complications to the developer's workflow. That said, you should remember that it can be time consuming to configure the rest of the components that Vault needs.
HashiCorp Vault isn't specifically targeted at Kubernetes. However, as you saw earlier, the level of integration with different Kubernetes distributions, including AKS, is exceptional. You can create a Vault cluster using Helm and then inject secrets using sidecars or the Kubernetes Secrets Store CSI driver. For comparison, here is an example definition of the SecretProviderClass:
cat > spc-vault-database.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-database
spec:
provider: vault
parameters:
vaultAddress: "http://vault.default:8200"
roleName: "database"
objects: |
- objectName: "db-password"
secretPath: "secret/data/db-pass"
secretKey: "password"
As you can see, the most notable difference from AKV is that secrets are not accessed by name, but rather through API calls to Vault. But is this approach better? And more importantly, does HashiCorp Vault offer better integration with Kubernetes than AKV?
The answer to both questions is that it depends.
HashiCorp Vault is highly adaptable, allowing it to seamlessly integrate with any Kubernetes deployment. However, depending on your use case, you may not mind tying yourself to the Azure ecosystem and enjoying its integration with AKV.
In this article, you compared two best-in-class solutions that address the problem of handling secrets in Kubernetes. Both Azure Key Vault and HashiCorp Vault overcome all the drawbacks of the default Kubernetes solution for handling secrets. Both tools are effective at encrypting secrets at rest, offering access control, secrets management, audit trails, and more. Here's a side-by-side comparison:
Ultimately, the question of which solution is better comes down to your vision and needs. If you prefer an out-of-the-box solution based on Azure, then Key Vault is a great fit for your needs. If you'd rather not be tied to a specific vendor, HashiCorp's self-managed Vault is the perfect solution for you.
