Hashicorp Vault PKI + Cert-manager

Today, Kubernetes is the most popular container orchestration tool. It allow us to deploy all our applications without worry about networking, rolling update process, health checks, etc..

By default communications between application in the cluster are not encrypted, so we need to generate TLS certificate for each applications, and we need to automatize it !

That’s why here at Malt, we are using Hashicorp Vault and cert-manager to simplify the certificate generation process of each application.

Hashicorp Vault

Vault Installation

First we need to deploy our Hashicorp Vault cluster and configure our internal PKI.

To do this, we will use the official vault helm chart to install Vault in our kubernetes cluster.

kubectl create ns vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install -n vault vault hashicorp/vault \
  --set injector.enabled=false \
  --set server.ha.enabled=true \
  --set server.ha.raft.enabled=true

Once Vault is installed, it need to be initialized and unsealed. Here is a little script to init and unseal your vault :

# vault init
vault_init=$(kubectl -n vault exec -it vault-0 -- vault operator init -format=json)
export VAULT_TOKEN=$(echo "$vault_init" | jq -r '.root_token')

# generate unseal script
key_1=$(echo "$vault_init" | jq '.unseal_keys_b64[0]')
key_2=$(echo "$vault_init" | jq '.unseal_keys_b64[1]')
key_3=$(echo "$vault_init" | jq '.unseal_keys_b64[2]')
echo "VAULT_ADDR=http://localhost:8200" > /tmp/vault_unseal.sh
echo "VAULT_TOKEN=$VAULT_TOKEN"     >> /tmp/vault_unseal.sh
echo "vault operator unseal $key_1" >> /tmp/vault_unseal.sh
echo "vault operator unseal $key_2" >> /tmp/vault_unseal.sh
echo "vault operator unseal $key_3" >> /tmp/vault_unseal.sh

# copy unseal script in each node
kubectl -n vault cp /tmp/vault_unseal.sh vault-0:/vault/file/vault_unseal.sh
kubectl -n vault cp /tmp/vault_unseal.sh vault-1:/vault/file/vault_unseal.sh
kubectl -n vault cp /tmp/vault_unseal.sh vault-2:/vault/file/vault_unseal.sh

# unseal the first node
kubectl -n vault exec -it vault-0 -- ash /vault/file/vault_unseal.sh
sleep 10

# join cluster
kubectl -n vault exec vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl -n vault exec vault-2 -- vault operator raft join http://vault-0.vault-internal:8200

# unseal other nodes
kubectl -n vault exec -it vault-1 -- ash /vault/file/vault_unseal.sh
kubectl -n vault exec -it vault-2 -- ash /vault/file/vault_unseal.sh

Note that script generate and copy an unseal script directly inside vault containers. You should consider using vault auto unseal capabilities instead.

Vault Configuration

We need to configure Hashicorp Vault to enable the Vault PKI. And what is the best way to configure a Hashicorp product ? An other Hashicorp product of course ! Let’s use Hashicorp Terraform to configure Hashicorp Vault.

First, we need to configure terraform vault provider

terraform {
  required_providers {
    vault = {
      source = "hashicorp/vault"
      version = "3.1.1"
    }
  }
}

provider "vault" {}

We have to configure vault policies. Here is my policies/certs.hcl file :

path "auth/kubernetes/login" {
  capabilities = ["read"]
}
path "pki/issue/*" {
  capabilities = ["create", "read", "update"]
}
path "pki/certs" {
  capabilities = ["list"]
}
path "pki/cert/*" {
  capabilities = ["read"]
}
path "pki/sign/*" {
  capabilities = ["create", "read", "update"]
}
path "pki/ca/*" {
  capabilities = ["read"]
}
path "pki/ca_chain" {
  capabilities = ["read"]
}
path "pki/crl/*" {
  capabilities = ["read", "list"]
}

And add it to vault :

resource "vault_policy" "certs" {
  name   = "certs"
  policy = file(format("%s/policies/certs.hcl", path.module))
}

Then, we need to enable the Kubernetes authentication method on vault.

resource "vault_auth_backend" "kubernetes" {
  type                  = "kubernetes"
  path                  = "kubernetes"
  description           = "Kubernetes authentication backend mount"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend            = vault_auth_backend.kubernetes.path
  kubernetes_host    = "https://kubernetes.default.svc"
  kubernetes_ca_cert = base64decode("<your k8s certificate>")
}

resource "vault_kubernetes_auth_backend_role" "kubernetes-certs" {
  role_name                        = "certs"
  backend                          = vault_auth_backend.kubernetes.path
  bound_service_account_names      = ["cert-manager"]
  bound_service_account_namespaces = ["cert-manager"]
  token_policies = [
    "certs",
  ]
}

With this role and policy, the kubernetes service account « cert-manager » in the « cert-manager » namespace will be able to login to vault directly with his service account token. Then once logged, it will be able to request certificate signatures from vault.

All have to do now is configure the PKI in vault :

# vault root pki mount point
resource "vault_mount" "pkirootca" {
  path                  = "pki-rootca"
  type                  = "pki"
  description           = "Self signed Vault root CA"
  max_lease_ttl_seconds = 20 * 365 * 24 * 3600
}

# vault root pki certificate
resource "vault_pki_secret_backend_root_cert" "pkirootca" {
  backend            = vault_mount.pkirootca.path
  type               = "internal"
  common_name        = "vault-active.vault.svc"
  ttl                = "630720000"
  format             = "pem"
  private_key_format = "der"
  key_type           = "rsa"
  key_bits           = 4096
}

# vault root pki urls
resource "vault_pki_secret_backend_config_urls" "pkirootca" {
  backend                 = vault_mount.pkirootca.path
  issuing_certificates    = ["http://vault-active.vault.svc:8200/v1/${vault_mount.pkirootca.path}/ca"]
  crl_distribution_points = ["http://vault-active.vault.svc:8200/v1/${vault_mount.pkirootca.path}/crl"]
}


# vault intermediate pki mount point /pki
resource "vault_mount" "pki" {
  path                  = "pki"
  type                  = "pki"
  description           = "Self signed Vault intermediate CA"
  max_lease_ttl_seconds = 3650 * 24 * 3600
}

# certificate request
resource "vault_pki_secret_backend_intermediate_cert_request" "pki" {
  backend            = vault_mount.pki.path
  type               = "exported"
  common_name        = "vault-active.vault.svc"
  alt_names          = ["vault.vault.svc", "vault-standby.vault.svc"]
  format             = "pem"
  private_key_format = "der"
  key_type           = "rsa"
  key_bits           = 2048
}

# sign certificate request by root ca
resource "vault_pki_secret_backend_root_sign_intermediate" "pkirootca-signs-pki" {
  backend        = vault_mount.pkirootca.path
  csr            = vault_pki_secret_backend_intermediate_cert_request.pki.csr
  use_csr_values = true
  common_name    = vault_pki_secret_backend_intermediate_cert_request.pki.common_name
  ttl            = vault_mount.pki.max_lease_ttl_seconds
}

# set signed certificate to vault intermediate pki
resource "vault_pki_secret_backend_intermediate_set_signed" "pki" {
  backend     = vault_mount.pki.path
  certificate = "${vault_pki_secret_backend_root_sign_intermediate.pkirootca-signs-pki.certificate}\n${vault_pki_secret_backend_root_sign_intermediate.pkirootca-signs-pki.issuing_ca}"
}

# vault pki urls
resource "vault_pki_secret_backend_config_urls" "pki" {
  backend                 = vault_mount.pki.path
  issuing_certificates    = ["http://vault-active.vault.svc:8200/v1/${vault_mount.pki.path}/ca"]
  crl_distribution_points = ["http://vault-active.vault.svc:8200/v1/${vault_mount.pki.path}/crl"]
}

# pki roles
resource "vault_pki_secret_backend_role" "pki-application" {
  backend            = vault_mount.pki.path
  name               = "application"
  ttl                = 35.5 * 24 * 3600
  max_ttl            = 36 * 24 * 3600
  generate_lease     = false
  allow_bare_domains = true
  allow_glob_domains = true
  allow_ip_sans      = true
  allow_localhost    = true
  allow_subdomains   = false
  allowed_domains = [
    "*.default.svc",
    "*.default.svc.cluster.local",
  ]
  key_bits  = 2048
  key_type  = "rsa"
  key_usage = ["DigitalSignature", "KeyAgreement", "KeyEncipherment"]
}

This block is more consistent but it does a lot of things.

• First it creates a root pki in vault with a self signed certificate.
• Then it create an intermediate pki (the one that will be used by cert-manager) with a certificate signed by the root pki.
• Finally it create a pki role « application » that allow certificates with domains « *.default.svc » to be signed by vault.

To apply this, you must export the vault required environment variables and create a tunnel to the vault port :

kubectl -n vault port-forward svc/vault-active 8200:8200 &

Then apply it

export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=<the root token generated during initialization>
terraform init
terraform apply

Cert-Manager

The cert-manager installation process is much more simpler :

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

That’s it ! cert-manager is installed, and can now be configured thanks to CRDs.

We can now configure a Vault Issuer for cert manager :

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    path: pki/sign/application
    server: http://vault-active.vault.svc:8200
    auth:
      kubernetes:
        role: my-app-1
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: default-token-xkcd
          key: token

Note that you can also define Cert-manager « ClusterIssuer » instead of Namespaced Issuer. That kind will allow you to declare one issuer that will be available to all namespaces. The Vault pki role must be configured with the correct allowed domains to do this.

Also cert manager is not able yet to determine the secret name associated to a kubernetes service account. So you have to find the secret name of the service account you want to use for vault authentication.

We are now all good ! Let’s create a Certificate for an application in the default namespace :

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foo-certificate
  namespace: default
spec:
  secretName: foo-certificate
  secretTemplate:
    annotations:
      cert-manager-secret-transform: tls.der
  duration: 2160h0m0s # 90d
  renewBefore: 360h0m0s # 15d
  subject:
    organizations:
      - company
  commonName: foo.default.svc
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
    - foo.default.svc
    - foo.default.svc.cluster.local
  uris:
    - https://foo.default.svc
    - https://foo.default.svc.cluster.local
  ipAddresses:
    - 127.0.0.1
  # Issuer references are always required.
  issuerRef:
    name: vault-issuer
    kind: Issuer
    group: cert-manager.io

We should be able to see a new Kubernetes secret in the default namespace « foo-certificate ».

This secret can be mounted as a volume inside the application container. It contain a tls.crt and a tls.key with our certificate and private key.