How to migrate from ansible vault to hashicorp vault

Ansible Vault

Today, all our project passwords are stored in ansible vault files, on file per environnement, stored in a « passwords » directory

They are loaded at the beginning of the main playbook :

---
- hosts: all:!localhost
  any_errors_fatal: true
  gather_facts: true
  pre_tasks:
    - include_vars: password/password_{{ env }}.yml
      connection: local
      no_log: True
      tags:
        - password
        - always

Usually, to add or modify a password, we have to fetch the ansible vault key from a remote server in the environment ; copy it into our workspace, then run the command

ansible-vault edit --vault-id .vault.dev passwords/password_dev.yml

Then we can edit the password, and re-commit / push the new vaulted password file to the git repository.

Hashicorp Vault

We are planning to migrate from ansible vault to hashicorp vault. This will simplify password management / re-generation with the great vault-ui.

Migration

To simplify the migration to Hashicorp Vault I’ve developped a tiny script to :

• decrypt all ansible vault password files
• convert each password file to json
• import each password file into hashicorp vault
• replace all passwords with the ansible lookup  hashivault into the yaml file.

The requirement to this scripts are :

• jq : for json parsing
• curl : for hashicorp vault api authentication
• j2y : for converting yaml into json
• vault binary : for password import into hashicorp vault
• sed : for yaml replacement
• All ansible vault keys are in current directory (named « .vault.<env>« )

here is the script :

#!/bin/sh

export VAULT_ADDR=https://vault.company.fr
export VAULT_ROLE_ID=0acee709-b16c-4abd-20a3-12159d6c8b36
export VAULT_SECRET_ID=fcf81d75-bd31-6e8f-239b-fc09c5dd744e

for i in $(find . -name 'password_*.yml'); do
  file=$(echo $i | cut -d '/' -f 2)
  env=$(echo $file | sed 's/password_//' | cut -d '.' -f 1)
  if [ -f ".vault.$env" ]; then
      cp $i $i.decrypt
      ansible-vault decrypt --vault-id .vault.$nna.$env $i.decrypt
    fi
    else
      cp $i $i.decrypt
  fi
  # converting to json $i.decrypt
  j2y -r $i.decrypt > password_$env.json
  rm -f $i.decrypt

  # Authenticate to $VAULT_ADDR
  data="{\"role_id\": \"${VAULT_ROLE_ID}\", \"secret_id\": \"${VAULT_SECRET_ID}\" }"
  export VAULT_TOKEN=$(curl -s -k -X POST ${VAULT_ADDR}/v1/auth/approle/login -d "$data" | jq '.auth.client_token')


  vault write secret/ida/$nna/$env @password_$env.json

  # converting yml file with lookup hashi_vault
  cp $i.decrypt $i.new
  sed -i "s!\([^:]*\):.*!\1: \"{{ lookup('hashi_vault', 'secret=secret/app/$env:\1') }}\"!" $i.new
  dos2unix $i.new
done

Now I just have to run the ansible_vault_to_hashicorp.sh script and all my passwords are migrated.

Jenkins

To use the ansible lookup hashivault plugin the environment variables VAULT_ADDR and VAULT_TOKEN must be exported.

All our playbooks are launched from jenkins, So we just have to connect to hashicorp vault from Jenkins before ansible playbook.

To do that we use the Hashicorp Vault Jenkins Plugin that add a new Jenkins Hashicorp Vault credentialtials scoped to our application folder. Then before each build, the vault plugin will export the required variables.

Finally the ansible lookup plugin will fetch the passwords directly from hashicrop vault at runtime 🙂