How to migrate from ansible vault to hashicorp vault

Ansible Vault

Today, all our project passwords are stored in ansible vault files, on file per environnement, stored in a « passwords » directory

They are loaded at the beginning of the main playbook :

---
- hosts: all:!localhost
  any_errors_fatal: true
  gather_facts: true
  pre_tasks:
    - include_vars: password/password_{{ env }}.yml
      connection: local
      no_log: True
      tags:
        - password
        - always

Usually, to add or modify a password, we have to fetch the ansible vault key from a remote server in the environment ; copy it into our workspace, then run the command

ansible-vault edit --vault-id .vault.dev passwords/password_dev.yml

Then we can edit the password, and re-commit / push the new vaulted password file to the git repository.

Hashicorp Vault

We are planning to migrate from ansible vault to hashicorp vault. This will simplify password management / re-generation with the great vault-ui.

Migration

To simplify the migration to Hashicorp Vault I’ve developped a tiny script to :

  • decrypt all ansible vault password files
  • convert each password file to json
  • import each password file into hashicorp vault
  • replace all passwords with the ansible lookup  hashivault into the yaml file.

The requirement to this scripts are :

  • jq : for json parsing
  • curl : for hashicorp vault api authentication
  • j2y : for converting yaml into json
  • vault binary : for password import into hashicorp vault
  • sed : for yaml replacement
  • All ansible vault keys are in current directory (named « .vault.<env>« )

here is the script :

#!/bin/sh

export VAULT_ADDR=https://vault.company.fr
export VAULT_ROLE_ID=0acee709-b16c-4abd-20a3-12159d6c8b36
export VAULT_SECRET_ID=fcf81d75-bd31-6e8f-239b-fc09c5dd744e

for i in $(find . -name 'password_*.yml'); do
  file=$(echo $i | cut -d '/' -f 2)
  env=$(echo $file | sed 's/password_//' | cut -d '.' -f 1)
  if [ -f ".vault.$env" ]; then
      cp $i $i.decrypt
      ansible-vault decrypt --vault-id .vault.$nna.$env $i.decrypt
    fi
    else
      cp $i $i.decrypt
  fi
  # converting to json $i.decrypt
  j2y -r $i.decrypt > password_$env.json
  rm -f $i.decrypt

  # Authenticate to $VAULT_ADDR
  data="{\"role_id\": \"${VAULT_ROLE_ID}\", \"secret_id\": \"${VAULT_SECRET_ID}\" }"
  export VAULT_TOKEN=$(curl -s -k -X POST ${VAULT_ADDR}/v1/auth/approle/login -d "$data" | jq '.auth.client_token')


  vault write secret/ida/$nna/$env @password_$env.json

  # converting yml file with lookup hashi_vault
  cp $i.decrypt $i.new
  sed -i "s!\([^:]*\):.*!\1: \"{{ lookup('hashi_vault', 'secret=secret/app/$env:\1') }}\"!" $i.new
  dos2unix $i.new
done

Now I just have to run the ansible_vault_to_hashicorp.sh script and all my passwords are migrated.

Jenkins

To use the ansible lookup hashivault plugin the environment variables VAULT_ADDR and VAULT_TOKEN must be exported.

All our playbooks are launched from jenkins, So we just have to connect to hashicorp vault from Jenkins before ansible playbook.

To do that we use the Hashicorp Vault Jenkins Plugin that add a new Jenkins Hashicorp Vault credentialtials scoped to our application folder. Then before each build, the vault plugin will export the required variables.

Finally the ansible lookup plugin will fetch the passwords directly from hashicrop vault at runtime 🙂

 

Ansible Mitogen

Today I discovered a new ansible strategy module that increase ansible performance a lot : Ansible Mitogen.

Mitogen is a python library for writing distributed self-replicating programs.

You can read a great article about this here :

https://sweetness.hmmz.org/2018-03-06-quadrupling-ansible-performance-with-mitogen.html

After some benchmark, I confirm : Mitogen is very fast ! I’ve divised my deployment by 2 :

For example, for a small playbook to deploy and configure 3 kafka nodes :

Before mitogen :

PLAY RECAP **********************************************************************************************************************************************************************************************
brok01 : ok=18 changed=0 unreachable=0 failed=0
brok02 : ok=18 changed=0 unreachable=0 failed=0
brok03 : ok=18 changed=0 unreachable=0 failed=0

Monday 28 May 2018 15:05:21 +0200 (0:00:02.680) 0:02:13.012 ************
===============================================================================
kafka : configuration "projet" ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 11.85s
kafka : import ca erdf dans keystore ------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.10s
kafka : configuration kafka ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.51s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.04s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.03s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.68s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.67s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.22s
kafka : kafka directories ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.19s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.15s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.14s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.13s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.08s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.99s
kafka : keystore jks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.52s
java : installation de java ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.00s
kafka : kafka service rhel7 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.85s
kafka_metrics_reporter : copie du jar kafka metrics reporter ------------------------------------------------------------------------------------------------------------------------------------- 0.79s
kafka : kafka user ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.69s
kafka : kafka exists ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.66s

real 2m16,558s
user 0m37,302s
sys 0m4,845s

With Mitogen_linear strategy :

PLAY RECAP **********************************************************************************************************************************************************************************************
brok01 : ok=18 changed=0 unreachable=0 failed=0
brok02 : ok=18 changed=0 unreachable=0 failed=0
brok03 : ok=18 changed=0 unreachable=0 failed=0

Monday 28 May 2018 15:07:01 +0200 (0:00:01.775) 0:01:02.035 ************
===============================================================================
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.01s
kafka : configuration "projet" ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.57s
kafka : import ca erdf dans keystore ------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.90s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.36s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.93s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.78s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.48s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.46s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.45s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.45s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.40s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.38s
kafka : configuration kafka ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.06s
java : installation de java ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.99s
kafka : keystore jks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.70s
include_vars ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.58s
kafka : kafka directories ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.24s
kafka : kafka service rhel7 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.24s
kafka_metrics_reporter : copie du jar kafka metrics reporter ------------------------------------------------------------------------------------------------------------------------------------- 0.21s
Attend que le broker en question soit dans le cluster avant de restart un autre ------------------------------------------------------------------------------------------------------------------ 0.20s

real 1m5,575s
user 0m27,440s
sys 0m2,207s

All tasks are divided by 2.

Installation

Installation is super easy :

git clone https://github.com/dw/mitogen.git

Update ansible.cfg :

strategy_plugins = ~/git/seuf/mitogen/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

That’s it !

You  can now run your playbooks faster !

Tips : I’ve updated my sudoers configuration file to allow the commands

deploy = (ALL) NOPASSWD:/usr/bin/python -c*

Prometheus : Monitor docker services with grafana

Here is a little tutorial to setup Prometheus monitoring for a local docker daemon and host metrics with docker-compose and :

  • prometheus node exporter for system metrics
  • cadvisor for docker metrics

First create a prometheus.yml config file like this :

global:
alerting:
  alertmanagers:
    - static_configs:
      - targets:
#       - alertmanager:9093
rule_files:
 # - "first_rules.yml"
 # - "second_rules.yml"
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.yml'

And a file service discovery config file_sd_configs.yml:

- targets:
  - grafana:3000
  - prom-node-exporter:9100
  - cadvisor:8080

Now it’s time to start all services with a docker-compose file.

I’m still using the great traefik for reverse proxy and ssl let’s encrypt certificate generation.

So here is my docker-compose :

version: '2'
services:
   traefik:
    image: traefik
    hostname: traefik
    container_name: traefik
    command: -c /etc/traefik/traefik.toml
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/data/traefik/traefik.toml:/etc/traefik/traefik.toml"
      - "/data/traefik/acme.json:/etc/traefik/acme.json"
      - "/data/traefik/ssl:/etc/traefik/ssl"
      - "/var/run/docker.sock:/var/run/docker.sock"

  prometheus:
    image: prom/prometheus
    hostname: prometheus
    container_name: prometheus
    volumes:
      - "/data/docker/volumes/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"
       - "/data/docker/volumes/prometheus/file_sd_configs.yml:/etc/prometheus/file_sd_configs.yml"
     labels:
      - "traefik.frontend.rule=Host:prometheus.aperogeek.fr"
 
  prom_node_exporter:
    image: prom/node-exporter
    hostname: prom-node-exporter
    container_name: prom-node-exporter
    labels:
      - "traefik.enable=false"
 
  cadvisor:
    image: google/cadvisor:latest
    hostname: cadvisor
    container_name: cadvisor
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
    labels:
      - "traefik.enable=false"
 
  grafana:
    hostname: grafana
    container_name: grafana
    image: grafana/grafana
    environment:
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel
    volumes:
      - /data/grafana:/var/lib/grafana
    labels:
      - "traefik.frontend.rule=Host:grafana.aperogeek.fr"

Then I needed to create a prometheus datasource in grafana

And finally I’ve just imported some public grafana dashboards. 

And voila 🙂

Genymobile Screen Copy

Juste un petit post pour parler de ma découverte du jour : scrcpy !

https://github.com/Genymobile/scrcpy

Un petit outil qui permet de faire du miroir d’écran Android sur son PC via USB (adb).

Pour cela, il suffit simplement de télécharger la dernière version de scrcpy, d’activer le debug USB dans les options développeur de son téléphone et de lancer scrcpy !

C’est magique !

Merci Genymobile, ça peut être bien pratique 🙂

 

Ansible 2.5 grafana modules

 

At work, we needed to automatize grafana installation and grafana provisionning (datasources, plugins and dashboards).

So I’ve created 3 new ansible modules that will be released with the next version of ansible 2.5

The first module is grafana_datasource. If you have to create a lot of different datsources for your grafana instance in multiples organisations, I suggest you to use it.

with a single ansible task, you can create all your datasources. For example if I want to create multiple datasources :

 - name: create elasticsearch datasource
   grafana_datasource:
     name: "{{ item.name }}"
     grafana_url: "{{ grafana_url }}"
     grafana_user: "{{ grafana_user }}"
     grafana_password: "{{ grafana_password }}"
     ds_type: "{{ item.type }}"
     url: "{{ item.url }}"
     database: "{{ item.database }}"
     basic_auth_user: "{{ item.basic_auth_user | default('') }}"
     basic_auth_password: "{{ item.basic_auth_password | default('') }}"
     esVersion: "{{ item.es_version | default(5) }}"
     time_field": "{{ item.time_field | default('@timestamp') }}"
     state: present
  with_items: "{{ grafana_datasources }}"

where the grafana_datasources variable is :

grafana_datasources:
  - name: "es_index1"
    ds_type: "elasticsearch"
    url: "http://elasticsearch.aperogeek.fr:9200"
    database: "index_[YYYY.mm.dd]"
    basic_auth_user: "grafana"
    basic_auth_password: "{{ grafana_es_password }}"
    es_version: 56
  - name: "influxdb"
    ds_type: "influxdb"
    url: "http://elasticsearch.aperogeek.fr:9200"
    database: "telegraf"

 

The second module is grafana_plugin. With this one, you can automate the installation and the upgrade of all your grafana plugins. For example :

 - name: install - update Grafana piechart panel plugin
   grafana_plugin:
     name: grafana-piechart-panel
     version: latest

And the last one is grafana_dashboard. This one is very great because it allow you to import or backup all your existing dashboards.

  - name: import grafana dashboard foo
   grafana_dashboard:
     grafana_url: http://grafana.company.com
     grafana_api_key: XXXXXXXXXXXX
     state: present
     message: "updated by ansible"
     overwrite: true
     path: /path/to/dashboards/foo.json

 - name: export dashboard
   grafana_dashboard:
     grafana_url: http://grafana.company.com
     grafana_api_key: XXXXXXXXXXXX
     state: export
     slug: foo
     path: /path/to/dashboards/foo.json

Hope thoses new ansible modules will be usefull for someone 🙂

If you have some suggestion of missing feature in this modules, you can comment in this article or make a pull requests in ansible github repo.

HAPROXY : client certificate validation

Today at the office, the security team ask me to secure our reverse proxy by adding a client certificate validation to only trust the client host CN.

So here is my method to verify the client certificate CN according to the expected one :

frontend frontend_foo
  mode tcp
  bind *:443 ssl crt /etc/ssl/certs/haproxy_reverse.proxy.company.com.pem ca-file /etc/ssl/certs/autorite_chain_haproxy.pem crl-file /etc/ssl/certs/crl-bundle_haproxy.pem verify required ca-ignore-err all crt-ignore-err all
 default_backend backend_foo

backend backend_foo
  mode tcp
  option httpchk

  acl cert_from_trusted_client ssl_c_s_dn(CN) -m reg ^trusted\.client\.(site1|site2)\.company\.(com|fr)$
  tcp-response inspect-delay 2s
  tcp-response content reject unless cert_from_trusted_client

  server srv_load01 backend.company.com:443 check ssl crt /etc/ssl/certs/haproxy_reverse.proxy.company.com.pem ca-file /etc/ssl/certs/autorite_chain_haproxy.pem verify required

With this configuration, only hosts with a certificate with a CN like  « trusted.client.site1.company.fr » , « trusted.client.site2.company.fr », « trusted.client.site1.company.com », « trusted.client.site2.company.com » can connect to the revperse proxy.

Hope this will help someone 😛

Marcus Bière

Aperogeek c’est de l’actu Geek, mais aussi de l’actu Bières !

Du coup je vais vous présenter aujourd’hui une brasserie que j’aprécie particulièrement : Marcus Bière, la bière de la drôme !

Situé dans le petit village de Saou au coeur de la drôme, Marcus Bière est une brasserie artisanale qui fait de la super bonne bière !


Voici quelques petites photos histoire de vous donner envie d’y aller 🙂

Comme vous le voyez le coin est très agréable. En plus c’est juste a côté de chez beau papa, donc j’y vais régulièrement 😛

Meetup Grafana Lyon


J’ai créé le groupe Meetup Grafana Lyon, et je compte organiser un premier meetup Grafana prochainement !

Le rendez vous est fixé au mardi 12 Septembre dans un pub que je connais bien : L’antidote ^_^’

 

Au niveau des présentations, il y aura :

  • Automatisation de la gestion des datasources grafana avec un module ansible (moi)
  • Monitoring Docker avec Telegraf + InfluxDB + Grafana (moi)
  • monitorer un cluster Cassandra avec graphite_exporter / node_exporter, prometheus et grafana (Christophe Schmitz)

N’hésitez pas à vous inscrire, il reste de la place !

 

EDIT : le Meetup s’est bien passé. voici les liens vers les slides que j’ai présenté :

Kapacitor : Alerting for your timeseries

I already talk about monitoring docker with Telegraf, InfluxDB and Grafana. It’s nice, we have pretty dashboards, but it doesn’t do alerting ! Unless you sit in front of your screen all the day, you will not be warned when a container is crashing or when a friend connect on your Teamspeak channel !

Fortunaletly, in the TICK Stack of Influxdata, there is the « K » of Kapacitor.

Kapacitor is an Open source framework for processing, monitoring, and alerting on time series data.

To do that, Kapacitor use TickScripts, small scripts written in a custom DSL Langage, very simple to understand and deploy.

For exmple, if you want to send a warning level alert on your Slack Channel, when the CPU usage of one of your servers is greater than 70%, and a critical level alert when above 85% :

stream
    |from()
        .measurement('cpu_usage_idle')
        .groupBy('host')
    |window()
        .period(1m)
        .every(1m)
    |mean('value')
    |eval(lambda: 100.0 - "mean")
        .as('used')
    |alert()
        .message('{{ .Level}}: {{ .Name }}/{{ index .Tags "host" }} has high cpu usage: {{ index .Fields "used" }}')
        .warn(lambda: "used" > 70.0)
        .crit(lambda: "used" > 85.0)

        // Slack
        .slack()
        .channel('#alerts')

With this kind of DSL langage, we can create any rule we want. By requesting InfluxDB with InfluxQL queries then aggregating metrics by host or any tag, adding a filter based on any criteria (between 8am. and 7pm. from monday to friday for example), etc..

When the alert rule is ready, Kapacitor can use any alerting system, like :

  • sending an email,
  • post to slack or mattermost,
  • write in a log file,
  • send a pager duty message,
  • upscale or downscale a docker swarm/kubernetes stack
  • or simply execute a custom bash script.

Here is an example of alert generated by Kapacitor in Slack :

kapacitor alerting slack

And for thoses who doesn’t want to get their hands dirty, there is Chronograf (the « C » in TICK Stack).

chronograf dashboard

Chronograf is an open-source web application written in Go and React.js designed to visualize your monitoring data from influxDB.

We are far from a Grafana in term of features (and community), but it’s getting better every day. It allow you to explore your data very efficiently :

chronograf data explorer

My favorite feature is the web based interface to easily create alerting and automation rules for Kapacitor.chronograf kapacitor ruleOf course the web interface limits you in term of Kapacitor DSL langage (an expert mode is on the way), but you can easily, in 3 clics, create simple rules like a threshold, detect a delta during a time period or even send a alert when there is no data (deadman) !

So, theses tools are pretty youngs, but are very interestings : I’ll keep a watch on it !