Ansible Mitogen

Today I discovered a new ansible strategy module that increase ansible performance a lot : Ansible Mitogen.

Mitogen is a python library for writing distributed self-replicating programs.

You can read a great article about this here :

https://sweetness.hmmz.org/2018-03-06-quadrupling-ansible-performance-with-mitogen.html

After some benchmark, I confirm : Mitogen is very fast ! I’ve divised my deployment by 2 :

For example, for a small playbook to deploy and configure 3 kafka nodes :

Before mitogen :

PLAY RECAP **********************************************************************************************************************************************************************************************
brok01 : ok=18 changed=0 unreachable=0 failed=0
brok02 : ok=18 changed=0 unreachable=0 failed=0
brok03 : ok=18 changed=0 unreachable=0 failed=0

Monday 28 May 2018 15:05:21 +0200 (0:00:02.680) 0:02:13.012 ************
===============================================================================
kafka : configuration "projet" ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 11.85s
kafka : import ca erdf dans keystore ------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.10s
kafka : configuration kafka ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.51s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.04s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.03s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.68s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.67s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.22s
kafka : kafka directories ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.19s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.15s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.14s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.13s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.08s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.99s
kafka : keystore jks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.52s
java : installation de java ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.00s
kafka : kafka service rhel7 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.85s
kafka_metrics_reporter : copie du jar kafka metrics reporter ------------------------------------------------------------------------------------------------------------------------------------- 0.79s
kafka : kafka user ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.69s
kafka : kafka exists ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.66s

real 2m16,558s
user 0m37,302s
sys 0m4,845s

With Mitogen_linear strategy :

PLAY RECAP **********************************************************************************************************************************************************************************************
brok01 : ok=18 changed=0 unreachable=0 failed=0
brok02 : ok=18 changed=0 unreachable=0 failed=0
brok03 : ok=18 changed=0 unreachable=0 failed=0

Monday 28 May 2018 15:07:01 +0200 (0:00:01.775) 0:01:02.035 ************
===============================================================================
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.01s
kafka : configuration "projet" ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.57s
kafka : import ca erdf dans keystore ------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.90s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.36s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.93s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.78s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.48s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.46s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.45s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.45s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.40s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.38s
kafka : configuration kafka ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.06s
java : installation de java ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.99s
kafka : keystore jks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.70s
include_vars ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.58s
kafka : kafka directories ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.24s
kafka : kafka service rhel7 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.24s
kafka_metrics_reporter : copie du jar kafka metrics reporter ------------------------------------------------------------------------------------------------------------------------------------- 0.21s
Attend que le broker en question soit dans le cluster avant de restart un autre ------------------------------------------------------------------------------------------------------------------ 0.20s

real 1m5,575s
user 0m27,440s
sys 0m2,207s

All tasks are divided by 2.

Installation

Installation is super easy :

git clone https://github.com/dw/mitogen.git

Update ansible.cfg :

strategy_plugins = ~/git/seuf/mitogen/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

That’s it !

You  can now run your playbooks faster !

Tips : I’ve updated my sudoers configuration file to allow the commands

deploy = (ALL) NOPASSWD:/usr/bin/python -c*

Prometheus : Monitor docker services with grafana

Here is a little tutorial to setup Prometheus monitoring for a local docker daemon and host metrics with docker-compose and :

  • prometheus node exporter for system metrics
  • cadvisor for docker metrics

First create a prometheus.yml config file like this :

global:
alerting:
  alertmanagers:
    - static_configs:
      - targets:
#       - alertmanager:9093
rule_files:
 # - "first_rules.yml"
 # - "second_rules.yml"
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.yml'

And a file service discovery config file_sd_configs.yml:

- targets:
  - grafana:3000
  - prom-node-exporter:9100
  - cadvisor:8080

Now it’s time to start all services with a docker-compose file.

I’m still using the great traefik for reverse proxy and ssl let’s encrypt certificate generation.

So here is my docker-compose :

version: '2'
services:
   traefik:
    image: traefik
    hostname: traefik
    container_name: traefik
    command: -c /etc/traefik/traefik.toml
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/data/traefik/traefik.toml:/etc/traefik/traefik.toml"
      - "/data/traefik/acme.json:/etc/traefik/acme.json"
      - "/data/traefik/ssl:/etc/traefik/ssl"
      - "/var/run/docker.sock:/var/run/docker.sock"

  prometheus:
    image: prom/prometheus
    hostname: prometheus
    container_name: prometheus
    volumes:
      - "/data/docker/volumes/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"
       - "/data/docker/volumes/prometheus/file_sd_configs.yml:/etc/prometheus/file_sd_configs.yml"
     labels:
      - "traefik.frontend.rule=Host:prometheus.aperogeek.fr"
 
  prom_node_exporter:
    image: prom/node-exporter
    hostname: prom-node-exporter
    container_name: prom-node-exporter
    labels:
      - "traefik.enable=false"
 
  cadvisor:
    image: google/cadvisor:latest
    hostname: cadvisor
    container_name: cadvisor
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
    labels:
      - "traefik.enable=false"
 
  grafana:
    hostname: grafana
    container_name: grafana
    image: grafana/grafana
    environment:
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel
    volumes:
      - /data/grafana:/var/lib/grafana
    labels:
      - "traefik.frontend.rule=Host:grafana.aperogeek.fr"

Then I needed to create a prometheus datasource in grafana

And finally I’ve just imported some public grafana dashboards. 

And voila 🙂