How to Use Ansible with Jenkins and Gitlab

Ansible

In this article I will try to explain how, here at OI-ERDF Lyon we use ansible Through Jenkins.

Ansible is an Open source Deployment and automation tool.

It is based on ssh, so no agents are required and it’s compatible on Linux / Aix / Solaris and even Windows !

It use playbooks, writted in YAML, to describe the list of tasks that have to be executed on remote servers.

Each task can be associated to a specific host or host group defined in an inventory.

For example :

we have an inventory like this:

dev\

    group_vars\
        main.yml
    host_vars\
        main.yml
    hosts.txt

recette\

    group_vars\
        main.yml
    host_vars\
        main.yml
    hosts.txt

production

    group_vars\
        main.yml
    host_vars\
        main.yml
    hosts.txt

 

 

where a host file looks like :

[apache]

server1

server2

[database]

server3

server4

[mysql]

server3

[postgresql]

server4

 

 

then a playbook define a list of tasks:

hosts: all

  - tasks:

    - name: "display host type"

      shell: uname

      register: host_type

    - debug: host_type.stdout

hosts: apache

  - roles:

    - role: apache

hosts: mysql

  - roles:

    - roles: mysql_server

hosts: postgresql

  - roles:

    - roles: postgresql_server

hosts: database

  - roles:

    - role: mysql_client

 

 

Here we can see that all hosts will execute the uname command and display the result, then each inventory group is affected to the corresponding role.

Simple, clear, powerfull.

to launch a playbook just use :

ansible-playbook -i dev/hosts.txt playbook.yml

Gitlab

First, we need a version control system to store all our ansible stuffs.

We choosed Gitlab, cause it has a lot of good features :

• User friendly
• Wiki integrated for all projects
• Issue tracker
• ldap auth integration
• Open source
• hooks
• and more…

One it’s installed with the community edition packages (rpm or deb) on a dedicated server, you juste have to create a new group « ANSIBLE » in which all the galaxy will be stored.

Ansible Galaxy

 

The best way to capitalize and re-use Ansible roles, modules and plugins is the Ansible Galaxy.

Our servers are in DMZ so we can’t access the internet web site ansible galaxy. We have to create our own Galaxy with our own ansible roles and module.

In the ANSIBLE group previously created in Gitlab juste create as many role, modules and plugins as you need for ansible deployments.

To find easlily our modules we have defined some dev rules :

• all ansible roles starts with « role_ »
• all ansible modules starts with « module_ »
• all ansible plugins starts with « plugin_ »

a role has the following structure :

role_example:
    defaults\
        main.yml
    files\
        file1
        file2
    handlers\
        main.yml
    meta\
        main.yml
    tasks\
        main.yml
        include1.yml
        include2.yml
    templates\
        template1.conf.j2
        template2.conf.j2
    vars\
        main.yml
    README.md

Where :

• defaults is the directory where all the defaults variables for the role are defined. Each var can be overloaded by a project by a host var / group var or extra var.
• files is the directory where the role can store fixed files (almost not used)
• handlers is the directory where the ansible handlers tasks are defined for the role (i.e. restart service etc…)
• meta is used to store the ansible galaxy metadata (author / description / dependencies / tags / etc..)
• tasks is the directory where all the the role main tasks are defined (create FS, apt-get, yum, deploy config file, restart service, etc…)
• templates is the directory where we store the role’s Jinja2 template files (withe the {{ variables }} replaced)
• vars is a directory where are stored role specific variables. This variable cannot be overloaded !
• README.md is the Markdown readme file used to describe the role and defaults vars that can be overridden.

Then publish your role to a new project role_example in your ansible group in Gitlab.

Ansible Projects

Now we can create projects that will use the galaxy.

To use the Ansible Galaxy, just create re requirements.yml file and put

- src: git+https://gitlab.aperogeek.fr/ansible/role_apache.git
  path: roles
  name: role_apache

- src: git+https://gitlab.aperogeek.fr/ansible/role_mysql_server.git
  path: roles
  name: role_mysql

- src: git+https://gitlab.aperogeek.fr/ansible/role_postgresql.git
  path: roles
  name: role_postgresql

- src: git+https://gitlab.aperogeek.fr/ansible/role_mysql_client.git
  path: roles
  name: role_mysql_client

- src: git+https://gitlab.aperogeek.fr/ansible/plugin_tail.git
  path: filter_plugins
  name: tail

- src: git+https://gitlab.aperogeek.fr/ansible/module_assert.git
  path: library
  name: assert

 

 

And install it to your project by using the command

ansible-galaxy install -r requirements.yml

This will fetch all roles, plugins and modules into the corrects directory.

Now you can use this roles into your project and overload roles defaults variable with your inventory.

Of course the project use Gitlab to store the playbooks,  inventory and requirements.

Jenkins

Now we have :

• Gitlab to store ansible galaxy and projects playbooks and inventory
• Ansible Galaxy with our own roles that respect our dev rules and the enterprise standards
• Projects that use the ansible galaxy to deploy their apps

All we need is a web UI to simplify the deployments.

Jenkins is a unit test / task launcher with a lot of plugins like GIT, MultiSCM, Rebluid, etc..

We have created jobs that will :

• Checkout our project playbook, inventory and requirements file (with the Jenkins MultiSCM plugin)
• Install the requirements in the job workspace (ansible-galaxy install -r requirements.yml)
• launch the ansible playbook (ansible-playbook -i ${env}/hosts.txt playbook.yml

The job has some parameters like

• git branch : the name of the git branch to use (master by default)
• env : the name of the environnement we want to deploy (aka the inventory file)
• extra var : if you want to specify some more extra vars
• ansible options : to allow users to specify more options (–limit=apache for example).

 

Then the job use the git or MultiSCM plugin to fetch playbooks from gitlab

 

Don’t forget to create a deploy key in gitlab with the public key of the jenkins user (www-data for example).

And finally launch the playbook from the job workspace :

Now every user with jenkins permissions can run deployments on any environment with just a few clicks

Conclusion

Ansible Galaxy with Gitlab and Jenkins give a lot of benefits :

• Stabilization or Production deployments (no more forgotten tasks)
• Shortest deployments duration (we can deploy all applications stack from scratch in less than an hour with ansible (a week before) )
• Capitalization : each project can re-use roles from the galaxy and contribute !
• Documentation : Gitlab integrate a wiki, so each role of the galaxy is documented. (We also have example projects, 10 steps tutorials)
• Collaboration through Gitlab issue system.
• Simplicity : Just click on the BIG DEPLOY button !

And you : which automation system are you using ? Chef ? Puppet ?