ARA

Ansible Run Analysis

+

Linux Montréal - Janvier 2018

David Moreau Simard

@dmsimard

$ whoami

David moreau simard

Senior software engineer @ red hat

  • OpenStack: SysAdmin "root" et core reviewer de l'infrastructure du projet OpenStack
  • RDO: Distribution OpenStack open source communautaire pour CentOS et RHEL

Ce que je fais

  • 10 ans chez iWeb (acquis par Internap), fournisseur de centre de données, serveurs et cloud

Ce que j'ai fait

Agenda

  • openstack

  • ansible

  • la vie avant ara

  • comment ara fonctionne

  • la vie après ara

  • Questions

"Open source software for creating private and public clouds"

"Open source software for creating private and public clouds"

  • 960 projets

  • 40 000 commits

  • 146 000 code reviews

  • 208 compagnies

  • 2400 contributeurs

  • Plus de 10 000+ jobs "first-party" chaque jour

OpenStack Pike, stackalytics.org

<3

  • Infrastructure pour le projet

  • Installation, déploiement, configuration d'Openstack

  • Opérations

  • Modules Ansible "upstream" pour consommer des clouds OpenStack

  • Projets OpenStack-Ansible, Kolla-Ansible, etc.

Continuous Integration (CI)

<3

CI: Zuul

<3

CI: Zuul

C'est quoi ansible ?

Ansible is a radically simple IT automation system.

It handles configuration-management, application deployment, cloud provisioning, ad-hoc task-execution, and multinode orchestration - including trivializing things like zero downtime rolling updates with load balancers.

> https://github.com/ansible/ansible/blob/devel/README.md

les vulnérabilités meltdown et spectre ?

Mettre à jour le kernel sur des centaines, milliers de serveurs ?

C'est quoi ansible ?

commandes "ad-hoc"

$ ansible -i hosts -m package -a "name=kernel state=latest" all
# Don't try this at home
$ ansible -i hosts -m command -a "shutdown -r now"

C'est quoi ansible ?

- hosts: all
  tasks:
    - name: Read kernel configuration
      shell: cat /boot/config-$(uname -r)
      register: kernel_config
      changed_when: false

    # Install latest kernel if it's not patched
    - when: '"CONFIG_KAISER=y" not in kernel_config.stdout'
      block:
        - name: Install latest kernel
          package:
            name: kernel
            state: latest

        - name: Reboot the server
          command: shutdown -r +1

        - name: Wait for the server to come back up
          wait_for_connection:
            timeout: 180

Playbooks

C'est quoi ansible ?

Playbooks

C'est quoi ansible ?

$ ansible-playbook -i hosts playbook.yml 
PLAY [all] *********************************************

TASK [Read kernel configuration] ***********************
ok: [centos]

TASK [Install latest kernel] ***************************
changed: [centos]

TASK [Reboot the server] *******************************
changed: [centos]

TASK [Wait for the server to come back up] *************
ok: [centos]

PLAY RECAP *********************************************
centos: ok=5    changed=2    unreachable=0    failed=0

Playbooks

C'est quoi ansible ?

$ ansible-playbook -i hosts playbook.yml 
PLAY [all] *********************************************

TASK [Read kernel configuration] ***********************
ok: [centos]

TASK [Install latest kernel] ***************************
skipped: [centos]

TASK [Reboot the server] *******************************
skipped: [centos]

TASK [Wait for the server to come back up] *************
skipped: [centos]

PLAY RECAP *********************************************
centos: ok=2    changed=0    unreachable=0    failed=0
$ ANSIBLE_CALLBACK_WHITELIST=profile_tasks ansible-playbook -i hosts playbook.yml

[...]
PLAY RECAP **************************************************************
centos: ok=4    changed=2    unreachable=0    failed=0   

Tuesday 09 January 2018  15:01:36 -0500 (0:00:00.483) 0:01:37.440 ******* 
=========================================================================
Install latest kernel -------------------------------------------- 95.06s
Read kernel configuration ----------------------------------------- 1.37s
Wait for the server to come back up ------------------------------- 0.48s
Reboot the server ------------------------------------------------- 0.46s

Callback plugins enable you to hook into Ansible events for display or logging purposes.

> http://docs.ansible.com/ansible/dev_guide/developing_plugins.html

PlUGINS (callbacks)

C'est quoi ansible ?

la vie avant ara...

35 000 lignes
10 MB (si votre navigateur ne plante pas)
10 000 fois par jour ?

"human_log"

changed: [localhost] => {"changed": true, "invocation": {"module_args": 
{"name": "python-openstackclient", "state": "present"}, "module_name": "dnf"}, 
"results": ["Installed: python-openstackclient-1.0.3-3.fc23.noarch", 
"Installed: python-unittest2-0.8.0-3.fc23.noarch", "Installed: 
python-keyring-5.0-2.fc23.noarch", "Installed: 
python-neutronclient-2.4.0-2.fc23.noarch", "Installed: 
python-keystoneclient-1:1.3.0-2.fc23.noarch", "Installed: 
python-jsonpointer-1.9-2.fc23.noarch", "Installed: 
python-warlock-1.0.1-3.fc23.noarch", "Installed: 
python-oslo-i18n-1.5.0-4.fc23.noarch", "Installed: 
python-novaclient-1:2.23.0-2.fc23.noarch", "Installed: 
python-fixtures-0.3.14-4.fc23.noarch", "Installed: 
python-testtools-1.8.0-2.fc23.noarch", "Installed: 
python-oslo-serialization-1.4.0-2.fc23.noarch", "Installed: 
python-iso8601-0.1.10-5.fc23.noarch", "Installed: 
python-oslo-utils-1.4.0-2.fc23.noarch", "Installed: 
python-msgpack-0.4.6-3.fc23.x86_64", "Installed: 
python-mimeparse-0.1.4-5.fc23.noarch", "Installed: 
python-webob-1.4.1-2.fc23.noarch", "Installed: 
python-jsonpatch-1.2-6.fc23.noarch", "Installed: 
python-cinderclient-1.2.1-1.fc23.noarch", "Installed: 
python-extras-0.0.3-7.fc23.noarch", "Installed: 
python-httplib2-0.9.1-2.fc23.noarch", "Installed: 
python-jsonschema-2.4.0-2.fc23.noarch", "Installed: 
python-glanceclient-1:0.17.0-3.fc23.noarch", "Installed: 
python-netifaces-0.10.4-2.fc23.x86_64"]}
results:
Installed: python-openstackclient-1.0.3-3.fc23.noarch
Installed: python-unittest2-0.8.0-3.fc23.noarch
Installed: python-keyring-5.0-2.fc23.noarch
Installed: python-neutronclient-2.4.0-2.fc23.noarch
Installed: python-keystoneclient-1:1.3.0-2.fc23.noarch
Installed: python-jsonpointer-1.9-2.fc23.noarch
Installed: python-warlock-1.0.1-3.fc23.noarch
Installed: python-oslo-i18n-1.5.0-4.fc23.noarch
Installed: python-novaclient-1:2.23.0-2.fc23.noarch
Installed: python-fixtures-0.3.14-4.fc23.noarch
Installed: python-testtools-1.8.0-2.fc23.noarch
Installed: python-oslo-serialization-1.4.0-2.fc23.noarch
Installed: python-iso8601-0.1.10-5.fc23.noarch
Installed: python-oslo-utils-1.4.0-2.fc23.noarch
Installed: python-msgpack-0.4.6-3.fc23.x86_64
Installed: python-mimeparse-0.1.4-5.fc23.noarch
Installed: python-webob-1.4.1-2.fc23.noarch
Installed: python-jsonpatch-1.2-6.fc23.noarch
Installed: python-cinderclient-1.2.1-1.fc23.noarch
Installed: python-extras-0.0.3-7.fc23.noarch
Installed: python-httplib2-0.9.1-2.fc23.noarch
Installed: python-jsonschema-2.4.0-2.fc23.noarch
Installed: python-glanceclient-1:0.17.0-3.fc23.noarch
Installed: python-netifaces-0.10.4-2.fc23.x86_64

ara

Ansible Run Analysis

ARA records Ansible playbook runs and makes the recorded data available and intuitive for users and systems.

> https://github.com/openstack/ara/readme.rst

# TL;DR
$ pip install ansible ara
[...]
$ export ANSIBLE_CALLBACK_PLUGINS=/usr/lib/python2.7/site-packages/ara/plugins/callbacks
$ ansible-playbook -i hosts playbook.yml
[...]
$ ara playbook list
+--------+----------+---------------------+----------+----------+-----------------+
| ID     | Path     | Time Start          | Duration | Complete | Ansible Version |
+--------+----------+---------------------+----------+----------+-----------------+
| <uuid> | test.yml | 2017-04-09 01:49:48 | 0:00:04  | True     | 2.2.2.0         |
+--------+----------+---------------------+----------+----------+-----------------+
$ ara-manage runserver
 * Running on http://127.0.0.1:5000/

Installation et utilisation

État des playbooks

Comment ça marche ?

Comment ça marche ?

# https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback
def v2_on_any(self, *args, **kwargs):
def v2_runner_on_failed(self, result, ignore_errors=False):
def v2_runner_on_ok(self, result):
def v2_runner_on_skipped(self, result):
def v2_runner_on_unreachable(self, result):
def v2_runner_on_no_hosts(self, task):
def v2_playbook_on_start(self, playbook):
def v2_playbook_on_task_start(self, task, is_conditional):
def v2_playbook_on_setup(self):
def v2_playbook_on_play_start(self, play):
def v2_playbook_on_stats(self, stats):
def v2_playbook_on_include(self, included_file):
[...]

ARA fourni un callback Ansible

# https://github.com/openstack/ara/blob/master/ara/plugins/callbacks/log_ara.py
[...]
class CallbackModule(CallbackBase):
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'notification'
    CALLBACK_NAME = 'ara'

    # [...]
    def v2_playbook_on_start(self, playbook):
        self.playbook = models.Playbook(
            ansible_version=ansible_version,
            path=os.path.abspath(playbook._file_name)
        )
        db.session.add(self.playbook)
    
    def v2_runner_on_ok(self, result, **kwargs):
        self.log_task(result, 'ok', **kwargs)
    [...]

ARA fourni un callback Ansible

- name: Test playbook
  hosts: localhost
  tasks:
    - name: Get git version of playbooks
      command: git rev-parse HEAD
      register: git_version

    - name: Record git version
      ara_record:
        key: "git_version"
        value: "{{ git_version.stdout }}"

... et des modules ansible

# https://github.com/openstack/ara/blob/master/ara/plugins/actions/ara_record.py
[...]
class ActionModule(ActionBase):
    def run(self, tmp=None, task_vars=None):
        result = super(ActionModule, self).run(tmp, task_vars)

        key = self._task.args.get('key', None)
        value = self._task.args.get('value', None)
        type = self._task.args.get('type', 'text')

        try:
            self.create_or_update_key(playbook_id, key, value, type)
            result['msg'] = 'Data recorded in ARA for this playbook.'
        except Exception as e:
            result['failed'] = True
            result['msg'] = 'Data not recorded in ARA: {0}'.format(str(e))
        return result

... et des modules ansible

Demo  !

(Improvisée...)

ou en savoir plus ?

Source:        github.com/openstack/ara
IRC:           #ara @ freenode
Slack:         github.com/openstack/ara
Twitter:       @ara_community
Demo:          bit.ly/ara-demo-013
Source:        github.com/openstack
IRC:           #openstack @ freenode
               #openstack-infra @ freenode

Merci !

Questions ?

@dmsimard
Source:        github.com/openstack/ara
IRC:           #ara @ freenode
Slack:         github.com/openstack/ara
Twitter:       @ara_community
Demo:          bit.ly/ara-demo-013
Source:        github.com/openstack
IRC:           #openstack @ freenode
               #openstack-infra @ freenode

ARA: Ansible Run Analysis

By David Moreau Simard

ARA: Ansible Run Analysis

  • 445
Loading comments...

More from David Moreau Simard