molecule-ansible-docker-aws

Example project showing how to test Ansible roles with Molecule using Testinfra and a multiscenario approach with Vagrant, Docker & AWS EC2 as the infrastructure under test.

There are already two blog posts complementing this repository:

• Test-driven infrastructure development with Ansible & Molecule
• Continuous Infrastructure with Ansible, Molecule & TravisCI
• Continuous cloud infrastructure with Ansible, Molecule & TravisCI on AWS

...and some articles in the german iX Magazin:

Table of Contents

• TDD for Infrastructure code with Molecule!
• Prerequisites
• Project structure
• Molecule configuration
• Execute Molecule
• Multi-Scenario Molecule setup
• Ubuntu based Docker-in-Docker builds
  • Docker-in-Docker with ubuntu:bionic
  • Testing the Docker-in-Docker installation
• Molecule with AWS EC2 as the infrastructure provider
  • Configure Molecule to use EC2
  • Having a look into the create.yml
  • Install & configure AWS CLI
  • Configure Region & VPC subnet id
  • Choosing an Ubuntu 18.04 AMI
  • Creating a EC2 instance with Molecule
  • Run a first Test on EC2 with Molecule
  • Cleaning up
  • Final check: molecule test
• Use TravisCI to execute Molecule with EC2 infrastructure
  • Problems with boto on Travis
  • Use pipenv with TravisCI
• Use CircleCI to execute Molecule with EC2 infrastructure
  • Use pipenv with CircleCI
  • Schedule regular CircleCI builds with workflow triggers & cron
• Upgrade to Molecule v3
• Use Vagrant on TravisCI to execute Molecule
  • Using VirtualBox locally furthermore - but switching to libvirt on TravisCI
  • Install Python 3 for sudo access & pipenv based on Python 3
  • Run our Molecule Vagrant Scenario

TDD for Infrastructure code with Molecule!

Molecule seems to be a pretty neat TDD framework for testing Infrastructur-as-Code using Ansible. As previously announced on September 26 2018 Ansible treats Molecule as a first class citizen from now on - backed by Redhat also.

Molecule executes the following steps:

In the Then phase Molecule executes different Verifiers, one of them is Testinfra, where you can write Unittest with a Python DSL.

Beware of old links to v1 of Molecule! As I followed blog posts that were just from a year or two ago, I walked into the trap of using Molecule v1. This is especially problematic if you try to install Molecule via homebrew where you currently also get the v1 version. So always double check, if there´s no /v1/ in the docs´ url:

Just start here: Molecule docs

Prerequisites

• brew cask install virtualbox
• brew cask install vagrant

Please don´t install Ansible and Molecule with homebrew on Mac, but always with pip3 since you only get old versions and need to manually install testinfra, ansible, flake8 and other packages

And as we learned with pulumi-python-aws-ansible and escpecially in Replace direct usage of virtualenv and pip with pipenv, we should use a great Python dependency management tool like pipenv instead of no or non-pinned (requirements.txt) dependencies.

Therefore let's install pipenv:

pip3 install pipenv

Then create a virtual environment for the current project (pinned to Python 3.9+):

pipenv shell --python 3.9

And finally install needed pip packages with:

pipenv install

This will install all needed dependencies which are listed incl. their versions in the Pipfile.

This project uses the following molecule modules:

• molecule-docker (needs molecule-docker and docker pip packages)
• molecule-ec2 (needs molecule-ec2 and awscli and boto3 pip packages)
• molecule-vagrant for Vagrant support (needs molecule-vagrant and python-vagrant pip packages)

Project structure

To initialize a new Molecule powered Ansible role named docker with the Vagrant driver and the Testinfra verifier you have to execute the following command:

pipenv run molecule init role docker --verifier-name testinfra --driver-name docker

This will give:

--> Initializing role docker... Successfully initialized new role in /Users/jonashecht/dev/molecule-ansible-vagrant/docker.

Molecule introduces a well known project structure (at least for a Java developer like me):

As you may notice the role standard directory tasks is now accompanied by a tests directory inside the molecule folder where the Testinfra testcases reside.

This repository uses a Ansible role that installs Docker into an Ubuntu Box:

- name: add Docker apt key
  apt_key:
    url: https://download.docker.com/linux/ubuntu/gpg
    id: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
    state: present
  ignore_errors: true

- name: add docker apt repo
  apt_repository:
    repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_lsb.codename }} stable"
    update_cache: yes
  become: true

- name: install Docker apt package
  apt:
    pkg: docker-ce
    state: latest
    update_cache: yes
  become: true

- name: add vagrant user to docker group.
  user:
    name: vagrant
    groups: docker
    append: yes
  become: true

With Testinfra we can assert on things we want to achieve with our Ansible role: Install the docker package and add the user vagrant to the group docker. Testinfra uses pytest to execute the tests. Our testcases could be found in test_docker.py:

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


def test_is_docker_installed(host):
    package_docker = host.package('docker-ce')

    assert package_docker.is_installed


def test_vagrant_user_is_part_of_group_docker(host):
    user_vagrant = host.user('vagrant')

    assert 'docker' in user_vagrant.groups
    

More Testinfra code examples:

As you´re not an in-depth Python hacker (like me), you´ll be maybe also interested in example code. Have a look at:

https://github.com/philpep/testinfra#quick-start

https://github.com/openmicroscopy/ansible-role-prometheus/blob/0.2.0/tests/test_default.py

https://github.com/mongrelion/ansible-role-docker/blob/master/molecule/default/tests/test_default.py

https://blog.octo.com/test-your-infrastructure-topology-on-aws/

Molecule configuration

The molecule.yml configures Molecule:

scenario:
  name: vagrant-ubuntu

driver:
  name: vagrant
  provider:
    name: virtualbox
platforms:
  - name: vagrant-ubuntu
    box: ubuntu/bionic64
    memory: 512
    cpus: 1

provisioner:
  name: ansible
  lint:
    name: ansible-lint
    enabled: false
  playbooks:
    converge: playbook.yml

lint:
  name: yamllint
  enabled: false
verifier:
  name: testinfra
  directory: ../tests/
  env:
    # get rid of the DeprecationWarning messages of third-party libs,
    # see https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning
    PYTHONWARNINGS: "ignore:.*U.*mode is deprecated:DeprecationWarning"
  lint:
    name: flake8
  options:
    # show which tests where executed in test output
    v: 1

There's a special PYTHONWARNINGS: "ignore:.*U.*mode is deprecated:DeprecationWarning" environment variable definition. If you don´t configure this you´ll end up with bloated test logs like this:

If you use the PYTHONWARNINGS environment variable you gather beautiful and green test executions:

In case everything runs green you may notice that there´s is no hint which tests were executed. But I think that´s rather a pity since we want to see our whole test suite executed. That was the whole point why we even started to use a testing framework like Molecule!

But luckily there´s a way to get those tests shown inside our output. As Molecule uses Testinfra which itself leverages pytest to execute our test cases and Testinfra is able to invoke pytest with additional properties. And pytest has many options we can experiment with. To configure a more verbose output for our tests in Molecule, add the following to the verifier section of your molecule.yml:

verifier:
  name: testinfra
...
  options:
    # show which tests where executed in test output
    v: 1
...

If we now execute a molecule verify we should see a much nicer overview of which test cases where executed:

Execute Molecule

Now we´re able to run our first test. Go into docker directory and run:

molecule test

As Molecule has different phases, you can also explicitely run molecule converge or molecule verify - the commands will recognice required upstream phases like create and skips them if they where already run before (e.g. if the Vagrant Box is running already).

Multi-Scenario Molecule setup

With Molecule we could not only test our Ansible roles against one infrastructure setup - but we can use multiple of them! We only need to leverage the power of Molecule scenarios.

To get an idea on how this works I sligthly restructured the repository. We started out with the Vagrant driver / scenario. Now after also implementing a Docker driver in this repository on it´s own: https://github.com/jonashackt/molecule-ansible-docker I integrated it into this repository.

And because Docker is the default Molecule driver for testing Ansible roles I changed the name of the Vagrant scenario to vagrant-ubuntu. Don´t forget to install docker:

pip3 install molecule-docker docker

Using molecule test as we´re used to will now execute the Docker (e.g. default) scenario. This change results in the following project structure:

To execute the vagrant-ubuntu Scenario you have to explicitely call it´s name:

molecule test --scenario-name vagrant-ubuntu

All the files which belong only to a certain scenario are placed inside the scenarios directory. For example in the Docker scenario there's only the molecule.yml and in Vagrant one this is an additional prepare.yml. Also the molecule.yml files have to access the playbook.yml and the testfiles differently since they are now separate from the scenario directory to be able to reuse them over all scenarios:

...
provisioner:
  name: ansible
...
  playbooks:
    converge: ../playbook.yml
...
verifier:
  name: testinfra
  directory: ../tests/
...

Now we can also integrate & use TravisCI in this repository since the default scenario Docker is supported on Travis! :)

Ubuntu based Docker-in-Docker builds

As we don´t have a Vagrant environment in a Cloud CI system available for us right now (see https://github.com/jonashackt/vagrant-ansible-on-appveyor), we should be enable us somehow to test our Ansible role only with the Docker driver.

The standard Docker-in-Docker image provided by Docker Inc is based on Alpine Linux (see https://stackoverflow.com/a/53459483/4964553 & the dind tags in https://hub.docker.com/_/docker/). But our role is designed for Ubuntu and thus uses the apt package manager instead of apk. So we can´t use the standard Docker-in-Docker image.

But there should be a way to do Docker-in-Docker installation with a Ubuntu base image like ubuntu:bionic! And there is :)

Let´s assume the standard Ubuntu Docker installation our starting point. Everything described there is done inside our Ansible role under test docker inside the playbook docker/tasks/main.yml.

All the extra steps needed to install Docker inside a Ubuntu Docker container will be handled inside the prepare step. Therefore we´ll use Molecules´ prepare.yml playbook:

The prepare playbook executes actions which bring the system to a given state prior to converge. It is executed after create, and only once for the duration of the instances life. This can be used to bring instances into a particular state, prior to testing.

Configure a custom prepare step

The Docker-in-Docker build is only used (and should only be used) inside our CI pipeline. The prepare step inside our Molecule test suites´s default Docker scenario will be only executed for testing purposes.

So let´s configure our default Docker scenario to use a prepare.yml which could be done inside the molecule.yml:

...
  playbooks:
    prepare: prepare-docker-in-docker.yml
    converge: ../playbook.yml
...

Docker-in-Docker with ubuntu:bionic

Now we should have a look into the prepare-docker-in-docker.yml:

# Prepare things only necessary in Ubuntu Docker-in-Docker scenario
- name: Prepare
  hosts: all
  tasks:
    # We need to anticipate the installation of Docker before the role execution...
  - name: use our role to install Docker
    include_tasks: ../../tasks/main.yml

  - name: create /etc/docker
    file:
      state: directory
      path: /etc/docker

  - name: set storage-driver to vfs via daemon.json
    copy:
      content: |
        {
          "storage-driver": "vfs"
        }
      dest: /etc/docker/daemon.json

  # ...since we need to start Docker in a complete different way
  - name: start Docker daemon inside container see https://stackoverflow.com/a/43088716/4964553
    shell: "/usr/bin/dockerd -H unix:///var/run/docker.sock > dockerd.log 2>&1 &"

First the Docker installation has to be executed just in the same way as on a virtual machine using Vagrant. So we simply re-use the existing Docker role here - so we´re not forced to copy code!

Then really being able to run Docker-in-Docker we need to do three things:

1. Run Docker with --priviledged (which should really only be used inside our CI environment, because it grant´s full access to the host environment (see https://hub.docker.com/_/docker/))
2. Use the storage-driver vfs, which is slow & inefficient but is the only one guaranteed to work regardless of underlying filesystems
3. Start the Docker daemon with /usr/bin/dockerd -H unix:///var/run/docker.sock > dockerd.log 2>&1 &, or otherwise you´ll run into errors like Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (see https://stackoverflow.com/a/43088716/4964553)

You may noticed that 2. & 3. are handled by our prepare-docker-in-docker.yml already. To enable the 1. --priviledged mode we need to configure Molecules´ Docker driver inside the molecule.yml:

...
driver:
  name: docker
platforms:
  - name: docker-ubuntu
    image: ubuntu:bionic
    privileged: true
...

Testing the Docker-in-Docker installation

The last step - or the first, if you leverage the power of Test-Driven-Development (TDD) - was to create a suitable testcase. This test should verify whether the Docker-in-Docker installation was successful.

Therefore we can use the hello-world Docker image. Let´s execute a docker run hello-world straight inside our test case test_run_hello_world_container_successfully in our test suite test_docker.py:

def test_run_hello_world_container_successfully(host):
    hello_world_ran = host.run("docker run hello-world")

    assert 'Hello from Docker!' in hello_world_ran.stdout

This will verify that

1. the Docker client is able to contact the Docker daemon
2. the Docker daemon successfully pulled the image hello-world from the Docker Hub
3. the Docker daemon created a new container from that image and runs the executable inside
4. the Docker daemon streamed the executables output containing Hello from Docker! to the Docker client, which send it to the terminal

Molecule with AWS EC2 as the infrastructure provider

Now that we successfully used Vagrant & Docker as infrastructure providers for Molecule, we should now start to use a cloud provider like AWS.

We should be able to use Molecule's EC2 driver, which itself uses Ansible's ec2 module to interact with AWS.

First we need to install the Boto Python packages. They will provide interfaces to Amazon Web Services:

pip3 install molecule-ec2 awscli boto3

Configure Molecule to use EC2

Then we just init a new Molecule scenario inside our existing multi scenario project called aws-ec2-ubuntu. Therefore we can leverage Molecule's molecule init scenario command:

cd molecule-ansible-docker-aws

molecule init scenario --driver-name ec2 --role-name docker --scenario-name aws-ec2-ubuntu

That should create a new directory aws-ec2-ubuntu inside the molecule folder. We'll integrate the results into our multi scenario project in a second.

Now let's dig into the generated molecule.yml:

scenario:
  name: aws-ec2-ubuntu

driver:
  name: ec2
platforms:
  - name: instance
    image: ami-a5b196c0
    instance_type: t2.micro
    vpc_subnet_id: subnet-6456fd1f

provisioner:
  name: ansible
  lint:
    name: ansible-lint
    enabled: false
  playbooks:
    converge: ../playbook.yml

lint:
  name: yamllint
  enabled: false

verifier:
  name: testinfra
  directory: ../tests/
  env:
    # get rid of the DeprecationWarning messages of third-party libs,
    # see https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning
    PYTHONWARNINGS: "ignore:.*U.*mode is deprecated:DeprecationWarning"
  lint:
    name: flake8
  options:
    # show which tests where executed in test output
    v: 1

As we already tuned the molecule.yml files for our other scenarios like docker and vagrant-ubuntu, we know what to change here. provisioner.playbook.converge needs to be configured, so the one playbook.yml could be found.

Also the verifier section has to be enhanced to gain all the described advantages like supressed deprecation warnings and the better test result overview.

As you may noticed, the driver now uses ec2 and the platform is already pre-configured with a concrete Amazon Machine Image (AMI) and instance_type etc. I just tuned the instance name to aws-ec2-ubuntu, like we did in our Docker and Vagrant scenarios.

Having a look into the create.yml

The generated create.yml and destroy.yml Ansible playbooks look rather stunning - especially to AWS newbees. Let's have a look into the create.yml:

- name: Create
  hosts: localhost
  connection: local
  gather_facts: false
  no_log: "{{ not (lookup('env', 'MOLECULE_DEBUG') | bool or molecule_yml.provisioner.log|default(false) | bool) }}"
  vars:
    ssh_user: ubuntu
    ssh_port: 22

    security_group_name: molecule
    security_group_description: Security group for testing Molecule
    security_group_rules:
      - proto: tcp
        from_port: "{{ ssh_port }}"
        to_port: "{{ ssh_port }}"
        cidr_ip: '0.0.0.0/0'
      - proto: icmp
        from_port: 8
        to_port: -1
        cidr_ip: '0.0.0.0/0'
    security_group_rules_egress:
      - proto: -1
        from_port: 0
        to_port: 0
        cidr_ip: '0.0.0.0/0'

    keypair_name: molecule_key
    keypair_path: "{{ lookup('env', 'MOLECULE_EPHEMERAL_DIRECTORY') }}/ssh_key"
  tasks:
    - name: Create security group
      ec2_group:
        name: "{{ security_group_name }}"
        description: "{{ security_group_name }}"
        rules: "{{ security_group_rules }}"
        rules_egress: "{{ security_group_rules_egress }}"

    - name: Test for presence of local keypair
      stat:
        path: "{{ keypair_path }}"
      register: keypair_local

    - name: Delete remote keypair
      ec2_key:
        name: "{{ keypair_name }}"
        state: absent
      when: not keypair_local.stat.exists

    - name: Create keypair
      ec2_key:
        name: "{{ keypair_name }}"
      register: keypair

    - name: Persist the keypair
      copy:
        dest: "{{ keypair_path }}"
        content: "{{ keypair.key.private_key }}"
        mode: 0600
      when: keypair.changed

    - name: Create molecule instance(s)
      ec2:
        key_name: "{{ keypair_name }}"
        image: "{{ item.image }}"
        instance_type: "{{ item.instance_type }}"
        vpc_subnet_id: "{{ item.vpc_subnet_id }}"
        group: "{{ security_group_name }}"
        instance_tags:
          instance: "{{ item.name }}"
        wait: true
        assign_public_ip: true
        exact_count: 1
        count_tag:
          instance: "{{ item.name }}"
      register: server
      with_items: "{{ molecule_yml.platforms }}"
      async: 7200
      poll: 0

    - name: Wait for instance(s) creation to complete
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: ec2_jobs
      until: ec2_jobs.finished
      retries: 300
      with_items: "{{ server.results }}"

    # Mandatory configuration for Molecule to function.

    - name: Populate instance config dict
      set_fact:
        instance_conf_dict: {
          'instance': "{{ item.instances[0].tags.instance }}",
          'address': "{{ item.instances[0].public_ip }}",
          'user': "{{ ssh_user }}",
          'port': "{{ ssh_port }}",
          'identity_file': "{{ keypair_path }}",
          'instance_ids': "{{ item.instance_ids }}", }
      with_items: "{{ ec2_jobs.results }}"
      register: instance_config_dict
      when: server.changed | bool

    - name: Convert instance config dict to a list
      set_fact:
        instance_conf: "{{ instance_config_dict.results | map(attribute='ansible_facts.instance_conf_dict') | list }}"
      when: server.changed | bool

    - name: Dump instance config
      copy:
        content: "{{ instance_conf | to_json | from_json | molecule_to_yaml | molecule_header }}"
        dest: "{{ molecule_instance_config }}"
      when: server.changed | bool

    - name: Wait for SSH
      wait_for:
        port: "{{ ssh_port }}"
        host: "{{ item.address }}"
        search_regex: SSH
        delay: 10
        timeout: 320
      with_items: "{{ lookup('file', molecule_instance_config) | molecule_from_yaml }}"

    - name: Wait for boot process to finish
      pause:
        minutes: 2

If we skim over the code, we notice the usage of Ansible's ec2_group module. It is used to maintains AWS EC2 security groups. Using the parameters rules and rules_egress, it configures appropriate firewall inbound and outbound rules.

Thereafter Ansible's ec2_key module is used to create a new EC2 key pair & store it locally, if non exists on your local machine already.

And then the ec2 module takes over, which is able to create or terminate AWS EC2 instances without the need to hassle with the GUI.

Waiting for the EC2 instance to come up, the create.yml playbook uses the async_status module on the pre-registered variable server.results from the ec2 module.

The following Ansible module are solely used to create an inline Ansible inventory, which is finally used to connect into the EC2 instance via SSH.

The generated destroy.yml playbook is just the opposite to the create.yml playbook and tears the created EC2 instance down.

Install & configure AWS CLI

Now let's give our configuration a shot. Just be sure to meet some prerequisites.

First we need to sure to have the AWS CLI installed. We can also do this via pip package manager with:

pipenv install awscli

Now we should check, if AWS CLI was successfully installed. The aws --version command should print out sometime like:

$ aws --version
aws-cli/1.16.80 Python/3.7.2 Darwin/18.2.0 botocore/1.12.70

Now configure the AWS CLI to use the correct credentials. According to the AWS docs, the fastest way to accomplish that is to run aws configure:

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-central-1
Default output format [None]: json

Configure Region & VPC subnet id

As there currently is a discrepancy in the UX of Molecule, where every configuration parameter is generated correctly by a molecule init --driver-name ec2 command and picked up from ~/.aws/credentials. Except the region configuration from ~/.aws/config!

Running a Molecule test without setting the region correctly as environment variable, currently results in the (already documented) error:

"msg": "Either region or ec2_url must be specified",

So for now we need to set the region manually before running our Molecule tests against EC2:

export AWS_REGION=eu-central-1

And there's another thing to do: We need to configure the correct vpc_subnet_id inside our molecule.yml - if not, we get an error like this:

    "    raise self.ResponseError(response.status, response.reason, body)",
    "boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request",
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
    "<Response><Errors><Error><Code>InvalidSubnetID.NotFound</Code><Message>The subnet ID 'subnet-6456fd1f' does not exist</Message></Error></Errors><RequestID>1c971ba2-0d86-4335-9445-d989e988afce</RequestID></Response>"
]

We need to somehow figure out the correct VPC subnet id of our region. Therefore, you can simply use Ansible to gather that for you by using the ec2_vpc_subnet_facts module:

    - name: Gather facts about all VPC subnets
      ec2_vpc_subnet_facts:

This will give the correct String inside the field subnets.id:

    ok: [localhost] => {
        "changed": false,
        "invocation": {
            "module_args": {
                "aws_access_key": null,
                ...
            }
        },
        "subnets": [
            {
                "assign_ipv6_address_on_creation": false,
                "availability_zone": "eu-central-1b",
                "availability_zone_id": "euc1-az3",
                ...
                "id": "subnet-a2efa1d9",
                ...
            },

As an alternative you can use AWS CLI with aws ec2 describe-subnets to find the correct ID inside the field Subnets.SubnetId:

$ aws ec2 describe-subnets

{
    "Subnets": [
        {
            "AvailabilityZone": "eu-central-1b",
            "AvailabilityZoneId": "euc1-az3",
            ...
            "SubnetId": "subnet-a2efa1d9",
            ...
        },
        {
            "AvailabilityZone": "eu-central-1c",
            ...
        },
        {
            "AvailabilityZone": "eu-central-1a",
            ...
        }
    ]
}

Now head over to your molecule.yml and edit the vpc_subnet_id to contain the correct value:

driver:
  name: ec2
  
platforms:
  - name: aws-ec2-ubuntu
    image_owner: 099720109477
    image_name: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*
    instance_type: t2.micro
    vpc_subnet_id: subnet-a2efa1d9
    instance_tags:
      - Name: molecule-aws-ec2-ubuntu
...

Choosing an Ubuntu 18.04 AMI

[As stated here[(https://askubuntu.com/a/1031156/451114), Amazon has a "official" Ubuntu 18.04 AMI - but this one isn't available under the free tier right now.

But the official image is just the same as from Canonical, which could be found with the ubuntu Amazon EC2 AMI Locator:

Fill in your region - e.g. for me this is eu-central-1 - together with the desired Ubuntu version - like 18.04 LTS - and the site should filter all available AMI images in this region:

Now choose the AMI id with the Instance Type hvm:ebs-ssd, which means that our EC2 instance will use Amazon Elastic Block Storage memory. Only this instance type is eligible for the free tier. In our region the the correct AMI id for Ubuntu 18.04 is ami-080d06f90eb293a27. We need to configure this inside our molecule.yml:

scenario:
  name: aws-ec2-ubuntu

driver:
  name: ec2
platforms:
  - name: aws-ec2-ubuntu
    image: ami-080d06f90eb293a27
    instance_type: t2.micro
    vpc_subnet_id: subnet-a2efa1d9
...

Creating a EC2 instance with Molecule

Now we should have everything prepared. Let's try to run our first Molecule test on AWS EC2 (including --debug so that we see what's going on):

molecule --debug create --scenario-name aws-ec2-ubuntu

Right now we have an issue with the new community module

TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item={'image_name': 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*', 'image_owner': '099720109477', 'instance_tags': [{'Name': 'molecule-aws-ec2-ubuntu'}], 'instance_type': 't2.micro', 'name': 'aws-ec2-ubuntu', 'vpc_subnet_id': 'subnet-a2efa1d9'})

TASK [Wait for instance(s) creation to complete] *******************************
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).
ok: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '397490056884.47351', 'results_file': '/Users/jonashecht/.ansible_async/397490056884.47351', 'changed': True, 'failed': False, 'item': {'image_name': 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*', 'image_owner': '099720109477', 'instance_tags': [{'Name': 'molecule-aws-ec2-ubuntu'}], 'instance_type': 't2.micro', 'name': 'aws-ec2-ubuntu', 'vpc_subnet_id': 'subnet-a2efa1d9'}, 'ansible_loop_var': 'item', 'index': 0, 'ansible_index_var': 'index'})

TASK [Populate instance config dict] *******************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: list object has no element 0\n\nThe error appears to be in '/Users/jonashecht/dev/molecule-ansible-docker-aws/molecule/aws-ec2-ubuntu/create.yml': line 112, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Populate instance config dict\n      ^ here\n"}

See https://github.com/ansible-community/molecule/discussions/2946, https://github.com/ansible-community/molecule/pull/2869/files, https://github.com/ansible-community/molecule-ec2/pull/34 (only merged 3 days ago)

We could have a sneak peak into our Ansible EC2 Management console under https://eu-central-1.console.aws.amazon.com/ec2/v2/home?region=eu-central-1. We should see our EC2 instance starting up:

If everything went fine, the molecule create command should succeed and your EC2 console should show the running EC2 instance:

Run a first Test on EC2 with Molecule

Now let's try to run our Ansible role against our new EC2 instance with Molecule:

molecule converge --scenario-name aws-ec2-ubuntu

That should just work fine. We then could head over to the verify-phase:

molecule verify --scenario-name aws-ec2-ubuntu

This should successfully execute our Testinfra test suite described in test_docker.py onto our EC2 instance. If everything ran fine, it should give something like:

If the last test function fails with an error like:

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

you maybe have to add sudo to the docker run hello-world statement:

def test_run_hello_world_container_successfully(host):
    hello_world_ran = host.run("sudo docker run hello-world")

    assert 'Hello from Docker!' in hello_world_ran.stdout

Cleaning up

Finally it's time to clean up. Let's run a molecule destroy command to tear down or EC2 instance:

molecule --debug destroy --scenario-name aws-ec2-ubuntu

If that command went well, we can have a look into our AWS EC2 management console again:

Our instance should have reached the state terminated, which is simmilar to stopped state - just for instances that used EBS storage (see the AWS EC2 Instance Lifecycle chart for more info).

Here's a full asciinema cast of all the steps:

Final check: molecule test

As you may remember, Molecule provides the consecutive list of steps that run one by one. For introductory reasons we choosed to execute each step manually.

But having everything configured and in place right now, we should be able to run the command that summarizes all the others: molecule test:

molecule test --scenario-name aws-ec2-ubuntu

Use TravisCI to execute Molecule with EC2 infrastructure

My ultimate goal of the whole Molecule journey was to be able to let TravisCI create Cloud environments and execute Ansible roles on them.

So we should bring all the things learned togehter now:

Using Molecule to develop and test an Ansible role - togehter with the infrastructure provider AWS EC2 - automatically executed by TravisCI after commits or regularly with TravisCI cron jobs.

So let's do it! First we need to configure TravisCI. Therefore we need to enhance our .travis.yml. As we need the same python package additions as locally, we need to install boto, boto3 and awscli:

sudo: required
language: python

env:
- EC2_REGION=eu-central-1

services:
- docker

install:
- pip install molecule docker

# install AWS related packages
- pip install boto boto3
- pip install --upgrade awscli
# configure AWS CLI
- aws configure set aws_access_key_id $AWS_ACCESS_KEY
- aws configure set aws_secret_access_key $AWS_SECRET_KEY
- aws configure set default.region $DEPLOYMENT_REGION
# show AWS CLI config
- aws configure list

script:
# Molecule Testing Travis-locally with Docker
- molecule test
# Molecule Testing on AWS EC2
- molecule create --scenario-name aws-ec2-ubuntu
- molecule converge --scenario-name aws-ec2-ubuntu
- molecule verify --scenario-name aws-ec2-ubuntu
- molecule destroy --scenario-name aws-ec2-ubuntu

After that, we need to configure our AWS CLI to use the correct credentials and AWS region. This can be achieved by usind the aws configure set command. Then we need to head over to the settings tab of our TravisCI project (for the current project this can be found at https://travis-ci.org/jonashackt/molecule-ansible-docker-vagrant/settings) and insert the three environment variables AWS_ACCESS_KEY, AWS_SECRET_KEY and DEPLOYMENT_REGION:

The last part is to add the molecule commands to the script section of the .travis.yml.

Problems with boto on Travis

As https://github.com/boto/boto/issues/3717 suggests, there are currently problems with the AWS connection library boto on TravisCI. They result in errors like:

        "msg": "Traceback (most recent call last):\n  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/async_wrapper.py\", line 147, in _run_module\n    (filtered_outdata, json_warnings) = _filter_non_json_lines(outdata)\n  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/async_wrapper.py\", line 88, in _filter_non_json_lines\n    raise ValueError('No start of json char found')\nValueError: No start of json char found\n", 
        "stderr": "Traceback (most recent call last):\n  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_ec2_payload_y3lfWH/__main__.py\", line 552, in <module>\n  File \"/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/boto/__init__.py\", line 1216, in <module>\n    boto.plugin.load_plugins(config)\nAttributeError: 'module' object has no attribute 'plugin'\n", 
        "stderr_lines": [
            "Traceback (most recent call last):", 
            "  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 113, in <module>", 
            "    _ansiballz_main()", 
            "  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 105, in _ansiballz_main", 
            "    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)", 
            "  File \"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\", line 48, in invoke_module", 
            "    imp.load_module('__main__', mod, module, MOD_DESC)", 
            "  File \"/tmp/ansible_ec2_payload_y3lfWH/__main__.py\", line 552, in <module>", 
            "  File \"/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/boto/__init__.py\", line 1216, in <module>", 
            "    boto.plugin.load_plugins(config)", 
            "AttributeError: 'module' object has no attribute 'plugin'"
        ]
    }
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=6    changed=4    unreachable=0    failed=1   
    
    
ERROR: 
The command "molecule --debug create --scenario-name aws-ec2-ubuntu" exited with 2.

To fix this, there are two changes needed inside our .travis.yml. We need to set sudo: false and use the environment variable BOTO_CONFIG=/dev/null:

sudo: false
language: python

env:
- EC2_REGION=eu-central-1 BOTO_CONFIG="/dev/null"
...

And we need to add the BOTO_CONFIG environment variable to the same line as the already existing variable - otherwise Travis starts multiple builds with only one variable set to each build! See the docs (https://docs.travis-ci.com/user/environment-variables/#defining-multiple-variables-per-item)

If you need to specify several environment variables for each build, put them all on the same line in the env array

Now head over to Travis and have a look into the log. It should look green and somehow like this: https://travis-ci.org/jonashackt/molecule-ansible-docker-vagrant/builds/477365844

Use pipenv with TravisCI

As described in Replace direct usage of virtualenv and pip with pipenv, we use the great Python dependency management tool like pipenv instead of no or non-pinned (requirements.txt) dependencies. So we should also use it with TravisCI!

And as TravisCI has the needed virtualenv already installed per default in it's python machines, we only need to do (and replace the pip install XYZ commands):

  # Install pipenv dependency manager
  - pip install pipenv
  # Install required (and locked) dependecies from Pipfile.lock. pipenv is smart enough to recognise the existing 
  # virtualenv without a prior pipenv shell command (see https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b)
  - pipenv install

Use pipenv with TravisCI

As described in Replace direct usage of virtualenv and pip with pipenv, we use the great Python dependency management tool like pipenv instead of no or non-pinned (requirements.txt) dependencies. So we should also use it with TravisCI!

And as TravisCI has the needed virtualenv already installed per default in it's python machines, we only need to do (and replace the pip install XYZ commands):

  # Install pipenv dependency manager
  - pip install pipenv
  # Install required (and locked) dependecies from Pipfile.lock. pipenv is smart enough to recognise the existing 
  # virtualenv without a prior pipenv shell command (see https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b)
  - pipenv install

Use CircleCI to execute Molecule with EC2 infrastructure

As TravisCI is just one example of a cloud CI provider, let's use another one also - so let's just pick CircleCI, let's go with this CI tool!

Using Molecule to develop and test an Ansible role - togehter with the infrastructure provider AWS EC2 - automatically executed by CircleCI after commits or regularly with CircleCI scheduled jobs.

So let's do it! First we need to configure CircleCI. Therefore we need to create a .circleci/config.yml. As we need the same python package additions as locally, we need to install boto, boto3 and awscli:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7.5

    environment:
      EC2_REGION: eu-central-1

    working_directory: ~/molecule-ansible-docker-vagrant

    steps:
      - checkout

      - run:
          name: Install Molecule dependencies
          command: |
            pip install molecule docker

      - run:
          name: Run Molecule Testing CircleCI-locally with Docker
          command: |
            molecule test

      - run:
          name: Install Molecule AWS dependencies
          command: |
            pip install boto boto3
            pip install --upgrade awscli

      - run:
          name: configure AWS CLI
          command: |
            aws configure set aws_access_key_id ${AWS_ACCESS_KEY}
            aws configure set aws_secret_access_key ${AWS_SECRET_KEY}
            aws configure set default.region ${EC2_REGION}
            aws configure list

      - run:
          name: Run Molecule Testing on AWS EC2
          command: |
            molecule create --scenario-name aws-ec2-ubuntu
            molecule converge --scenario-name aws-ec2-ubuntu
            molecule verify --scenario-name aws-ec2-ubuntu
            molecule destroy --scenario-name aws-ec2-ubuntu

After that, we need to configure our AWS CLI to use the correct credentials and AWS region. This can be achieved by usind the aws configure set command. Then we need to head over to the settings tab of our CircleCI project (for the current project this can be found at https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/edit#env-vars) and insert the three environment variables AWS_ACCESS_KEY & AWS_SECRET_KEY:

The last part is to add the molecule commands to our .circleci/config.yml.

CircleCI gives Permission denied error at pip install

Using circleci/python Docker image it seems that you can't simply use pip install xyz:

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/ptyprocess'

This error is also reported here. To avoid this error, we need to use sudo to be able to install the pip packages successfully:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7.5
...
    steps:
      - checkout

      - run:
          name: Install Molecule dependencies
          command: |
            sudo pip install molecule docker

      - run:
          name: Run Molecule Testing CircleCI-locally with Docker
          command: |
            molecule test

"msg": "Error connecting: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"

Molecule needs a Docker service available. Since CircleCI is mostly using pure Docker-based images to run it's CI jobs, there's no Docker service inside the Docker containers available. But there's help! The docs state:

If your job requires docker or docker-compose commands, add the setup_remote_docker step into your .circleci/config.yml

We need to use - setup_remote_docker

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7.5
...
    steps:
      - checkout  
      - setup_remote_docker 

Don't use docker_layer_caching: true, if you only have the free tier available (see https://github.com/jonashackt/molecule-ansible-docker-vagrant/issues/5)!!

Now head over to CircleCI and have a look into the log. It should look green and somehow like this: https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/16 :

Use pipenv with CircleCI

As we already use pipenv with our local project setup and with TravisCI, CircleCI builds should also depend on this great dependency-lock supporting Python build tool.

And in the examples there's already the part we need instead of pip install XYZ commands:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7.5

    environment:
      EC2_REGION: eu-central-1

    working_directory: ~/molecule-ansible-docker-vagrant

    steps:
      - checkout
      - setup_remote_docker

      - run:
          name: Install all dependencies with pipenv (incl. python-dev installation for pip packages, that need to be build & have a Python.h present)
          command: |
            sudo apt-get install python-dev
            sudo pip install pipenv
            pipenv shell
            pipenv install

This gets us (maybe) into the following error:

An error occurred while installing psutil==5.6.5 ; sys_platform != 'win32' and sys_platform != 'cygwin'
...
'    psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory', '     #include <Python.h>', '              ^~~~~~~~~~', '    compilation terminated.', "    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1", '    ----------------------------------------', 'ERROR: Command errored out with exit status 1: /home/circleci/.local/share/virtualenvs/molecule-ansible-docker-vagrant-082rgMyr/bin/python3.7 -u -c \'import sys, setuptools, tokenize; sys.argv[0] = \'"\'"\'/tmp/pip-install-fecrj6wj/psutil/setup.py\'"\'"\'; __file__=\'"\'"\'/tmp/pip-install-fecrj6wj/psutil/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-iium4npj/install-record.txt --single-version-externally-managed --compile --install-headers /home/circleci/.local/share/virtualenvs/molecule-ansible-docker-vagrant-082rgMyr/include/site/python3.7/psutil Check the logs for full command output.']

This seems to be a knows issue, if python-dev package isn't available, no PIP packages could be build locally.

So let's add a sudo apt-get install python-dev to our .circleci/config.yml.

Now using pipenv, we can't simply activate the pipenv shell with a pipenv shell -> this will stop our CircleCI builds right in the middle (see this build)

The docs don't mention that one clearly enough - BUT otherwise the commands like molecule etc wont be able to find, see this build!

The solution is to run every command with a prefixed pipenv run - like pipenv run molecule test.

Schedule regular CircleCI builds with workflow triggers & cron

As the docs at https://circleci.com/docs/2.0/triggers/#scheduled-builds state, we need to add the workflows section inside our .circleci/config.yml to be able to use cron-scheduled builds. First we introduce a on-commit: workflow, which will just be triggered by a normal git commit/push. Nothing new here.

But the second workflow item weekly-schedule: gets us where we wanted to be: using the cron trigger we can configure to run our Molecule tests e.g. once on friday 04:45 PM every week:

workflows:
  version: 2
  on-commit:
    jobs:
      - build
  weekly-schedule:
    triggers:
      - schedule:
          # 17:55 every friday (see https://en.wikipedia.org/wiki/Cron)
          cron: "55 17 * * 5"
          filters:
            branches:
              only:
                - master
    jobs:
      - build

Now we should see our builds triggered by this cron regularly:

Just mind the UTC+2 timezone - for me, configuring 17:55 actually means, that my job will be scheduled to run at 19:55 - so don't think your config is wrong, maybe you're just in another timezone :)

Upgrade to Molecule v3

With Molecule 3.x our project and especially the molecule.yml needs some refinement. Here's an good overview.

Especially the lint sections and the scenario-name has to go - the latter is derived from the directory name and is thus not doubled anymore. The linting is now configured separately (see this commit).

Additionally now only Docker, Podman and Delegated are supposed to be core-providers. All other providers are now regarded as community-supported providers. For us as Molecule users this means, we need to install separate dependencies - since these providers now also have their own GitHub repo (see https://github.com/ansible-community/molecule-vagrant for example).

To use Vagrant, we need to add a new dependency to our Pipfile:

molecule-vagrant = "==0.2"
testinfra = "==4.1.0"

As we Testinfra is now also not longer installed by default, we should also add it explicitely.

Use Vagrant on GitHub Actions to execute Molecule

Well that one was on my list for a long time - but as Travis defacto laid off the OpenSource support, I switched to GitHub Actions.

Since GHA is easily able to run Vagrant (see https://github.com/jonashackt/vagrant-github-actions), we can simply use it inside our GHA workflow file vagrant.yml:

name: vagrant

on: [push]

jobs:
  molecule-vagrant-ubuntu:
    runs-on: macos-10.15

    steps:
    - uses: actions/[email protected]

    - name: Cache pipenv virtualenvs incl. all pip packages
      uses: actions/[email protected]
      with:
        path: ~/.local/share/virtualenvs
        key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}
        restore-keys: |
          ${{ runner.os }}-pipenv-

    - uses: actions/[email protected]
      with:
        python-version: '3.9'

    - name: Install required dependecies with pipenv
      run: |
        pip install pipenv
        pipenv install

    - name: Molecule testing GHA-locally with Vagrant Ubuntu Bionic
      run: |
        pipenv run molecule create --scenario-name vagrant-ubuntu
        pipenv run molecule converge --scenario-name vagrant-ubuntu
        pipenv run molecule verify --scenario-name vagrant-ubuntu
        pipenv run molecule destroy --scenario-name vagrant-ubuntu

Caching the pipenv virtualenv we also get relatively fast cycle times here.