Terraform ICP Provision Module

This terraform module can be used to deploy IBM Cloud Private on any supported infrastructure vendor. Tested on Ubuntu 16.04 and RHEL 7 on SoftLayer, VMware, AWS and Azure.

Pre-requisites

If the default SSH user is not the root user, the default user must have password-less sudo access.

Inputs

Outputs

• icp_public_key
  • The public key used for boot master to connect via ssh for cluster setup
• icp_private_key
  • The public key used for boot master to connect via ssh for cluster setup
• install_complete
  • Boolean value that is set to true when ICP installation process is completed
• icp_version
  • The ICP version that has been installed
• cluster_ips
  • List of IPs of the cluster

ICP Version specifications

The icp-inception field supports the format org/repo:version. ibmcom is the default organisation and icp-inception is the default repo, so if you're installing for example version 2.1.0.2 from Docker Hub it's sufficient to specify 2.1.0.2 as the version number.

It is also supported to install from private docker registries. In this case the format is: username:password@private_registry_server/org/repo:version.

So for exmaple

myuser:[email protected]/ibmcom/icp-inception:2.1.0.2

Remote Execution Hooks

It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. Currently, the following hooks are defined. Each hook must be a list of commands to run.

Local Execution Hooks

It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. Currently, the following hooks are defined. Each hook must be a single command to run.

These hooks support the execution of a single command or a local script. While this is a local-exec command, passing additional interpreter/environment parameters are not supported and therefore everything will be treated as a BASH script.

Host groups

In ICP version 2.1.0.2 the concept of host groups were introduced. This allows users to define groups of hosts by an arbritrary name that will be labelled such that they can be dedicated to particular workloads. You can read more about host groups on the KnowledgeCenter

To support this an input map called icp-host-groups were introduced, and this can be used to generate the relevant hosts file for the ICP installer. When using this field it should be used instead of the icp-master, icp-worker, etc fields.

Usage example

Using hooks

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"

    icp-master  = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
    icp-worker  = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
    icp-proxy   = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]

    icp-inception = "3.1.2"

    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }

    generate_key = true

    ssh_user       = "ubuntu"
    ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"

    hooks = {
      "cluster-preconfig" = [
        "echo This will run on all nodes",
        "echo And I can run as many commands",
        "echo as I want",
        "echo ....they will run in order"
      ]
      "postinstall" = [
        "echo Performing some post install backup",
        "${ var.postinstallbackup != "true" ? "" : "sudo chmod a+x /tmp/icp_backup.sh ; sudo /tmp/icp_backup.sh" }"
      ]
    }
}

Using HostGroups

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"

    # We will define master, management, worker, proxy and va (Vulnerability Assistant) as well as a custom db2 group
    icp-host-groups = {
      master     = "${openstack_compute_instance_v2.icpmaster.*.access_ip_v4}"
      management = "${openstack_compute_instance_v2.icpmanagement.*.access_ip_v4}"
      worker     = "${openstack_compute_instance_v2.icpworker.*.access_ip_v4}"
      proxy      = "${openstack_compute_instance_v2.icpproxy.*.access_ip_v4}"
      va         = "${openstack_compute_instance_v2.icpva.*.access_ip_v4}"

      hostgroup-db2        = "${openstack_compute_instance_v2.icpdb2.*.access_ip_v4}"
    }

    # We always have to specify a node to bootstrap the cluster. It can be any of the cluster nodes, or a separate node that has network access to the cluster.
    # We will use the first master node as boot node to run the ansible installer from
    boot-node   = "${openstack_compute_instance_v2.icpmaster.0.access_ip_v4}"

    icp-inception = "3.1.2"

    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }
}

Community Edition

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"

    icp-master  = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
    icp-worker  = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
    icp-proxy   = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]

    icp-inception = "3.1.2"

    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }

    generate_key = true

    ssh_user       = "ubuntu"
    ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"

}

Enterprise Edition

From NFS location

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"

    icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
    icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
    icp-proxy  = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]

    image_location = "nfs:fsf-lon0601b-fz.adn.networklayer.com:/IBM02S6275/data01/ibm-cloud-private-x86_64-2.1.0.1.tar.gz"

    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }

    generate_key = true

    ssh_user       = "ubuntu"
    ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"

}

From HTTP location

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"

    icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
    icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
    icp-proxy  = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]

    image_location = "https://myserver.host/myfiles/ibm-cloud-private-x86_64-2.1.0.1.tar.gz"


    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }

    generate_key = true

    ssh_user       = "ubuntu"
    ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"

}

Load tarball already existing on the boot node.

module "icpprovision" {
    source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.1.0"

    icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
    icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
    icp-proxy  = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]

    image_location = "/opt/ibm/cluster/images/ibm-cloud-private-x86_64-3.1.2.tar.gz"


    cluster_size  = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"

    icp_configuration = {
      "network_cidr"              = "192.168.0.0/16"
      "service_cluster_ip_range"  = "172.16.0.1/24"
      "default_admin_password"    = "My0wnPassw0rd"
    }

    generate_key = true

    ssh_user       = "ubuntu"
    ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"

}

There are several examples for different providers available from IBM Cloud Architecture Solutions Group github page

• ICP on SoftLayer
• ICP on VMware

ICP Configuration

Configuration file is generated from items in the following order

1. config.yaml shipped with ICP (if config_strategy = merge, else blank)
2. config.yaml specified in icp_config_file
3. key: value items specified in icp_configuration

Details on configuration items on ICP KnowledgeCenter

• ICP 1.2.0
• ICP 2.1.0

Scaling

The module supports automatic scaling of worker nodes. To scale simply add more nodes in the root resource supplying the icp-worker variable. You can see working examples for softlayer in the icp-softlayer repository

Please note, because of how terraform handles module dependencies and triggers, it is currently necessary to retrigger the scaling resource after scaling down nodes. If you don't do this ICP will continue to report inactive nodes until the next scaling event. To manually trigger the removal of deleted node, run these commands:

1. terraform taint --module icpprovision null_resource.icp-worker-scaler
2. terraform apply

Module Versions

As new use cases and best practices emerge code will be added to and changed the in module. Any changes in the code leads to a new release version. The module versions follow a semantic versioning scheme.

To avoid breaking existing templates which depends on the module it is recommended to add a version tag to the module source when pulling directly from git.

Versions and changes

3.1.1

• Fix issues with offline install
• Fix issue with config generation

3.1.0

• Allow cluster directory to be specified
• Allow other targets to be called from icp-inception
• Fix issues when owner of cluster files are something other than ssh_user
• Allow the cluster directory to be owned by arbitrary user after install
• Accept local files as valid location for image_location

3.0.8

• Fix docker install from yum repo for non-root user on RHEL

3.0.7

• Fix password length to comply with ICP 3.1.2 password rules

3.0.6

• Fix verbose installation option

3.0.5

• Fix docker install on ppc64le and s390x

3.0.4

• Fix default_admin_password output when provided as empty string in icp_config

3.0.3

• Fix blank icp_configuration["default_admin_password"] not generating random password

3.0.2

• Fix remote hook issue

3.0.1

• Fix local-hook issue

3.0.0

• Retire parallel image pull
• Retire unused variables (enterprise-edition, image_file, ssh_key, ssh_key_file)
• Default to generate strong default admin password if no password is specified
• Detect inception image when installing from offline tarball
• Rename icp-version variable to more descriptive icp-inception
• Overhaul of scaler function
• Add support for automatic installation of docker on RHEL and Centos
• Add support for downloading multi-arch images
• Fix wget output spamming during wget image download
• Include BATS test for testing scripts locally
• Rewrite image-load to take username and password separately when downloading from HTTP source

2.4.0

• Add support for local hooks
• Support specifying docker version when installing docker with apt (Ubuntu only)
• Ensure /opt/ibm is present before copying cluster skeleton

2.3.7

• Add retry logic to apt-get when installing prerequisites. Sometimes cloud-init or some other startup process can hold a lock on apt.

2.3.6

• Retry ssh from boot to cluster nodes when generating /etc/hosts entries. Fixes issues when some cluster nodes are provisioned substantially slower.
• Report exit code from docker when running ansible installer, rather than the last command in the pipelist (tee)

2.3.5

• Skip blanks when generating config.yaml as yaml.safe_dump exports them as '' which ansible installer doesn't like

2.3.4

• Create backup copy of original config.yaml to keep options and comments
• Support nested dictionaries when parsing icp_configuration to convert true/false strings to booleans

2.3.3

• Fix empty icp-master list issue when using icp-host-groups
• Fix issue with docker package install from nfs source
• Make docker check silent when docker is not installed

2.3.2

• Fix issues with terraform formatting of boolean values in config.yaml

2.3.1

• Fix issue with non-hostgroups installations not generating hosts files
• Fix boot-node not being optional in non-hostgroups installations
• Fix issue with boot node trying to ssh itself
• Install docker from repository if no other method selected (ubuntu only)
• Fix apt install issue for prerequisites

2.3.0

• Add full support for separate boot node
• Save icp install log output to /tmp/icp-install-log.txt
• Add option for verbosity on icp install log output

2.2.2

• Fix issues with email usernames when using private registry
• Fix passwords containing ':' when using private registry

2.2.1

• Fix scaler error when using hostgroups

2.2.0

• Added support for hostgroups
• Updated preprequisites scripts to avoid emediate failure in airgapped installations
• Include module outputs

2.1.0

• Added support for install hooks
• Added support for converged proxy nodes (combined master/proxy)
• Added support for private docker registry

2.0.1

• Fixed problem with worker scaler

2.0.0

• Added support for ssh bastion host
• Added support for dedicated management hosts
• Split up null_resource provisioners to increase granularity
• Added support for parallel load of EE images
• Various fixes

1.0.0

• Initial release