Introduction to Nomad Box
An opinionated multi-provider multi-region Service Platform to easily and automatically handle development, packaging, testing, deployment and operations of Docker/GoLang/Java and legacy workloads. Should provide frictionless tools to ease in all stages of the application lifecycle.
Objective: Example platform designed to leverage the complete Hashicorp Suite of Products (and determine the gaps). Also as a learning exercise to compare/contrast against equivalent Kubernetes/Helm and Docker Swarm setup.
Architecture: Introduction to Nomad Box on Azure
Planned Technologies: Terraform, Consul, Nomad, Vault, Linkerd, Jaeger, Traefik, Hashi-UI, Vault-UI, Prometheus, BedrockDB, LXD, ZFS, Drone.io, Powershell.
NOTE: Only suitable in dev environments at the moment until I learn more Terraform, Consul, Nomad, Vault :P
Getting Started
Generate credentials using the Azure CLI. If you're not logged in, execute az login first. See the docs for more info.
Setup the first time:
leow$ make setup
Downloading Consul 0.8.1
cd bin && touch consul && rm consul* && \
curl -O "https://releases.hashicorp.com/consul/0.8.1/consul_0.8.1_darwin_amd64.zip" && \
unzip consul_0.8.1_darwin_amd64.zip
...
Setup the proper modules ..
cd ./terraform/env-development && time ../../bin/terraform get -update
Get: file:///Users/leow/Desktop/PROJECTS/DEVOPS/nomad-box/terraform/modules/foundation (update)
Get: file:///Users/leow/Desktop/PROJECTS/DEVOPS/nomad-box/terraform/modules/director (update)
Get: file:///Users/leow/Desktop/PROJECTS/DEVOPS/nomad-box/terraform/modules/worker (update)
Get: file:///Users/leow/Desktop/PROJECTS/DEVOPS/nomad-box/terraform/modules/experiment (update)
Optional: Setup different environment (defaults to env-development); if needed:
# leow$ export NOMAD_BOX_ENV=env-staging
Config: Copy the example terraform.tfvars.example file (it is inside the environment folder env-development) into terraform.tfvars. All statefile generated will be in the chosen environment
Credentials can be setup in Environment variable or inside the terraform.tfvars file
$ export ARM_SUBSCRIPTION_ID=abc-123-456
$ export ARM_CLIENT_ID=generated-app-id
$ export ARM_CLIENT_SECRET=generated-pass
$ export ARM_TENANT_ID=generated-tenant
Plan (after adjusting number of each type nodes in terraform.tfvars):
leow$ make plan
Planning ..
cd ./terraform/env-development && time ../../bin/terraform plan
...
azurerm_virtual_machine.foundation_node.2: Refreshing state... (ID: /subscriptions/3fdd09c8-f36c-417f-9d13-...hines/acme-nomad-dev-foundation-node-3)
No changes. Infrastructure is up-to-date.
Apply:
leow$ make apply
data.terraform_remote_state.nomadbox: Refreshing state...
azurerm_storage_blob.windows_cloudinit: Refreshing state... (ID: acme-nomad-dev-NomadBoxCloudInit.ps1)
...
azurerm_network_interface.windows_netif: Still destroying... (ID: /subscriptions/3fdd09c8-f36c-417f-9d13-...erfaces/acme-nomad-dev-windows-netif-1, 20s elapsed)
azurerm_network_interface.windows_netif: Destruction complete
Apply complete! Resources: 0 added, 0 changed, 3 destroyed.
Refresh:
leow$ make refresh
echo "Refreshing .."
Refreshing ..
cd ./terraform/env-development && time ../../bin/terraform refresh
azurerm_resource_group.director: Refreshing state... (ID: /subscriptions/3fdd09c8-f36c-�
...
Outputs:
bastion_pubip = [ a.b.c.d ]
director_internal_ips = []
director_public_fqdn = []
director_public_ips = []
...
virtual_network = acme-nomad-dev-foundation-vnet
worker_internal_ips = [
10.0.101.4
]
worker_resource_group_name = acme-nomad-dev-worker
Tunnel to Cluster:
leow$ make tunnel-up
Tunnel Up ..
sh -c "sshuttle -r [email protected] 10.0.0.0/16"
client: Connected.
Running Workloads (vis Nomad)
Example Nomad Jobs: see Example Jobs
Operations (-TODO-)
- Monitoring (Prometheus/Circonus)
- Tracing (Jaeger)
- Routing Functions: Canary, Blue/Green (Linkerd)
- Dashboard (Hashi-UI, Vault-UI)
- Autoscaling (Powershell Workflow w/ Control Theory)
- Load Balance (front Azure Traffic Manager/ AWS LB)
- Cluster Federation across providers
Security (-TODO-)
- Security Group + ACL
- Encryption between Nodes
- Leverage Vault fully
Destroy once cluster not needed:
leow$ make destroy
Removing infra ..
cd ./terraform/env-development && time ../../bin/terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
azurerm_resource_group.foundation: Refreshing state... (ID: /subscriptions/3fdd09c8-f36c-417f-9d13-...sourceGroups/acme-nomad-dev-foundation)```
...
data.terraform_remote_state.nomadbox: Destruction complete
Destroy complete! Resources: 4 destroyed.
Public Access
- Bastion Node: bastion_pubip
- Director Node (Traefik): director_public_ips
Internal Access (via tunnel)
- Foundation Nodes: 10.0.1.4, 10.0.2.4, 10.0.3.4 (until Consul has support for retry-join-azure)
- Director Nodes: director_internal_ips
- Worker Nodes: worker_internal_ips
- Experimental Nodes: experiment_internal_ips
Assumptions
- Internal Cluster Network CIDR 10.0.0.0/16
- sshuttle is installed
Inspiration
- cluster - https://github.com/mafonso/nomad-cluster
- konterraform - https://github.com/AppGyver/konterraform
- Turbo (great examples of multi-region clusters) https://github.com/jonmorehouse/turbo
- CoreOS Tectonic: https://github.com/coreos/tectonic-installer