Kubash
Build, provision, initialize, add common components, interact and tear down a cluster PDQ.
Build production ready clusters using a variety of technologies along the way.
By default, this will build an image using packer, then rebase that image for your nodes. Then initialize them using kubeadm, and install charts using helm.
Oneliner installation
Install with one easy line:
curl -L git.io/kubash|bash
Get started by making the example:
kubash -n example yaml2cluster examples/example-cluster.yaml
ls -l clusters/example
Now build an image kubash build --target-os bionic1.20.1 where bionic is the OS and 1.20.1 is the K8S version
Then kubash provision -n example
And finally kubash -n example init
By default kubash is quiet unless an error is hit (though many of the
programs called by kubash might not be very quiet so there is still
lot's of noise at VERBOSITY=0). If you like
watching noisy output crank up the verbosity by adding a few v flags
(i.e. -vvvv) or specify the verbosity --verbosity 100 or export it as
a environment variable e.g.
export VERBOSITY=100
kubash output will be denoted by appending #s in front of various
verbosity levels e.g.
############# Kubash, by Josh Cox
Alternative pipelines
There are also alternative methods available for the steps, for coreos there is an alternative builder that merely downloads the official images. And for initializing the default is to directly initialize with kubeadm, or can be alternatively done through ansible with the openshift or kubespray or kubeadm2ha methods.
Other provisioning beyond KVM/qemu is also being looked at, suggestions welcome in the issues. Keep in mind this started life as a ten line script of me just trying to duplicate the official instructions after both the kubespray and openshift playbooks went sideways on me and I determined I needed to refamiliarize myself with spinning up a cluster with another method. Somewhere along the way I came across this google doc
And I decided to combine all of the above into a unified forkable pipeline that automates the entire proces of building the images using a similar method to the one I used for CoreOS, using one of the initialization methods in a very repeatable way. And throw some of the additional common components I tend to add to every cluster.
Provisioners
KVM/qemu is used first and foremost because it is standard and built into Linux itself. I intend to add virtualbox, vmware, etc builders and provisioners in the future.
Builders
Right now you can build these OSs as your base image:
- Ubuntu
- Debian
- Centos
- CoreOS
Packer will build ubuntu, debian, and centos. And there is also a basic downloader for the CoreOS images.
Initializers
Although this script can just utilize ssh to run the kubeadm commands directly on the VMs, you can optionally use kubespray or openshift's ansible playbooks instead, I have had various issues with both and that's why i wrote this script so I can choose amongst a few different methods in my regular daily builds (I'm the sort of guy who likes to spin up clusters while sipping my morning coffee).
Configs
By default there is now a .kubash folder in your home directory. Inside this directory is a folder called clusters, make example in the .kubash directory will build the default example cluster.
Usage
This script automates the setup and maintenance of a kubernetes cluster
kubash -n clustername COMMAND
Commands:
yaml2cluster - Build a cluster directory from a yaml cluster file
json2cluster - Build a cluster directory from a json cluster file
build - Build a base image
build-all - Build all the base images in parallel
provision - Provision individual nodes
init - Initialize the cluster
reset - Reset the cluster by running kubeadm reset on all the hosts
decommission - Tear down the cluster and decommission nodes
copy - copy the built images to the provisioning hosts
ping - Perform ansible ping to all hosts
auto - Full auto will provision and initialize all hosts
hostnamer - Will rehostname all the hosts
refresh - will search for all the hosts using the appropriate method and recreate hosts.csv
Options
These options are parsed using GNU getopt
options:
-h --help - Print usage
-n --clustername - work with a named cluster (or by default it will use a cluster name of 'default')
-c --csv FILE - Set the csv file to be parsed
--parallel NUMBER_OF_THREADS - set the number of parallel jobs for tasks that support it
-v --verbose - Increase the verbosity (can set multiple times to incrementally increase e.g. `-vvvv`
--verbosity NUMBER - or you can set the verbosity directly
--debug - adds the debug flag
--oidc - enable the oidc auths
There is an example csv file in this repo which shows how to compose this file
Debugging
First try kubash COMMAND --help
See the debugging page for more
Interactive Mode
kubash -- alone will invoke an interactive shell
see the Interactive Mode documentation
Ingress
There are a few shortcuts for installing ingress into the cluster
Parallel jobs
To set the number of concurrent jobs export PARALLEL_JOBS e.g.
export PARALLEL_JOBS=10
GNU Parallel
This project takes advantage of GNU Parallel gnu parallel and so should you, for more info see:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
Pseudo-etymology
"The whole kubash" - a bastardization of "The whole kit and kaboodle", "The whole shebang" (#!) , kubernetes, and bash. The meaning here is that kubash is taking on everything else that kubeadm considers 'out of scope'. From building images, provisioning, to usage of kubeadm itself, on through to a quick shell for interacting with the running cluster, and finally decommissioning the cluster.
Troubleshooting
Sometimes your router will give new addresses to the MAC addresses and the kubash host will have stale arp table entries, flush them all:
ip -s -s neigh flush all
Another issue is that the kubash user will have conflicting known_hosts entries for ssh, move your known_hosts file temporarily to test.
Packer variables
variables starting with K8S KUBASH or PACKER PKR_VAR are automatically passed through. See packer.bash for the actual greps.
Registry Mirrorsf
There are four registry variables that can be defined:
$K8S_REGISTRY_MIRROR_HUB
$K8S_REGISTRY_MIRROR_QUAY
$K8S_REGISTRY_MIRROR_GRC
$K8S_REGISTRY_MIRROR_K8S
The first of which must be defined for the others to work:
$K8S_REGISTRY_MIRROR_HUB