KubeStitch

An easy to use tool for stitching together customized Kubernetes cluster environments that run locally the same as they do in the cloud.

About

A default deployment of Kubernetes is not extremely useful. Usually, one needs to layer several stacks of technology on top of Kubernetes before even beginning to think about pipelining specific workloads to the cluster. To start extracting real value from this powerful new platform one needs to stitch together their base environment. KubeStitch attempts to help with this task by making it much easier to vet out technical solutions for the base deployment level of a Kuberentes cluster. It squarely targets that middle section between the custom workloads and the kubernetes cluster deployment itself.

The diagram shows a few elements which may need deployment to your clusters. These can be done in one Istio operator driven deployment, as individual stacks, starting with ArgoCD which then gets used to deploy each stack as an 'App of Apps' GitOps approach, or any other method which suits your fancy. KubeStitch can help aid your efforts.

Goals

The short goal for this project would be to narrow the gap in feature parity between what your local environment looks like versus your deployment environment. One can use kubestitch to quickly bring up and tear down local kubernetes environments for testing and vetting out Kubernetes solutions that then get deployed remotely with little extra code refactoring. Additionally, multiple Kubernetes clustering technologies can be swapped out relatively quickly and easily using a single file with few variables.

Requirements

This project currently uses standard bash scripts, make, and docker. It has been tested on Linux and Mac. The following should be available on the system running these scripts.

• bash
• docker

Where required, all other stand-alone cli tools will be installed to a ./.local/ path within this folder to ensure portability.

If you ever run a git pull to update this project locally, it would be safe to run a make clean to ensure that all the latest updates are pulled down when running make cluster.

NOTE Mac users should be aware that while the dns-proxy-server will spin up with the correct dns internal zone forwarding configured to the correct ingress internal IP, it will not work from the mac automatically. First you will need to reconfigure your DNS resolver to point to 127.0.0.1. I could not figure out a way to make this work automatically due to how mdnsResponder takes over the DNS lookup process on Macs. I welcome pull requests to help resolve this automation snafu. More work in this area can be found in the Mac Issues section of this document.

Installing

This project is only a makefile with some scripts. So clone the repo, cd to it, then run make. If you want to run it from anywhere on your system then you will have to create an alias for the command like this from the cloned repo directory.

alias kubestitch="make --no-print-directory -C $(pwd)"

Then you can run any tasks using the kubestitch command.

kubestitch show
kubestitch PROFILE=monitoring show

# Note that the placement of override env vars in the command does not matter
kubestitch cluster PROFILE=vault

The rest of this document will use make in any examples.

Usage

Standard usage is pretty simple. After cloning this repo you can start a default kind based cluster for the current PROFILE (default=default) with one command.

make cluster

This aggregates several tasks which are broken down further below.

# Show tasks
make

# Install dependencies
make deps

# Start a local kind cluster and create a local kube
make cluster/start

# Perform a default deployment of helmfiles for the cluster defined in the profile (default: cicd)
make sync

When you are done you can destroy the cluster you just created.

make cluster/stop

Clusters

Before diving into profiles, it is important to be aware that each cluster that gets created will create its own configuration file within the ./.local/ path in the form of kube.<clustername>.conf (unless the KUBE_CONFIG environment variable is overwritten). This ensures your personal user kubernetes config file does not get polluted with KubeStitch testing cluster detritus.

This also means that, by default, you will not be able to interface with the cluster without some additional work. The simple way around this is to just point your 'KUBECONFIG' env var to the config file that gets generated. If you are tinkering with multiple versions of Kubernetes, then it may be useful to also set an alias for the kubectl binary that gets downloaded as well.

Both tasks can be done easily with the following single command after the cluster has been created (note that we are sourcing the output of make config as a command using backticks!).

`make config`

You only need to do this once for any cluster you are working on (per console session). If you recreate the cluster in your journey (done easily via make cluster) this file may get recreated but it shouldn't matter so much.

If you are changing between versions of Kubernetes using this framework you will have to clear out the kubectl binary when changing versions (between cluster builds) with make clean. Just ensure you also rerun the deps task to get the correct kubectl binary version afterwards.

Profiles

This project was originally created to support multiple clusters and teams. I've since pulled it back to simply targeting 'profiles'. This means all top level settings that you may want to overwrite are able to be put in a single file: ./profiles/<profilename>.env. To see the current profile, helmfile environment, and some additional variables, use make show.

To create a whole new target environment you can copy and modify the ./profiles/default.env to ./profiles/<newprofile>.env then pass in PROFILE=<newprofile> to all make commands (or export it at the start of your session). If you need a separate environment or helmfile cluster deployment then further files will need to be created to accommodate.

NOTE PROFILE is for using different cluster types and configurations. Testing the same set of helm charts to both k3d and kind (or any other target for that matter) might be facilitated by using a profile. An ENVIRONMENT is what you would use to create helmfile deployment environment configuration. ENVIRONMENT is always 'default' unless manually passed in and is only used in the * tasks. You can set ENVIRONMENT within the profile definition or overwrite it when calling the tasks via the command line. The istio example uses an Environment to define a set of istio specific helm settings to deploy.

## Launch a local k3d based kube cluster
make deps cluster/start PROFILE=k3d

## Launch a local kind based kube cluster
make deps cluster/start PROFILE=default

## Look at your first cluster
export KUBECONFIG=`make .kube/config/file PROFILE=k3d`
kubectl get nodes

## And your second one too
export KUBECONFIG=`make .kube/config/file PROFILE=default`
kubectl get nodes

## Destroy both clusters
make cluster/stop PROFILE=k3d
make cluster/stop  ## Default environment is 'default'

NOTE While you can run multiple clusters at once that really wasn't what this project was meant for and you will likely run into port or other conflicts. It is best to run each cluster then destroy it before changing and using another profile.

Profile - default

File: ./profiles/default.env

This profile is what we use for default deployments. It includes;

• A 2 node kind cluster running
• Kubernetes 1.18.2
• The calico CNI
• MetalLB

After the cluster starts up and you run the dnsforward/start task the following urls will be available.

• http://traefik.int.micro.svc

Profile - monitoring

File: ./profiles/monitoring.env

This profile is the same as the default profile but allows for the prometheus operator to also be deployed. After the cluster starts up and you run the dnsforward/start task the following urls will be available.

• http://traefik.int.micro.svc
• http://grafana.int.micro.svc
• http://alertmanager.int.micro.svc
• http://prometheus.int.micro.svc

The grafana site default login is admin/prom-operator

PROFILE - k3d

File: ./profiles/k3d.env

• A 2 node k3d cluster
• Kubernetes 1.18.3
• Builtin k3d loadbalancer

This is a k3d cluster that is similar to the kind environment

PROFILE - istio

File: ./profiles/istio.env

Here is an example environment running istio on a kind cluster. It includes;

• A separate 'istio' profile (./profiles/istio.env)
• An additional plugin to include istio specific commands (found in inc/makefile.istio)
• An istioctl based istio operator deployment
• Some istio tasks for monitoring ingress and such
• A helmfile based deployment of bookinfo:
  • Not a bad example of converting a straight yaml file to helmfile using the raw chart (as a crude shortcut)
  • Exhibits helmfile dependency chaining
  • Uses a local custom namespace chart to also enable the istio sidecar injection label upon deployment

PROFILE - vault

File: ./profiles/vault.env

This is another proof of concept environment that spins up a kind cluster running Hashicorp vault with a consul backend. It also includes installation of some dependencies for testing things out as well as vault-sync to test out seeding a base deployment via cli. This is a work in progress on how one might use mostly declarative configuration for a vault integrated kubernetes cluster.

You can access both http://consul.int.micro.svc and http://vault.int.micro.svc to immediately start exploring these cool products after bringing up the cluster.

If you use this profile the vault service will be provisioned via an ingress address which can be accessed using the vault client and some environment variables. To export these variables into your session use the following command:

make dnsforward/start
`make show/vault`
./.local/bin/vault status

NOTE - This assumes that dns lookups for your cluster works properly, dnsforward tasks currently only work on Linux hosts. Alternatively, you can update the deployment to expose the service via Loadbalancer IP as well and use the IP address instead.

PROFILE - vaultha

File: ./profiles/vaultha.env

This changes a few variables around so that a 3 node kind cluster is deployed and the vault chart gets deployed in ha mode instead of dev mode. The vault deployment is uninitialized and will need more love to get working correctly.

PROFILE - localstack

File: ./profiles/localstack.env

This profile is for setting up the venerable localstack AWS API simulation environment in kubernetes. The deployment was a 10 minute effort proof of concept done by converting the project's docker-compose file via kompose then plugging in the output into a custom helmfile.

This can be a great way to vet out terraform manifests or test out AWS based pipelines.

PROFILE - argocd

File: ./profiles/argocd.env

Create a local argocd cluster for testing out GitOps. Includes customization to allow for helmfile use in the application definitions.

To access the GUI interface (https://argocd.int.micro.svc/) the default ID is admin and the password can be retrieved by running make argocd/get/password (typically ends up being the argocd-server pod name)

NOTE: I'm still working on the helmfile argocd repository task runner. This is just a base environment and nothing more.

PROFILE - homeassist

Performs a local install/uninstall of a single node k3s cluster for use in a bare metal home assistant deployment. This assumes the local machine is Linux ready and uses sudo rights to setup the service. This profile is custom built for my own environment which includes an NFS storage backend and should not be used without modifications.

File: ./profiles/homeassist.env

NOTE: To do the same deployment locally via kind use the 'kind_homeassist' profile instead.

PROFILE - testing

File: ./profiles/testing.env

I use this to work on charts before they are published for a new release version. This uses the testing environment with overrides to the archetype chart source so that it uses the direct git repo master branch instead of the released version.

Tasksets

Sometimes there are additional commands required to vet out a particular solution. To accommodate for this need, we can add a new makefile directly in the 'inc' folder as ./inc/makefile.<taskset> and then add the <taskset> into your profile's ADDITIONAL_TASKSETS variable (space delimited if there is more than one). This will source in your new script tasks automatically when the profile is used.

You can see the tasksets which are available by listing the files within the 'inc' folder of this repo. Alternatively, run make show/tasksets to list out all those that are available.

Most tasksets are cross platform and will download and then run binaries specific to your platform (Linux/Mac only at this time).

NOTE: The Lens taskset is not cross platform so it is excluded from the default profile. You can insert this or any other non-profile specific taskset by adding CUSTOM_TASKSETS=lens to your make commands.

The following tasksets are loaded by default and are considered essential for KubeStitch to operate.

• helm
• kube
• common

To get help for loaded tasks run make help. This will show all non-hidden tasks. There are quite a few more hidden tasks which may be useful as well. To list these tasks, run make help/all and grep the output for what you might be looking to do.

# show all tasks, including those that are hidden and search for those related to ingress
❯ make help/all | grep ingress
.cluster/ingress               Start cluster ingress
.kube/ingress/hosts            Show ingress hosts

Provider Tasksets

A provider taskset is special as it is a specific taskset for provisioning kubernetes clusters. These are in the same location as the other tasksets but with a different naming convention, ./inc/makefile.provider.<taskset>. Only one provider taskset should be loaded at a time. Thus far kind and k3d are the most reliable and tested provider tasksets but I also include a minikube provider from earlier testing as well as the k3s taskset for my home server deployment.

NOTE: I abandoned minikube earlier on for a number of reasons but it would be feasible to fix this up and use minikube as well if that's your thing.

If you choose to add a new provider ensure that they include all the base tasks found in the other makefile.cluster.* files and you will find that its pretty easy to automatically install dependencies and customize this thing to your needs.

Helmfile

Helmfile is used extensively to declaratively stitch together deployments from external helm charts, my own monochart (called archetype), and the raw helm chart. Common repositories and default helmfile values are sourced into the various helmfiles using a common block at the top of each helmfile.

Helmfile uses 'environments' to help breakdown multiple configuration paths more effectively.

Environments

Environments are defined in config/environments.yaml. This points to an individual values file that contains any default values first. Optional override values can then be manually defined afterwards. Additionally, each relevant helmfile includes an optional override value file that can be used to rewrite the helm deployment for each environment if so desired. Any override files should be dropped in the environment folder in this format:

./config/<environment>/<chart>.override.yaml

To target another environment for helmfile follow a few extra steps.

1. Modify config/environments.yaml to add a new environment (copy/paste the default one to start if you like)
2. Minor changes can be made right in the environments.yaml file beneath the default values
3. For larger changes, create/update any override files as needed in config/<your_environment>/

Then, whenever you are targeting a specific environment, include ENVIRONMENT=<your_environment>. Optionally, you can create a new profile and set ENVIRONMENT within it instead.

I try to keep all settings in the default helmfile environment yaml definition and overwrite their installation in the environment level file when I need to do so. In this example, I disable monitoring elements from being deployed to the default environment even though the same setting in the ./config/default/values.yaml file is enabled.

# ./config/environments.yaml
environments:
  default:
    values:
    - ../config/default/values.yaml
    - prometheusoperator:
        enabled: false

As you can see, I purposefully set the prometheus operator deployment as disabled in the default environment. This is because one usually does not want an entire monitoring environment when doing local testing. But many helmfile stacks include monitors that I may otherwise need to deploy that have dependencies upon CRDs of the Prometheus Operator. To work around this chicken/egg scenario we simply disable the monitoring stack at the environment level and use that variable when deploying monitoring elements in the various stacks. The same type of logic is done with the 'ingress' stack so we can deploy several stacks and simply not deploy ingress by ensuring that ingress.enabled=false if we so desire.

NOTE: The approach I take is that all stacks are default enabled unless the environment overrides say otherwise. This does not mean that all stacks are deployed by default, this is what the cluster helmfiles are for. But we can add/remove individual stacks this way.

This effectively makes the ../config/default/values.yaml our default settings with anything under ./config/environments.yaml being the overrides. I source these settings in for all my helmfiles which allows me to update helm chart versions and sources from one values file. If I want to test a new chart I can create another environment with the value being overwritten in ./config/environments.yaml while retaining all other chart settings.

NOTE: In the past I've utilized many environment variables for these kinds of settings. While it is tempting to do so I recommend against this style of chart authoring as it makes for more complex pipelines and a less declarative approach to deployments.

If you wish to see the applied values after all these layers are merged, we can use a specially crafted helmfile stack to display the rendered values.

make build STACK=values PROFILE=<profile you wish to see merged values for>

Helmfile Stacks

For lack of better terminology I use 'stack' to define a set of helm charts that I've stitched together as a deployable unit. Typically this is a handful of charts all targeting a single namespace by the same name. I attempt to create helmfiles in a manner which can be deployed individually but there may be some cross dependencies for more complex stacks (like if I use cert manager CRDs for instance).

A set of stacks can be applied, in order, via a single helmfile. This is how I deploy whole cluster configurations declaratively. Below you can see the default cicd cluster helmfile. As it is simply another helmfile we keep the cluster helmfile definitions right alongside the rest of the stacks.

#/helmfiles/helmfile.cluster.cicd.yaml
---
bases:
- ../config/environments.yaml
---

helmfiles:
- ../helmfiles/helmfile.traefik.yaml
- ../helmfiles/helmfile.cert-manager.yaml
- ../helmfiles/helmfile.security.yaml
- ../helmfiles/helmfile.metricsserver.yaml

What makes this fun is that I have a whole other profile called 'vault' that creates its own cluster (aptly named 'vault') that I put together simply for the convenience of bringing up a full cluster configured with consul and vault in a single command. But I don't need to use the vault profile to run consul and vault. We can also add individual helmfile stacks or the entire cluster helmfile stack definition at any time to any cluster while in any profile.

# Use the default profile
unset PROFILE

# Setup a default cluster
make cluster

# Install only the consul stack
make sync STACK=consul

# Or apply the cluster stack entirely outside of the vault profile (note that consul is already applied and will not be applied again even though it is listed in the cluster.vault helmfile)
make sync STACK=cluster.vault 

# Now remove just consul for the hell of it (likely breaking vault though)
make destroy STACK=consul

This is a powerful way to quickly work on various deployments and allows for layering of stacks to get the results you are looking for in an iterative manner.

NOTE 1 Sometimes you may have to wait for the initial cluster to fully come up before running the apply for the entire cluster.

NOTE 2 Because many of the lookups involved with this framework are just simple scripts and I found no other good way to look up the helmfile values in a post rendered state, I use a special helmfile that is never meant to be deployed that is called lookup.placeholder.yaml. This allows me to render values based on environment using the helmfile build command. Then the output is parsed for the values I need. I use this workaround and a hidden task to render out ingress internal dns zone information for the dns proxy task for instance.

Examples

Here are a few example use cases worth looking over.

Example 1 - Local Istio

Here is an example of using a profile to start up a local istio cluster. Istio has a moderately more complex deployment path than a standard helm chart so this is a good example for showing how KubeStitch can be flexible. The profile definition is copied from the default profile and renamed with a few changes.

KUBE_PROVIDER=kind
CLUSTER=istio
KUBE_VERSION=1.18.2
KIND_IMAGE_POSTFIX=@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
ENVIRONMENT=istio
ADDITIONAL_TASKSETS=istio k9s metallb

Because istio will want to run its own ingress (and egress) gateways within the istio-system namespace we create another helmfile profile which looks very similar to the default profile:

  istio:
    missingFileHandler: Debug
    secrets:
    - ../config/{{ .Environment.Name }}/secrets.yaml
    values:
    - ../config/default/values.yaml
    - ../config/{{ .Environment.Name }}/values.yaml
    - prometheusoperator:
        enabled: false
    - ingress:
        internal:
          namespace: istio-system
    - traefik:
        enabled: false

We don't actually use any secrets or even a specific environment values.yaml file but we define them anyways (in case we ever want to drop overrides for chart locations, versions, or anything else that veers from the ../config/default/values.yaml definitions). We then put in a few overrides that will prevent deployment of the prometheus-operator and traefik (though the default cluster helmfile stack for the 'istio' cluster defined in the profile does not have these defined anyway). Finally, the ingress internal namespace is overwritten from the default values in here as well. We use this for the dnsforwarding tasks.

The istio operator chart wants to create its own namespace via its chart template so we let it do so via the istio-operator chart by not defining a namespace within the helmfile release. We also deploy the demo profile for istio using the CRD definition and a simple raw helm chart in the helmfile.istio.yaml file.

# Ensure we get the correct profile for the remaining commands
export PROFILE=istio

# Start the cluster
make cluster

# If you are on linux you can also start forwarding dns
make dnsforward/start

This installs the istio operator, istio itself, and some dashboard ingress definitions for these things.

NOTE Because we are using the operator, it can take a few minutes for the istio deployment to get rolled out.

If you want to see more than just a base deployment deploy the bookinfo helmfile stack as well.

# Deploy bookinfo demo app if you like
make sync STACK=bookinfo

At this point you should be able to go to the example bookinfo microservice deployment by visiting http://bookinfo.int.micro.svc/productpage from your local machine. Open the kiali dashboard at http://kiali.int.micro.svc (login: admin/admin) to see the istio magic at work.

You can then screw around with the cluster to your heart's content.

# configure local kubectl to access your cluster
export KUBECONFIG=`make .kube/config/file`

# view the virtual services (ingresses) for the dashboards
kubectl -n istio-system get virtualservice

To clean things up.

make dnsforward/stop cluster/stop
unset PROFILE KUBECONFIG

If k3d is more your style then you can repeat this entire set of directions with another profile I setup and tested in a few minutes after doing this with kind. Just use the istio-k3d profile instead!

Example 2 - K3S with HomeAssistant

This is a somewhat unique example. Basically I used this framework for a bare metal k3s cluster at home for home assistant. I created another kubernetes provider specifically for k3s (inc/makefile.cluster.k3s) along with a profile to use it (profiles/profile.homeassist.env). With this profile and provider I overwrite the default location of the KUBE_CONFIG so that after installation of the cluster elements the config is automatically updated for my standard account. This particular provider does require sudo rights.

A neat trick I use here is to set 'CLUSTER' to 'haas' in the profile and create a helmfile.cluster.haas.yaml file for use in deploying all stacks to this deployment even though it is not local.

export PROFILE=homeassist
make cluster

The k3s provider I created specifically disables many of the default deployment settings of k8s, specifically the default 'local-path' storage provider. I later use NFS mounts in my home setup using the nfs-client-provider. This gets installed via helmfile like 99% of hte cluster configuration.

While I was testing this out on an old Shuttle PC I found out that some memory sticks were bad. As I was running memtest86+ against the server (which takes forever and a day) I quickly setup another profile called 'kind_homeassist' so I could continue vetting out my kubernetes deployment stacks. In a few minutes I had the same deployment via the kind provider up and running.

export PROFILE=kind_homeassist
make cluster
make sync STACK=homeassistant

You may notice that I use the cicd cluster for the local kind install as I know it works locally and since I'm not using nfs mounts or anything like that in my docker based cluster it makes sense for me to use a default baseline cluster and simply manually add the stacks in that I'm testing afterwards.

Because I'm using some localized IP addresses for both the nfs provisioner chart and traefik, I created another helmfile profile in 'config/environments.yaml' with these override settings in place. I do not yet know how to extract the extrapolated values yaml from helmfile so for now I also have to put these values in the 'config/homeassist/values.yaml' file for the Metallb deployment scripts to locate and for dnsforward tasks to use.

NOTE: In this setup I run the KubeStitch from my server then access it remotely via kubectl. To do this I have to copy over the config file that gets generated to my workstation scp zloeber@servername:~/..kube/config ~/..kube/config then edit it to replace 127.0.0.1 with the IP address of the server.

Example 3 - Deployment As an Artifact

I've done a bit of a proof of concept around initializing a cluster's deployed stacks and figured I'd put it as a third example of how you can use this framework. The idea is that I'd like to turn the entire cluster deployment with helmfiles and definitions into a localized immutable artifact. This is because sometimes external helm charts change without notice or helm chart versioning being done.

Also, I just wanted to see what it would take to do a single push deployment to a cluster that then runs all the helmfile logic for that cluster. Turns out it is not so hard. To run such a thing for the default profile we just create a job that uses a dockerfile image created from this repo and set environment variables for ENVIRONMENT and CLUSTER. As long as we run it in a namespace that has cluster-admin rights it will run without issues. I created a helmfile to do this task.

# Start the base cluster without any helm charts being synced
make cluster/start

# Sync the clusterinit stack
make sync STACK=clusterinit

After a few minutes the local cluster will have traefik, certmanager, and rbacmanager installed (and you should see some successful jobs having been run in the cluster-init namespace).

The docker image itself does the repo and chart dep updates. When the image runs on the cluster we skip this step by using the --skip-deps flag in the helmfile sync command.

This is a proof of concept only. Next steps would be to make the process more secure by using a versioned image in the job (not latest) and an additional task should be run after the helmfile sync completes to selectively remove the role-binding on the namespace's default service account.

Secrets

Any secrets should be put into your environment for use in tasksets that require them. For this I personally use direnv with a local .envrc file.

## Example direnv file for exporting a gitlab token used to login to the gitlab cli
export GITLAB_TOKEN=<your token>

Lens (GUI Console)

Lens is a particularly useful GUI app for rucking about in Kubernetes. It is also cross-platform. If you are running a linux host you can use this framework to automatically download the app and access clusters with it. I've added some scriptwork to automatically clear the lens clusters and add new configuration in a few commands. Here is how it works.

# First ensure that the taskset gets loaded (it should not be in any profile by default)
export CUSTOM_TASKSETS=lens

# Download the lens AppImage and run it at least once then close out of it
make deps lens

# Assuming your cluster has been started already, you can add it to the lens config automatically then restart the app.
# By default, this will first clear any added clusters in the configuration (localized in the ./.local/Lens/lens-cluster-store.json file)
make lens/addcluster lens

The running cluster will be the first one in the list (at the upper left). Lens stores entire cluster configurations in its backend store so you will need to do this each time you rebuild a cluster and wish to use this app to manage it.

NOTE The Lens app is cross platform but the taskset is targeted towards linux hosts only currently as the AppImage format is a single download and able to be automated without root access. For a cross-platform console dashboard you can also use the k9s taskset. Otherwise you will have to manually download the appropriate version of Lens for your platform and remove/add the cluster configurations manually at this time.

Troubleshooting

To enable helm debugging pass in DEBUG_MODE=on to any command.

Additional Tips

• cluster/start task will always first try to destroy the cluster before starting it.
• cluster/start task always spins up a bare deployment (no helmfile stacks applied)
• cluster task runs cluster/start and sync tasks. Unless STACK=<stack> is specified this defaults to syncing the cluster named helmfile stack (helmfiles/helmfile.cluster.<profile name>.yaml)
• sync will default to STACK=cluster.$(CLUSTER)
• The above information means make cluster/start sync will always recreate a cluster from scratch then install the default cluster helmfile stack of charts for the current profile's cluster.
• All of the commands above are condensed into make cluster
• There is a ton of extra 'stuff' in this repo that still needs to be cleaned out or revisited, not all files serve a purpose (yet)
• Along the same lines as the prior statement, there are a ton eof 'helmfiles' in the wip folder that worked with env vars at one point. I'll slowly move these out of wip when I'm able to do so or the need comes up.
• MetalLB is used when a loadbalancer is required.
• Currently the loadbalancer deployment will use IP addresses between 172.17.0.100 and 172.17.0.110. This is the bridge IP subnet of docker on my workstation. You can modify this range in the config file within ./deploy/.metallb/metallb-config.yaml. This is/was only for kind and is now automatically determined when metallb is deployed to the cluster. You can still override this in the ./config/environment.default.yaml file by adding stacks.ingress.internalLBSubnet and assigning it a CIDR subnet instead. Otherwise it grabs the DOCKER_NETWORK subnet and replaces the last digits with 1.0/24 (in my case it now ends up being 172.21.1.0/24 since I kept screwing around with the bridge network...whoops.)
• Currently I source in all used repositories regardless if they are used or not, this slows down initial syncing and probably can be improved upon somehow. If you have already run make sync once and thus have remote chart repositories already synced up, you can more quickly itterate over local helmfile changes by switching over to using make charts STACK=<yourstack> instead. The helmfile charts subcommand is meant for offline chart deployments.
• The dnsforward/* tasks are a bit janky as it is really hard to account for all possible scenarios. Generically, only use dnsforward tasks for local cluster testing (and make any additional forwarding updates/changes via the web interface if the logic is not working for you).

Known Issues

I just want to note that currently, this framework will require all repositories for all helmfiles you wish to use. This list can grow quite quickly. It would not be very hard to have a per-environment configuration of each repo list. You will have to update most of the base helmfiles to accomodate though.

Minikube on Linux

I actively avoid minikube on Linux as it has given me nothing but problems. If you do happen to really want to use minikube on Linux here are some notes.

If you are running Ubuntu 20.04 here is a snippet to get docker-ce installed:

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt install docker-ce
sudo usermod -aG docker ${USER}

And a snippet to get podman installed:

source /etc/os-release
sudo sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O- | sudo apt-key add -
sudo apt update && sudo apt install -y podman

That is all I have for you, good luck.

Advanced Usage

Using Kubestich fully means you are likely needing to add/remove/change up your clusters frequently. This is the generic process of how to do so from my experiences.

1. You will likely onboard a new helm chart (usually converting a helm chart to a helmfile)
2. You may bundle several helmfiles into a singlular helmfile chart for a cluster deployment
3. If you get this far you may then also look to 'productionalize' the deployment to work within multiple environments or even ingress zones.

This all starts with converting a helm chart to a helmfile though.

Helm to Helmfile

Typically the development process starts with a need for a particular helm chart deployment. This deployment will typically need to be stripped of;

• default ingress definitions (we use our own to managed all of them from the ingress controller on down)
• subcharts (these should each be turned into a helmfile deployment)
• Any namespace generation
• default metrics collection or monitoring configurations (again, we will manage that on our own across the whole cluser)

To convert a typical helm chart a few templates have been dropped into the helmfiles/templates folder. Copy and edit one of these into the helmfiles directory as helmfile.chartname.yaml. Simply replace the 'chartname' with your chart and update the default details (and likely values). Additional noodling in the origin helm chart will need to be done to isolate any service names used so they can be used in your own ingress definitions (if the chart requires one that is).

At this stage, a well authored helmfile chart can technically stand on its own and be applied to any current Kubernetes cluster. But more should be done to future proof the deployment a bit and allow for more centralized updates and puzzle piecing down the line.

Config File Updates

Some files you should update at this point:

If all is well you should be able to apply your chart via kubestitch to any current cluster:

make sync STACK=chartname

Troubleshooting

This set of Makefiles, scripts, glue, and prayers works well for me but may be problematic for some. Here are specific notes which may or may not help you fix issues you are running into.

Mac Issues

There are two issues I've run into while getting this to work properly on a mac. They are largely due to difficulties with the networking stack of the most recent releases of OSX. These issues fall into 2 categories, DNS resolution and routable interface handling in hyperkit (the mac virtualization layer).

DNS

TBD

Virtual Network Interfaces

The best article on this particular issue can be read here. Be warned that getting this working requires additional installation of external software and other tricks to make function. It is almost easier to just run KubeStitch from a local Ubuntu virtual machine in virtualbox or similar hypervisor.

The overall steps to get things working are as follows:

brew bundle --file ./config/osx/Brewfile
osascript -e 'quit app "Docker"'
./config/osx/docker-tuntap-osx/sbin/docker_tap_install.sh
./config/osx/docker_tap_up.sh
sudo route -v add -net 172.18.0.1 -netmask 255.255.0.0 10.0.75.2

Once this has been done technically you should be able to bring up things as normal and access them via the Metallb provisioned loadbalancer IP that gets assigned to the traefik loadbalancer service. I do not believe this accounts for being connected to a VPN though.

Why Makefiles?

Great question! One which I sometimes struggle to answer. There are dozens of other taskers out there but make is probably one of the older ones that is well supported across a number of different systems. I personally use it when I might use bash scripts otherwise. It allows me to quickly view the commands being run (if I've not purposefully hidden them with a well placed prepended @ symbol). Plus I'm sort of used to slinging makefiles so why not? Besides, most repos worth their salt have at least one of these to bootstrap things in some manner so it doesn't hurt to get to know them a bit I'd say.

NOTE If you are copying task definitions out for stand alone scripts remember to replace '$$' for '$' where ever you see them.

Resources

Helmfile - The ultimate helm chart stitcher

My Archetype Chart - I use this quite a bit for standardized ingress among other things

Manage Helm Charts With Helmfile - Good article on some techniques to implement DRY helmfile deployments

Helm Secrets Plugin - Helm plugin for secrets management

Istio Practice Deployment - Inspired me to finish this framework

Fury Kubernetes Distribution - The basic concept of 'stacks' that I have been putting together using helmfiles has been done in this distribution using kustomize instead. Inspiring work but far too difficult to modify and use for rapid stack stitching from my experience.

CloudPosse's Helmfiles - A very impressive and well written set of helmfiles. Inspirational and if I'm honest, better than my own charts.

MetalLB - The software loadbalancer used in several of the cluster profiles

DPS - DNS Proxy Server container used to forward ingress zone requests to local clusters