pd-tech-fest-2019

Demo code for Programmers Developers Tech Fest 2019

There are multiple options for scaling with Kubernetes and containers in general. This demo uses Kubernetes-based Event Driven Autoscaling (KEDA). RabbitMQ is used as an event source.

Prerequisites

Azure Susbscription to create AKS cluster
kubectl logged into kubernetes cluster
Powershell
Postman
Helm
DockerHub account (optional)

If you wish to use Kubernetes cluster apart from AKS, you can skip the Step 2.1 of provisioning the cluster and install KEDA on your own kubernetes cluster.

Similarly, if you do not wish to execute the Powershell scripts, you can execute the commands which are part of those scripts manually.

1 - Code organization

Contains the source code for a model classes for a hypothetical Tech Talks management application. TechTalksMQProducer contains the code for generating the events / messages which are published to a RabbitMQ queue. TechTalksMQConsumer contains the consumer code for processing RabbitMQ messages.

Both the Producer and Consumer uses the common data model. In order to build these using Dockerfile, we define the TechTalksMQProducer and TechTalksMQConsumer. These are built docker-compose-build file.

The docker images can be built using the following command

Measure-Command { docker-compose -f docker-compose-build.yml build | Out-Default }

Once the images are built successfully, we can push them to the DockerHub registry using the command

Measure-Command { docker-compose -f docker-compose-build.yml push | Out-Default }

Powershell

Contains the helper Powershell scripts to provision AKS cluster, to proxy into the Kubernetes control plane, to delete the resource group, to deploy the application and also to delete the application.

Contains Kubernetes manifest files for deploying the Producer, Consumer and the Autoscalar components to the Kubernetes cluster.

helm

Contains the Helm RBAC enabling yaml file which add the Cluster Role Binding for RBAC enabled Kubernetes cluster. This was required before Helm 3.0 for the Tiller service.With helm 3.0, Tiller is no longer required.

2 - Demo setup

2.1 Initialize AKS cluster with KEDA

Run initializeAKS powershell script with default values from Powershell directory

.\initializeAKS.ps1

Note: The default options can be overwritten by passing arguments to the initializeAKS script. In the below example, we are overriding the number of nodes in the AKS cluster to 4 instead of 3 and resource group name as kedaresgrp.

.\initilaizeAKS `
-workerNodeCount 4 `
-resourceGroupName "kedaresgrp"

2.2 Deploy KEDA

.\deployKEDA.ps1

Verify KEDA is installed correctly on the Kubernetes cluster.


kubectl get all -n keda

2.3 Deploy RabbitMQ queue

.\deployRabbitMQ.ps1

2.4 Deploy RabbitMQ Producer & Consumers

Execute the deployTechTalks-AKS.ps1 powershell script.

.\deployTechTalks-AKS.ps1

The deployTechTalks powershell script deploys the RabbitMQConsumer and RabbitMQProducer in the correct order. Alternately, all the components can also be deployed directly using the kubectl apply command recursively on the k8s directory as shown below.

Run the kubectl apply recursively on k8s directory


kubectl apply -R -f .

2.5 Deploy Auto scalar for RabbitMQ consumer deployment

Execute the deployAutoScaler.ps1 powershell script.

.\deployAutoScaler.ps1

If you do not wish to run the individual PowerShell scripts, you can run one single script which will deploy all the necessary things by running the above scripts in correct order.

.\deployAll.ps1

2.6 Get list of all the services deployed in the cluster

We will need to know the service name for RabbitMQ to be able to do port forwarding to the RabbitMQ management UI and also the public IP assigned to the TechTalks producer service which will be used to generate the messages onto RabbitmQ queue.


kubectl get svc

As we can see above, RabbitMQ service is available within the Kubernetes cluster and it exposes 4369, 5672, 25672 and 15672 ports. We will be using 15672 port to map to a local port.

Also note the public LoadBalancer IP for the techtalksapi service. In this case the IP is 52.139.237.252. Note: This IP will be different when the services are redeployed on a different Kubernetes cluster.

2.7 Watch for deployments

The rabbitmq ScaledObject will be deployed as part of the deployment. Watch out for the deployments to see the changes in the scaling as the number of messages increases


kubectl get deployment -w

kubectl get deploy -w

Initially there is 1 instance of rabbitmq-consumer and 2 replicas of the techtalksapi (producer) deployed in the cluster.

2.8 Port forward for RabbitMQ management UI

We will use port forwarding approach to access the RabbitMQ management UI.


kubectl port-forward svc/rabbitmq 15672:15672

2.9 Browse RabbitMQ Management UI

http://localhost:15672/

Login to the management UI using credentials as user and PASSWORD. Remember that these were set during the installation of RabbitMQ services using Helm. If you are using any other user, please update the username and password accordingly.

2.10 Generate load using `Postman`

I am using Postman to submit a POST request to the API which generates 1000 messages onto a RabbitMQ queue named hello. You can use any other command line tool like CURL to submit a GET request.

Use the EXTERNAL-IP -52.139.237.252 with port 8080 to submit a GET request to the API. http://52.139.237.252:8080/api/TechTalks/Generate?numberOfMessages=2000

Note that we are setting the number of messages to be produced by Producer as 2000 in this case. You can change the number to any other integer value.

After building the GET query, hit the blue Send button on the top right. If everything goes fine, you should receive a 200 OK as status code.

The Producer will produce 2000 messages on the queue named hello. The consumer is configured to process 10 messages in a batch. The consumer also simulates a long running process by sleeping for 2 seconds.

2.11 Auto-scaling in action

See the number of containers for consumer grow to adjust the messages and also the drop when messages are processed.

On the left hand side of the screen you can see the pods auto scaled and on the right we see the deploymnets autoscaled progressively to 2,4,8, 16 and 30.

While the messages are being processed, we can also observe the RabbitMQ management UI.