Kubernetes Response Engine powered by OpenFaaS
Although Falco can be used to detect any intrusion attempts and sends alerts to channels according to the given rules afterwards, it does not have any remediation system. This is why we need something called Kubernetes Response Engine. It simply aims to catch alerts and take actions on it. These actions can be designed as fine-grained serverless functions.
Think of a scenario you want to take action to your alerts that being notified by Falco, but the important thing that we should notice here is we are going to do different things for the different levels of alerts. For example, we are going to send the notification for notice level alert but for the warning level alert we are going to delete the pod, the overall architecture will be like this:
+-----------+
| Falco +
+-----^-----+
|
+-------v-------+
>OpenFaaS (dispatch-fn)+
+-------v-------+
+-----------+ | +-----------+
| notify-fn <---------+----------> delete-fn |
+-----v-----+ notice warning +-----+-----+
| |
| send alert delete pod |
| |
+-----v-----+ +-----v-----+
| Slack | | Pwned Pod |
+-----------+ +-----------+🎁 Table of Contents
Prerequisites
civo cli v0.7.6
Helm v3.5.1
faas-cli 0.13.9- arkade 0.7.10
kubectl v1.20.2
We are going to do this demo on macOS Catalina 1.15.7, you can find the prerequisites on brew.
Target Audience
If you want to:
- Set up a K3S cluster on CIVO
- Set up the Falco
- Create custom serverless functions using OpenFaaS
- Subscribe Falco events from serverless functions
What is ...?
CIVO
Built for speed and simplicity, with K3s under the hood
Join the #Kube100 beta: Apply to join today (Get free credit to test-drive the world’s first K3s-powered, managed Kubernetes service)
- Simple, fast and powerful
Spin up Kubernetes in under 2 minutes, without the bloat, using the lightweight K3s distribution
- Management CLI
Manage your clusters with the custom-built CLI and streamline your deployment with simple REST API.
- Application marketplace
Launch clusters with preinstalled applications, or install on the fly using Civo's Kubernetes marketplace.
- Bandwidth Pooling
Use all your bandwidth allowance across all your Civo services. Only pay for what you use.
- Marketplace
Install applications quickly and easily to your clusters during or after set up with Civo marketplace. Also you can contribute an application to marketplace
- Custom Firewalls
Configure custom firewalls from your Civo dashboard using OpenStack’s security groups.
K3S
The certified Kubernetes distribution built for IoT & Edge computing
- Perfect for Edge
K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. Simplified & Secure
- Simplified & Secure
K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster.
- Optimized for ARM
Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server.
Falco
Cloud-Native runtime security, de facto Kubernetes threat detection engine
- Strengthen container security
The flexible rules engine allows you to describe any type of host or container behavior or activity.
- Reduce risk via immediate alerts
You can immediately respond to policy violation alerts and integrate Falco within your response workflows.
- Leverage most current detection rules
Falco out-of-the box rules alert on malicious activity and CVE exploits.
OpenFaaS
OpenFaaS® makes it simple to deploy both functions and existing code to Kubernetes
- Anywhere
Avoid lock-in through the use of Docker. Run on any public or private cloud.
- Any code
Build both microservices & functions in any language. Legacy code and binaries.
- Any scale
Auto-scale for demand or to zero when idle.
Hands-on Demonstration
Create CIVO Playground
- Download the civo cli from here
- Copy and save the API key from your Civo account - Security Dashboard
$ civo apikey save my-awesome-key $KEY
Saved the API Key $KEY as my-awesome-key$ civo apikey list
+----------------+---------+
| Name | Default |
+----------------+---------+
| my-awesome-key | $KEY |
+----------------+---------+$ civo apikey current my-awesome-key
Set the default API Key to be my-awesome-key- Create a cluster
$ civo kubernetes create playground --size=g3.k3s.medium --nodes=3 --region NYC1 --wait
Created Kubernetes cluster playground- Show the playground on Kubernetes Dashboard
$ civo kubernetes show playground
ID : 79435efe-2dac-403d-bfd2-f6644988830a
Name : playground
Nodes : 3
Size : g3.k3s.medium
Status : ACTIVE
Version : 1.20.0-k3s2
API Endpoint : https://212.2.243.151:6443
Master IP : 212.2.243.151
DNS A record : 79435efe-2dac-403d-bfd2-f6644988830a.k8s.civo.com
Nodes:
+-------------+---------------+--------+---------------+-----------+------+----------+
| Name | IP | Status | Size | Cpu Cores | Ram | SSD disk |
+-------------+---------------+--------+---------------+-----------+------+----------+
| master-7c8a | 212.2.243.151 | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
| node-04ed | | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
| node-5258 | | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
+-------------+---------------+--------+---------------+-----------+------+----------+
- Configure the playground
$ civo kubernetes config playground --save --local-path ./kubeconfig
Access your cluster with:
KUBECONFIG=./kubeconfig kubectl get node- Ensure all is OK
$ export KUBECONFIG=./kubeconfig
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k3s-playground-66b18d51-node-04ed Ready <none> 40h v1.20.2+k3s1You can find more details about civo cli here.
- Set up OpenFaaS
You can install OpenFaaS from Civo marketplace -
or via arkade
arkade provides a portable marketplace for downloading your favourite devops CLIs and installing helm charts, with a single command. You can also download CLIs like kubectl, kind, kubectx and helm faster than you can type "apt-get/brew update"
$ arkade install openfaas
Using Kubeconfig: /Users/batuhan.apaydin/.kube/config
Using Kubeconfig: /Users/batuhan.apaydin/.kube/config
Client: x86_64, Darwin
2021/03/11 21:35:24 User dir established as: /Users/batuhan.apaydin/.arkade/
"openfaas" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nats" chart repository
...Successfully got an update from the "kyverno" chart repository
...Successfully got an update from the "dq-helm-charts" chart repository
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
VALUES values.yaml
Command: /Users/batuhan.apaydin/.arkade/bin/helm [upgrade --install openfaas openfaas/openfaas --namespace openfaas --values /var/folders/pf/6h9t0mnd4d342ncgpjq_3zl80000gp/T/charts/openfaas/values.yaml --set basicAuthPlugin.replicas=1 --set basic_auth=true --set clusterRole=false --set gateway.directFunctions=false --set openfaasImagePullPolicy=IfNotPresent --set faasnetes.imagePullPolicy=Always --set queueWorker.maxInflight=1 --set serviceType=NodePort --set operator.create=false --set gateway.replicas=1 --set ingressOperator.create=false --set queueWorker.replicas=1]
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Thu Mar 11 21:35:30 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:
kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
=======================================================================
= OpenFaaS has been installed. =
=======================================================================
# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh
# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &
# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdin
faas-cli store deploy figlet
faas-cli list
# For Raspberry Pi
faas-cli store list \
--platform armhf
faas-cli store deploy figlet \
--platform armhf
# Find out more at:
# https://github.com/openfaas/faas
Thanks for using arkade!Also, there are other ways to install OpenFaaS, here is the official documentation.
- Set up Falco
$ arkade install falco \
--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://gateway.openfaas:8080/function/dispatch-fn
...After everyting is ok, configure the faas-cli in order to deploy functions.
# Forward the gateway to your machine
$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &
# If basic auth is enabled, you can now log into your gateway:
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$ echo -n $PASSWORD | faas-cli login --username admin --password-stdinLet's see what functions present.
$ faas-cli list
Function Invocations ReplicasYou should see nothing.
NOTE: Don't forget to apply roles.yaml for delete-pod-fn because this function will remove the pod if necessarry, so, it needs some priviliges for that.
Now Deploy the functions.
$ kubectl apply -f roles.yaml
$ faas-cli deploy -f stack.yml
...Once it's completed, you should see the pods that's running on openfaas-fn namespace, and you should also see
functions by typing faas-cli list command.
$ faas-cli list
Function Invocations Replicas
dispatch-fn 37 1
notifier-fn 23 1
delete-pod-fn 0 1
$ kubectl get pods --namespace=openfaas-fn
Found existing alias for "kubectl get pods". You should use: "kgp"
NAME READY STATUS RESTARTS AGE
dispatch-fn-b5b68df77-r7crl 1/1 Running 0 7m23s
notifier-fn-5fd846c887-hgpf4 1/1 Running 0 4m48s
delete-pod-fn-55cc89965c-fs55q 1/1 Running 0 37sIf everything works well, you should see the Slack messages in the #falco-alert channel. Here is the screenshot above that shows this.
Learn more about how you can set up a Slack workspace from this link.
The next thing that you need to do this, you should create an alert at Warning level, in order to that, you need to create a Pod based on the alpine image, and you should connect to it, and access some important folders like /etc/shadow. Once this is done, you should notice that the Pod gets killed by the delete-pod-fn function.
Let's test it with creating a Pod.
$ kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 600"
Found existing alias for "kubectl". You should use: "k"
pod/alpine createdConnect to it.
$ kubectl exec -ti alpine -- sh
Found existing alias for "kubectl exec -ti". You should use: "keti"
/ #Once you connect it, open a second terminal and watch the state of the Pod.
$ watch kubectl get pods
Every 2.0s: kubectl get pods
NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 106sThen, in the first terminal, run the cat /etc/shadow, then look at the second terminal, you should notice that Pod's
status changed from Running to Terminating, and your connection to the Pod is lost.
