CARV-ICS-FORTH / frisbee

Frisbee - A Test Automation Framework For Kubernetes

Website | Blog | Docs | Contact

What is Frisbee ?

Frisbee is the first Kubernetes framework designed to support declarative end-to-end testing of containerized applications.

✔️ Setup application and dependency stack – easily!

✔️ Test against actual, close to production software - no mocks!

✔️ Replay complex workloads written in an intuitive language!

✔️ Combine Chaos Engineering with large-scale performance testing!

✔️ Assert actual program behavior and side effects.

To learn more about Frisbee, check the QuickStart tutorial or visit our Website.

Use-Cases and Testing Patterns

In declarative testing, a test scenario focuses on what to accomplish rather than on the imperative details of how to manipulate the state of an application under test and verify the final application state against an expected state.

This approach not make tests more readable, maintainable, and reproducible, but it also help devops in identifying testing patterns.

Here, you can see some testing patterns we have identified across different application domains.

👉 Databases

👉 Federated Learning

👉 Filesystems

👉 HPC

👉 Networking

Getting Started

Before starting, Make sure that kubectl and Helm are installed on your system.

For quick testing, you can use microk8s. In that case, make sure you have added the addons and created aliases to the commands.

sudo microk8s config > ~/.kube/config
sudo microk8s enable dns ingress helm3
sudo snap alias microk8s.kubectl kubectl
sudo snap alias microk8s.helm3 helm

Install Frisbee CLI and Frisbee Platform

Then, run the install.sh, that will deploy the Frisbee Terminal as an extension to kubectl.

curl -sSLf https://frisbee.dev/install.sh | sudo bash

Through Frisbee Terminal we can easily install the Frisbee Platform.

kubectl frisbee install production

Once installed, you can follow the controller's output using:

kubectl logs -l control-plane=frisbee-operator -n frisbee --follow

Finally, download the Frisbee repo from GitHub.

git clone [email protected]:CARV-ICS-FORTH/frisbee.git

This step is not really needed for the installation. We use it to get local access in examples and charts directories.

• examples: contains a list of test-cases.
• charts: contains Helm charts that provide templates used in the test-cases.

Submit a Testing Job.

Let's start by running the hello-world.

kubectl frisbee submit test demo- ./examples/1.hello-world.yml

The name can be explicit (e.g, my-demo), or autogenerated given a prefix followed by a - (e.g, demo-).

In any case, submit returns the test id (| awk /test:/'{print $4}) so that it can be used to downstream commands.

Inspect Submitted Jobs.

To get a list of submitted tests, use:

kubectl frisbee get tests

Note that every test-case runs on a dedicated namespace (named after the test). To further dive into execution details use:

kubectl frisbee inspect tests demo-482

Live Progress Monitoring

Let's try something more complex to demonstrate the integration of Frisbee with Prometheus/Grafana.

kubectl frisbee submit test demo- examples/15.performance-monitoring.yml charts/system charts/networking/iperf2

Notice that we modified the command to include dependencies required for the execution of the scenario.

• charts/systems: provides the templates for the telemetry stack.
• charts/networking/iperf: provides the templates for iperf benchmark.

All that it takes now is to open the URLs of section Visualization Dashboards in your browser. You can it either manually or via the one-liner:

firefox $(kubectl frisbee inspect tests demo-710 | grep grafana- | awk '{print $3'})

In contrast to the vanilla Grafana which plots only the performance metrics, Frisbee provides Contextualized Visualizations that contain information for:

• Joining nodes (blue vertical lines)
• Exiting nodes (orange vertical lines)
• Fault-Injection (red ranges)

Information like that helps in root-cause analysis, as it makes it easy to correlate an observed behavior back to a testing event.

For example, in the next figure, it fairly easy to understand that INSERT_ERROR messages (yellow line) are triggered by a fault-injection event.

Features

👉 Workflow templating to store commonly used workflows in the cluster.

👉 DAG based declaration of testing workflows.

👉 Step level input & outputs (template parameterization).

👉 Conditional Execution (Time-Driven, Status-Driven, Performance-Driven).

👉 Live Progress monitoring via Prometheus/Grafana.

👉 Assertions and alerting of SLA violations.

👉 Placement Policies (affinity/tolerations/node selectors).

👉 Archiving Test results after executing for later access.

👉 On-Demand reliable container attached storage.

👉 Garbage collection of completed resources.

👉 Chaos-Engineering and Fault-Injection via Chaos-Mesh.

👉 On-Demand reliable container attached storage.

👉 CLI applications to test management and test inspection.

To learn how to use these features, check the Walkthrough.

Citation

If you publish work that uses Frisbee, please cite Frisbee as follows:

@article{nikolaidis2021frisbee,
title={Frisbee: automated testing of Cloud-native applications in Kubernetes},
author={Nikolaidis, Fotis and Chazapis, Antony and Marazakis, Manolis and Bilas, Angelos},
journal={arXiv preprint arXiv:2109.10727},
year={2021}
}

Contributing to Frisbee

We welcome contributions. Please see CONTRIBUTING.md to get started!

Acknowledgements

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 894204 (Ether, H2020-MSCA-IF-2019).