Awesome Chaos Engineering 
Testing in production (TiP) is gaining steam as an accepted practice in DevOps and testing communities, but no amount of preproduction QA testing can foresee all the possible scenarios in your real production deployment.
The prevailing wisdom is that you will see failures in production; the only question is whether you'll be surprised by them or inflict them intentionally to test system resilience and learn from the experience.
The latter approach is chaos engineering.
To understand all this knowledge is very important have a good background in Chaos Engineering, containers, fault injection, monitoring and observability.
Contents
- 0. Introduction
- 1. Chaos in Practice
- 2. Principles of Chaos Engineering
- 3. Fault Injection
- 4. Observability
- 5. Incident Management Tool
- 6. Cost of SEVs
- 7. Chaos As A Sevice
- 8. Gamedays
- 9. Forums and Groups
- 10. References
- 11. License
- 12. Contributing
0. Introduction
Chaos engineering is defined as "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production" (Principles of Chaos Engineering, http://principlesofchaos.org/).
In other words, it's a software testing method focusing on finding evidence of problems before they are experienced by users.
It's a common misconception that chaos engineering is only about randomly breaking things in production. It's not. Although running experiments in production is a unique part of chaos engineering (more on that later), it's about much more than that—anything that helps us be confident the system can withstand turbulence.
IMPORTANT!: Chaos engineering is not just about randomly breaking things ;-)
It interfaces with site reliability engineering (SRE), application and systems performance analysis, and other forms of testing.
Practicing chaos engineering can help you prepare for failure, and by doing that, learn to build better systems, improve existing ones, and make the world a safer place.
Motivations for chaos engineering
There are at least three good reasons to implement chaos engineering:
- Determining risk and cost and setting service-level indicators, objectives, and agreements
- Testing a system (often complex and distributed) as a whole
- Finding emergent properties you were unaware of
1. Chaos in Practice
To specifically address the uncertainty of distributed systems at scale, Chaos Engineering can be thought of as the facilitation of experiments to uncover systemic weaknesses. These experiments follow four steps:
- Start by defining ‘steady state’ as some measurable output of a system that indicates normal behavior.
- Hypothesize that this steady state will continue in both the control group and the experimental group.
- Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc.
- Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.
The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system. If a weakness is uncovered, we now have a target for improvement before that behavior manifests in the system at large.
2. Principles of Chaos Engineering
A chaos experiment is defined as the following five points by the Principles of chaos engineering
- Build a Hypothesis around Steady State Behavior
- Vary Real-world Events
- Run Experiments in Production
- Automate Experiments to Run Continuously
- Minimize Blast Radius
More details in the following link ;-)
3. Fault Injection
Generic Tools
- The Simian Army - A suite of tools for keeping your cloud operating in top form.
- Chaos Monkey - A resiliency tool that helps applications tolerate random instance failures.
- Chaos Toolkit - A chaos engineering toolkit to help you build confidence in your software system.
- Chaos Toolkit Turbulence - This is an extension for Chaos Toolkit which adds support for Turbulence attacks.
- Monarch - This is a series of tools for Chaos Toolkit.
- Muxy - A chaos testing tool for simulating a real-world distributed system failures.
- Chaos Blade - Chaosblade is an experimental tool that follows the principles of Chaos Engineering and is used to simulate common fault scenarios, helping to improve the recoverability of faulty systems and the fault tolerance of faults.
- Cthulhu - Chaos Engineering tool that helps evaluating the resiliency of microservice systems simulating various disaster scenarios against a target infrastructure in a data-driven manner.
- Namazu - Programmable fuzzy scheduler for testing distributed systems.
- Chaos Scimmia - Chaos Engineering for Redis.
- HavocLeopard - A set of simple chaos engineering apps that can be used to royally screw up your on-prem servers.
- Arcdata - Open source incident management and volunteer scheduling application for Red Cross Disaster Services.
- AWS Chaos Scripts - Collection of python scripts to run failure injection on AWS infrastructure.
CPU's
- Cpu Troll - Dedicated to raising CPU latency by the requested percentage and timespan.
Memory
File system
Disk
Networking
- Toxiproxy - A TCP proxy to simulate network and system conditions for chaos and resiliency testing.
- Comcast - A tool designed to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets.
Security
- Infection Monkey - Open source security tool for testing a data center's resiliency to perimeter breaches and internal server infection. The Monkey uses various methods to self propagate across a data center and reports success to a centralized Monkey Island server.
- ChaoSlingr - Introducing Security Chaos Engineering. ChaoSlingr focuses primarily on the experimentation on AWS Infrastructure to proactively instrument system security failure through experimentation.
Languages
Compilation time
- ChaosCat - Chaos engineering for Pull Requests - Taking a not-even-good joke a bit too far.
Run time
- Byteman - A Swiss Army Knife for Byte Code Manipulation.
- Byte-Monkey - Bytecode-level fault injection for the JVM. It works by instrumenting application code on the fly to deliberately introduce faults like exceptions and latency.
- Perses - A project to cause (controlled) destruction to a JVM application.
- Wiremock - API mocking (Service Virtualization) which enables modeling real world faults and delays.
- MockLab - API mocking (Service Virtualization) as a service which enables modeling real world faults and delays.
- Flaw - Inject failures on api calls for local chaos engineering.
- Havoc - Havoc is a collection of dangerous code that wreck havoc in .NET applications and the operating system for chaos-engineering.
- Utilities for frontend chaos engineering - Utilities for frontend chaos engineering.
- CHAOS GOPHER - A collection of unix style tools in GO to do chaos engineering or testing.
- Chaos Monkey for Spring Boot - Injects latencies, exceptions, and terminations into Spring Boot applications.
- React Chaos - Chaos Engineering for your React apps.
- Chaos QoaLa - ChaosQoaLa is a chaos engineering tool for injecting failure into JavaScript backed GraphQL end points.
- Chaos Reverse-engineering - Chaos engineering approach by Reverse-engineering.
Application
- ChaosMachine - Tool to do chaos engineering at the application level in the JVM.
- TripleAgent - System for fault injection for Java applications. .
Kernel & Operating System
Containers & Orchestrators
- ChaosOrca - Tool for doing Chaos Engineering on containers by perturbing system calls for processes inside containers.
- POBS - Automatic Observability and Chaos for Dockerized Java Applications.
- Pumba - Chaos testing and network emulation for Docker containers (and clusters).
- Blockade - Docker-based utility for testing network failures and partitions in distributed applications.
- Chaos Engineering for Docker - Chaos Engineering for Docker.
- Chaos Engineering with Docker EE - Chaos Engineering with Docker EE.
- Chaos Util - Docker image with utilities for Chaos Engineering.
- Drax - DC/OS Resilience Automated Xenodiagnosis tool. It helps to test DC/OS deployments by applying a Chaos Monkey-inspired, proactive and invasive testing approach.
- Pod-Reaper - A rules based pod killing container. Pod-Reaper was designed to kill pods that meet specific conditions that can be used for Chaos testing in Kubernetes.
- Chaoskube - ChaosKube periodically kills random pods in your Kubernetes cluster.
- Litmus - Framework for Kubernetes environments that enables users to run test suites, capture logs, generate reports and perform chaos tests.
- Chaos Operator - Chaos engineering via kubernetes operator.
- Kube Entropy - A little chaos engineering application for kubernetes resilience testing.
- kubernetes-chaos-lab - A brief guide to setting up your first chaos engineering lab on Kubernetes!.
- Chaos Mesh - A Chaos Engineering Platform for Kubernetes.
Hypervisors
- VMware Mangle - Orchestrating Chaos Engineering.
- Turbulence - Tool focused on BOSH environments capable of stressing VMs, manipulating network traffic, and more. It is very simmilar to Gremlin.
Cloud
- Glooshot - Chaos engineering framework to help you Immunize your service mesh.
- kube-monkey - An implementation of Netflix's Chaos Monkey for Kubernetes clusters.
- Powerful Seal - PowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.
- KubeInvaders - Gamfied Chaos engineering tool for Kubernetes Clusters.
- Kube DOOM - The next level of chaos engineering is here! Kill pods inside your Kubernetes cluster by shooting them in Doom.
- Testing Amazon Aurora Using Fault Injection Queries - Testing Amazon Aurora Using Fault Injection Queries.
- Azure Fault Analysis Service
- Include controlled Chaos in Service Fabric clusters - Include controlled Chaos in Service Fabric clusters.
- Chaos Lambda - Randomly terminate ASG instances during business hours.
- GomJabbar - ChaosMonkey for your private cloud.
- Chaos Engineering on Google Cloud Platform - Chaos Engineering on Google Cloud Platform.
- Chaos SSM Documents - Collection of AWS SSM Documents to perform Chaos Engineering experiments.
Examples Projects
- A Chaos Engineering Bootcamp - A Chaos Engineering Bootcamp.
- HW4 - Express servers were used to implement service topologies.
- Serverless Chaos Engineering Demo - This example demonstrates how to use Adrian Hornsby's Failure Injection Layer to perform chaos engineering experiments on a serverless environment.
- Chaos Engineeing Demo - Simple project demonstrating chaos engineering with Chaos Monkey and Resiliance4J.
- Chaos Engineering Demo - resilience4j + chaos toolkit + wiremock + chaos monkey for spring boot sample application.
- How to Create a Kubernetes Cluster on Ubuntu 16.04 with kudeadm and Weave Net
4. Observability
Specific tools
General Use
5. Incident Management Tool
- Banjaxed - Open source incident management tool.
6. Cost of SEVs
- Availability Calculator - Calculate how much downtime should be permitted in your SLA.
7. Chaos As A Sevice
- Gremlin Inc. - Failure as a Service.
- Chaos Engineering Experiment Automation - Chaos Engineering Experiment Automation.
- Pystol.org - The cloud chaos engineering toolbox.
- Chaos Platform - Chaos Engineering Platform for Everyone.
- Chaos Hub - Chaos Hub stands on the shoulders of the Chaos Toolkit to provide a complete, user-friendly, platform to automate and collaborate on your Chaos Engineering and Resiliency efforts.
- steadybit - Chaos Engineering platform that helps to proactively reduce downtime and provide visibility into systems to detect issues.
8. Gamedays
- Target: What is a Gameday? - Chaos Gamedays experience by Target.
- Codecentric: Chaos Engineering Gamedays - Chaos Gamedays by Codecentric.
- New Relic: How to run a Gameday? - Chaos Gamedays experience by New Relic.
- Dius: Gamedays resources - Resources for getting started with GameDay and Chaos Engineering.
- Gremlin: Gamedays - Resources for getting started with GameDay and Chaos Engineering.
- Gremlin: Planning your own Chaos Day - Example of a Gameday with DynamoDB by Gremlin.
- Gremlin: How to run a Gameday? - Methodology to run Gamedays according Gremlin.
- Gremlin DB: Breaking Dynamo DB - Example of a Gameday with DynamoDB by Gremlin.
- Gremlin: Introduction to Gameday - What is a Gameday according Gremlin.
- Gremlin: Inside Gremlin 2019 Gremlin Gamedays Roadmap - Chaos Gamedays experience by Gremlin.
- Gremlin: What I lerned running the Chaos Lab with Kafka - Example of a Gameday with Kafka by Gremlin.
- Chaos Toolkit: Chaos Engineering with Humans in the loop - Article about Chaos Gamedays.
- GooCardless: All fun and games until you start with Gamedays - Article about Chaos Gamedays.
- InfoQ: Gamedays - Achieving Resilience through Chaos Engineering - InfoQ Presentation with experiences about Chaos Gamedays.
9. Forums and Groups
- CNCF Chaos Engineering Working Group
- CNCF Chaos Engineering Working Group Slack: #chaosengineering (slack.cncf.io)
- CNCF Chaos Engineering Working Group GitHub
- Chaos Toolkit Slack Community
10. References
- https://github.com/seeker89/chaos-engineering-book
- What is security chaos engineering and why is it important?
- Security Chaos Engineering: A new paradigm for cybersecurity
- Injecting chaos experiments into security log pipelines
- Purple testing and chaos engineering in securityexperimentation
- A new approach to security instrumentation
- https://blog.qaware.de/posts/chaos-engineering-the-status-quo/
- https://blog.qaware.de/posts/chaos-engineering-chaostoolkit/
- https://github.com/chaoseng/wg-chaoseng/blob/master/WHITEPAPER.md
- https://docs.google.com/document/d/1BeeJZIyReCFNLJQrZjwA4KMlUJelxFFEv3IwED16lHE/edit?ts=5ace0eab#heading=h.ephtflhfpd1d
- https://github.com/dastergon/awesome-chaos-engineering
- https://techbeacon.com/app-dev-testing/chaos-engineering-testing-34-tools-tutorials
- https://github.com/dastergon/awesome-chaos-engineering
- https://www.techrepublic.com/article/chaos-engineering-a-cheat-sheet/
- https://medium.com/capital-one-tech/4-real-world-scenarios-that-read-like-chaos-engineering-experiments-8dbf40c5f247
- https://thenewstack.io/gremlins-tammy-butow-on-the-business-side-of-chaos-engineering/
- https://learnk8s.io/blog/kubernetes-chaos-engineering-lessons-learned
- https://gocardless.com/blog/game-days-at-gc/
- https://engineering.grab.com/chaos-engineering
- https://blog.newrelic.com/engineering/chaos-engineering-explained/
- https://slack.engineering/disasterpiece-theater-slacks-process-for-approachable-chaos-engineering-3434422afb54
- https://www.usenix.org/system/files/osdi18-veeraraghavan.pdf
- https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
- https://people.ucsc.edu/~palvaro/fit-ldfi.pdf
- https://landing.google.com/sre/book.html
- http://the-cloud-book.com/
- https://www.infoq.com/minibooks/emag-chaos-engineering
- https://www.pagerduty.com/blog/failure-fridays-four-years/
- https://www.slideshare.net/zgrinch/monkeys-lemurs-and-locusts-oh-my
- https://softwareengineeringdaily.com/2018/02/02/chaos-engineering-with-kolton-andrus/
- https://blog.codeship.com/embracing-the-chaos-of-chaos-engineering/
- https://sharpend.io/chaos-monkey-for-fun-and-profit/
- https://queue.acm.org/detail.cfm?id=2353017
- https://dl.acm.org/citation.cfm?id=3177123.3158134
- https://dl.acm.org/citation.cfm?id=2723711
- https://azure.microsoft.com/en-us/blog/inside-azure-search-chaos-engineering/
- https://devops.com/netflix-the-simian-army-and-the-culture-of-freedom-and-responsibility/
- http://www.oreilly.com/webops-perf/free/chaos-engineering.csp
- http://www.oreilly.com/webops-perf/free/antifragile-systems-and-teams.csp
- http://shop.oreilly.com/product/0636920251897.do
- https://www.gremlin.com/community/tutorials/chaos-engineering-the-history-principles-and-practice/
- https://www.gremlin.com/blog/the-discipline-of-chaos-engineering/
- http://kth.diva-portal.org/smash/get/diva2:1366436/FULLTEXT01.pdf
- https://arxiv.org/pdf/1907.13039.pdf
- https://arxiv.org/abs/1404.3056
- https://arxiv.org/abs/1702.05843
- https://arxiv.org/abs/1702.05849
- https://arxiv.org/abs/1805.05246
- https://arxiv.org/abs/1812.10706
- https://medium.com/@bbideep/why-should-chaos-be-part-of-your-distributed-systems-engineering-5bcb21497660
- https://medium.com/@njones_18523/chaos-engineering-traps-e3486c526059
- https://medium.com/@adhorn/chaos-engineering-ab0cc9fbd12a
- https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
- https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116
- https://medium.com/netflix-techblog/automated-failure-testing-86c1b8bc841f
- https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa
- https://medium.com/netflix-techblog/chap-chaos-automation-platform-53e6d528371f
- https://medium.com/@crochefolle/how-to-convince-your-boss-to-make-them-say-yes-to-chaos-engineering-796ba119bd7
- https://medium.com/chaosiq/cloud-native-and-chaos-engineering-20842ee2fa8a
- https://www.wired.com/story/netflix-ddos-attack/
- https://github.com/gremlin/chaos-engineering-tools
- https://github.com/greenlearner01/Chaos-Engineering/blob/master/Chaos-Engineering.md
11. License
12. Contributing
Contributions welcome! Read the contribution guidelines first.
Thank you!