Chaos Injection for AWS resources using Amazon SSM Run Command and Automation

Collection of SSM Documents.

These documents let you perform chaos engineering experiments on resources (applications, network, and infrastructure) in the AWS Cloud.

SSM Automation documents:

To use SSM Automation, check the link

• Support for (randomly) stopping EC2 instances via API
• Support for (randomly) stopping EC2 instances via AWS Lambda
• Support for (randomly) terminating EC2 instances via API
• Support for detaching EBS volumes from EC2 instances via API (ec2, ebs)
• Support for rebooting RDS instance with proper tags via API
• Support for CPU stress scenario via Run Command

Upload an SSM Automation document:

aws ssm create-document --name "StopRandomInstances-API" --content file://stop-random-instance-api.yml --document-type "Automation" --document-format YAML

SSM Run Command documents:

To use SSM Run Command, please check this link

Support Canceling & Rollback (10s max)

• Support for latency injection using latency-stress.yml
• Support for latency with delta stress using latency-delta-stress.yml
• Support for CPU burn using cpu-stress.yml
• Support for IO stress using io-stress.yml
• Support for memory stress using memory-stress.yml
• Support for network stress using network-corruption-stress.yml
• Support for packet Loss stress using network-loss-stress.yml
• Support for killing a process by name using kill-process.yml
• Support for diskspace stress using diskspace-stress.yml

Experimental

• Support for configurable blackhole stress using blackhole-stress.yml
• Support for blackhole S3 stress using blackhole-s3-stress.yml
• Support for blackhole DynamoDB stress using blackhole-dynamo-stress.yml
• Support for blackhole EC2 stress using blackhole-ec2-stress.yml
• Support for blackhole DNS stress using blackhole-dns-stress.yml
• Support for latency injection to a particular AWS service latency-service-stress.yml

Prerequisites

• SSM Agent (Preinstalled on several Amazon Machine Images)
• stress-ng, tc, and jq (Automatic install of dependencies)

Upload one document at a time

cd chaos-ssm-documents/automation

aws ssm create-document --content file://cpu-stress.yml --name "cpu-stress" --document-type "Command" --document-format YAML

Upload all of the SSM Documents to the AWS region of your choice

cd chaos-ssm-documents/run-command

./upload-document.sh -r eu-west-1 (or other region of your choice)

Upload all of the SSM Documents using CloudFormation

cd chaos-ssm-documents/

run-command/create-cfn.sh run-command/ | tee cfn-chaos-ssm.yml

aws cloudformation create-stack --stack-name ChaosSsm --template-body file://cfn-chaos-ssm.yml

Specify AWS region using AWS CLI --region argument.

Once deployed, the stack cannot be updated. Remove existing stack and re-deploy to apply changes.

SOME WORDS OF CAUTION BEFORE YOU START BREAKING THINGS:

• To begin with, DO NOT use these chaos injection commands in production blindly!!
• Always review the SSM documents and the commands in them.
• Make sure your first chaos injections are done in a test environment and on test instances where no real and paying customer can be affected.
• Test, test, and test more. Remember that chaos engineering is about breaking things in a controlled environment and through well-planned experiments to build confidence in your application — and you own tools — to withstand turbulent conditions.

One-click Deploy via CloudFormation