Lambda Backup & BCP Functions for AWS

Overview

This is my dirty little collection of scripts (wrapped and executed via Lambda & an auto-spawning EC2 Instances, deployed using the Serverless Framework) used to backup all of your critical infrastructure within your entire AWS account (The scripts will iterate through EVERY AWS region and check for resources to be backed up). Backups are stored in a single S3 bucket, using standard file formats (such as DNS zone files, SQL Dumps etc) where ever possible.

The goal is to try and use standard existing scripts/tooling to perform these tasks rather than re-inventing the wheel - several existing scripts and tools from various authors have been either included or forked (and wrapped) in Serverless functions to perform these tasks. Standard tools such as mysqldump, cli53, dd etc that run on EC2 Instances or within Lambda Functions are used to create "standard" file formats, so that backups may be easily read/imported to whatever infrastructure you require in the future.

Infrastructure is auto-backed up and managed via "Resource Tags" (These can be defined virtually on any infrastructure within in the AWS console), allowing anyone performing DevOps within your organisation to quickly enable backups on a resource as required.

Backups may also optionally be copied from your S3 bucket to another Cloud or infrastructure provider of your choice if you are using these functions as part of your Disaster Recovery & Business Continuity Plan. If you're using Google Cloud, you should consider using the built-in sync tooling. See: Google Cloud Storage - Creating a Transfer Job from S3 to GCS.

If you require assistance deploying or customising this solution or any other enterprise grade Cloud infrastructure, please feel free to reach out to me - I provide consultancy services.

Current Features/AWS Services Covered:

• EC2 Worker Instances are launched via the Lambda Backup Functions each night, and then terminated upon completion. No special configuration is required other than the base configuration below.

The following additional features are currently a work in progress:

• Copy EBS snapshots/backups into S3 Bucket (stored as a GZipped ISO)
• Copy all data from "Backups S3 Bucket" to another Cloud Provider (Currently GCE is Supported) - This is intended for Business Continuity Planning.

Note that restoring your data is currently a manual process - These scripts are intended to just get your data out of AWS and backed up in another Cloud Provider for BCP purposes.

If you are looking at backing up ALL your infrastructure and not just the critical components - you should consider writing your Infrastructure as Code (using a tool such as Terraform.) and then checking this into your source control.

Getting Started

Once you have cloned down the project, you will need to configure some basic variables to get everything running...

Setting up IAM Roles (for worker instances)

EC2 instances that are auto-spawned from the lambda functions require access to your backup S3 bucket (to upload DB dumps etc into the bucket). You'll need to create an IAM policy (with access to the S3 bucket) and then a Role with the policy associated to it.

If you're happy using default access to S3 from the backup instance, you can use the template example role below -- Note you will need the AWS CLI Tools installed and configured on your machine before running the following:

# Be sure to set your AWS profile name if you have multiple accounts configured in your ~/.aws/config file...
export AWS_PROFILE="your-aws-profile-name"

POLICY_ARN=`aws iam create-policy --policy-name "AWSBackupTasksRoleForS3" \
    --description "Allows EC2 Instances spawned by the Backup Tasks Lambda functions to save files into specified S3 Backup Bucket" \
    --policy-document '{"Version": "2012-10-17","Statement": [{"Sid": "AWSBackupTasksRoleForS3","Effect": "Allow","Action": "s3:*","Resource": ["*"]}]}' \
    --output=text | awk '{print $2}'`

# Create IAM Role (This is the role that will be used for launching the EC2 backup instances)
aws iam create-role --role-name "AWSBackupTasksRoleForS3" \
    --description "Allows EC2 Instances spawned by the Backup Tasks Lambda functions to save files into specified S3 Backup Bucket" \
    --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Service": "ec2.amazonaws.com"},"Action": "sts:AssumeRole"}]}'

# Associate the IAM policy that was created in the first step with the IAM Role that we just made.
aws iam attach-role-policy  --role-name "AWSBackupTasksRoleForS3" \
    --policy-arn "$POLICY_ARN"

# We also need to create an Instance Profile, then assign the IAM role to this EC2 Instance Profile...
aws iam create-instance-profile --instance-profile-name "AWSBackupTasksRoleForS3"
aws iam add-role-to-instance-profile --role-name "AWSBackupTasksRoleForS3" \
    --instance-profile-name "AWSBackupTasksRoleForS3"

Setting environment variables & configuration for Lambda Functions

Some basic configuration is required before the scripts can do their thing - These are various variables that are passed through during the serverless deployment (as an environment variable) such as the S3 bucket to backup to etc.

cp config.yml.example config.yml

# Then, edit the config.yml file to suit your requirements
vi config.yml

Environment variables supported in config.yml currently include:

Deploying

This project should be manually deployed, however the tagging of resources that you want to back up may be done via standard Terraform provisioning tasks.

To manually deploy these backup functions into a new account you may:

# Install packages from package.json (includes serverless framework)
npm install

# Deploy using the AWS Credentials set in your ~/.aws/config file...
serverless deploy --aws-profile your-aws-profile-name

Backing up Infrastructure with AWS 'TAGS'

Managing EC2 Instance Volume Backups

By default the script will run each night and backup your EC2 instances that are tagged accordingly - Backups are retained for 30 days unless configured otherwise.

To backup an EC2 Instance you may simply add the following tags to the instance via the AWS Management Console... (Tags will be automatically copied from the instance to the EBS volumes every 15 minutes).

Managing RDS Dumps

Like EC2 Instance Backups, RDS dumps are run each night and then synchronised into the specified backup S3 bucket. Unlike EBS Volume Snapshots, there are no retention settings for nightly DB dumps (stored as gzipped sql files) - You will need to configure a policy on your S3 bucket.

Current Supported RDS Types:

• PostgreSQL
• Aurora PostgreSQL *See Comments Below re: Aurora Config

BackupConfiguration JSON Object Example:

The BackupConfiguration Tag currently accepts the following Json Keys:

• Username
• Password

To generate this configuration from the Command Line, simply run:

echo -n '{
    "Username": "<rdsRootUsername>",
    "Password": "<rdsRootPassword>"
}' | base64

(NB update the example to match the correct credentials for each RDS instance)

** Note: **

• The program dumps all accessible databases within the RDS instance into S3 - If you have hundreds of gb's of data this may not be effective and may need to look at other backup methods.

• RDS Backups are launched using latest Amazon Linux AMI (in each region), using the official DB vendor's docker image (i.e mysql or postgresql docker) ensuring that the CLI tools used by scripts are always the latest stable release.

• Instances are launched with the same VPC, Subnet & Security Groups that are assigned to the RDS instance that is being backed up - This assumes that the subnet has network access to S3 - If your subnet running the RDS instance does not have outbound internet access the scripts will not work.

Managing EFS Backups

If you have any Elastic Filesystem Deployments - files will be copied (via NFS) into your backup S3 bucket. Backups are run nightly. Unlike EBS Volume Snapshots, there are no retention settings for nightly EFS Sync's - you will need to set a S3 Lifecycle Policy on these directories if you wish to only retain X days of backups.

** Note: **

• EFS Backups are launched using latest Amazon Linux AMI (in each region) with the same VPC, Subnet & Security Groups that are assigned to the EFS Network Mount Point. This assumes that the subnet has outbound network/internet access to S3 - If your subnet running the EFS deployment does not have outbound internet access the scripts will not work.

Troubleshooting / Common Problems

• Instances are being launched but no backups appearing in my S3 bucket...

  • View/Check the "Instance Log"
  • Copy the "Launch Config" (View/Change User Data in EC2 Console) and launch a new instance with the same User Data to test the functions work locally...
  • Check the VPC that your Database/ElastiCache is in, has local Internet acces (Instances require access to the internet to install tools required to perform the backups - If your VPC does not have internet access, you may need to add a NAT Gateway)

• Instances are not stopping/deleting after backups

  • Try and SSH into the instance, and check that backup scripts are not still running... If you're backing up 1000's of GB's of data, it will take several hours for example, so you may only want to schedule the backups once per week.
  • Check your Lambda Logs for any errors relating to the script(s) and/or EC2 backup instance logs for further details.

Calling the Backup Functions Locally

Occasionally you may want to fire off the various Serverless functions locally without actually deploying the code into AWS Lambda.

This is relatively straightforward with the Serverless Framework and can be performed like so:

# 1 - Export any environment variables you may want to set

export AWS_PROFILE="aws-profile-name"
export EC2_KEYPAIR_NAME="keypair-name"

# 2 - Invoke the fun function directly i.e.
#     See the serverless.yml file for a list of function names.

serverless invoke local -f backupRDS

Additional Considerations

• Scripts do not currently take into consideration of the lifecycle of objects in your S3 bucket! - Create a S3 "Lifecycle Rule" to delete files after X days or Y Months if you do not wish to indefinitely store your backups created in your S3 bucket or are backing up large data quantities and want to keep your bill "lean"