aws-samples / rds-snapshot-export-to-s3-pipeline

RDS Snapshot Export to S3 Pipeline

This repository creates the automation necessary to export Amazon RDS snapshots to S3 for a specific database whenever an automated snapshot is created.

Usage

1. Install the Amazon Cloud Development Kit (CDK).
2. Clone this repository and cd into it.
3. Modify the arguments to the RdsSnapshotExportPipelineStack constructor in $/bin/cdk.ts according to your environment.
   • dbName: This RDS database must already exist.
   • rdsEventId: This should be RdsEventId.DB_AUTOMATED_AURORA_SNAPSHOT_CREATED for Amazon Aurora databases or RdsEventId.DB_AUTOMATED_SNAPSHOT_CREATED otherwise.
   • s3BucketName: An S3 bucket with the provided name will be created automatically for you.
4. Execute the following:
   • npm install
   • npm run cdk bootstrap
   • npm run cdk deploy
5. Open up your <dbName>-rds-snapshot-exporter function in the AWS Lambda console and configure a test event using the contents of $/event.json OR $/event.aurora.json as a template, depending on whether or not you're using Amazon Aurora.
   • NOTE: The example content is a subset of an SNS event notification containing the minimum valid event data necessary to successfully trigger the Lambda function's execution. You should modify the <SNAPSHOT_NAME> value within the Message key to match an existing RDS snapshot (e.g. rds:<dbName>-YYYY-MM-DD-hh-mm). You may also need to modify the MessageId if you are attempting to export the same snapshot more than once.
6. Click the Test button to start an export.

You can check on the progress of the export in the Exports in Amazon S3 listing. When that is finished, you can use the AWS Glue Crawler that was created for you to crawl the export, then use Amazon Athena to perform queries on the exported snapshot.

Cleanup

Execute npm run cdk destroy to delete resources pertaining to this example.

You will also need to delete the following manually:

• The S3 bucket that was created to store the snapshot exports.
• The CDKToolkit CloudFormation Stack created by npm run cdk bootstrap.
• The cdktoolkit-stagingbucket-<...> bucket.

Demo

License

This library is licensed under the MIT-0 License. See the LICENSE file.