Changestream
Changestream sources object-level change events from a MySQL Replication Master (configured for row-based replication) by streaming the MySQL binlog and transforming change events into JSON strings that can be published anywhere.
Currently, Amazon Simple Queuing Service (SQS), Amazon Simple Notification Service and Amazon S3 are supported with optional client-side message encryption via AES.
Documentation
- Why SNS+SQS?
- How Do I Consume My Events?
- The Stack
- Architecture
- Binlog Event Routing
- IntelliJ
- Configuration
Getting Started
Setup
First, start MySQL with
docker-compose up db
(or, for the hard way without Docker, follow the bespoke setup instructions).
You will then have access to a MySQL instance on localhost port 3306 with a root password of "password", and Changestream configured to listen for events.
Changestream is available on Docker Hub.
Next run the dev environment
docker-compose run dev
Then build and run Changestream using the sbt CLI:
$ sbt run
Building
Docker Image (requires a docker host to build)
Note: The Dockerfile will be written to target/docker/Dockerfile, and the docker image will be added to your local docker repo.
$ sbt docker
Debian Package (requires dpkg-deb, written to target/*.deb)
$ sbt debian:packageBin
Jar Package (written to target/scala-2.11/*.jar)
$ sbt package
Local development without docker
Requirements
- Scala 2.11
- Akka 2.5
- SBT 0.13
- MySQL 5.7 (5.5+ supported)
Contributing
Testing
Changestream is 100% unit and integration tested, with end-to-end testing on CircleCI with MySQL and Amazon SNS.
Mysql Test Config
In order to run tests locally you need to configure test.conf with a working MySQL connection configuration.
Make sure that your test user has permissions to create and drop databases and tables. By default, Changestream uses the changestream_test table for tests.
AWS Test Config
To run the integration tests, as well as the unit tests on the SqsActor and SnsActor, you also need to ensure that you have configured your AWS tokens to be available from the environment, and that your AWS tokens have access to create and add messages to queues (SQS) and create and publish to topics (SNS).
AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>
AWS_REGION=<your_region>
AWS_S3_BUCKET=<your_bucket>
AWS_S3_KEY_PREFIX=<some_key_prefix>
Running Tests
Once that's all set up, you can either run
$ sbt test
from the command line, or run the test command in IntelliJ (with the Scala and SBT plugins installed).
ScalaStyle
Changestream uses ScalaStyle for code linting--it is not automatic. Before submitting a PR, please take a look at the output of the sbt scalastyle command and correct any issues you find.
License
Changestream is available under the MIT License.
Authors
Peter Fry (@racerpeter) and the Mavengineering team.
Inspiration
Changestream was inspired in a large part by mardambey/mypipe by Hisham Mardam-Bey. The first proof-of-concept for changestream was a fork of mypipe - thank you Hisham.