Event Stream Processing Microservice Example

In an event-driven microservices architecture, the concept of a domain event is central to the behavior of each service. Popular practices such as CQRS (Command Query Responsibility Segregation) in combination with Event Sourcing are becoming more common in applications as microservice architectures continue to rise in popularity.

This reference architecture and sample project demonstrates an end-to-end example of building event-driven microservices that use Spring Boot and Spring Cloud. The project aims to show what an ideal development process might look like for building microservices that handle both HTTP and AMQP protocols for exchanging messages.

Demonstrated concepts:

• Event Sourcing
• Event Stream Processing
• Remediating Partial Failures
• Compensating Transactions
• Eventual Consistency
• Hypermedia Event Logs
• Serverless Functions

Spring projects used:

• Spring Boot
• Spring Cloud Netflix
• Spring Cloud Stream
• Spring Cloud Data Flow
• Spring Statemachine

Architecture

Each microservice in this reference architecture breaks down into three different independently deployable components.

The diagram above details the system architecture of the bounded context for Accounts, which includes deployable units for each Backing Service, Microservice, and AWS Lambda Function.

Usage

The repository contains two parent projects. Each microservice team is responsible for two separate component modules. The first component is the HTTP-driven web service for managing domain entities through a hypermedia-based REST API. The second component is an AMQP-driven event processor that drives the state of domain aggregates.

Deployment Model

Each microservice team will be managing two separate deployment artifacts, one for HTTP-based interactions and one for AMQP-based interactions.

The components break down into two separate concerns: Web and Worker roles.

Web role

The web role contains a REST API with transactions that requires strong consistency. Any workflow that can be transacted within the boundaries of the single microservice can be done within the scope of a single HTTP request/response.

Worker role

The worker role contains an AMQP-based message processor that listens for events generated by the web role and from other microservices. The worker role will be responsible for managing workflows that are eventually consistent. Any transaction the requires complex durable communication between separate microservices will be handled in the worker role.

Packaging

The deployment packaging for each microservice can vary depending on the desired size of the infrastructure footprint. You can either run both workloads as independently scalable components, or alternatively, you can package the components together into a single container.

Single Container Deployment

The first style of packaging is to combine together the HTTP (web role) and AMQP (worker role) components into a single container. The advantages with this approach have to do with minimizing the infrastructure footprint for operating the microservice.

Benefits:

• Run workloads side-by-side in a single container
• One microservice deployment per container
• Reduces infrastructure footprint

Trade-offs:

• Scales web and worker instances together
• Tight coupling of components prevents independent release cycles

Multi Container Deployment

The second style of deployment is to independently package the web application separate from the worker application. This approach provides multiple advantages at the cost of a larger and more complex infrastructure footprint.

Benefits:

• Allows separate release cycles for web and worker roles
• Independently scale web and worker application instances
• Changes to one component does not require deployment of other
• Reduces overall memory consumption

Trade-offs:

• Requires more application instances
• Higher operational complexity
• Requires two independent CI/CD pipelines

Running the Example

This repository is in an experimental state. To build and run the experimental project, run the following command:

mvn clean install -DskipTests -DskipDockerBuild

You will need to run Apache Kafka, Apache Zookeeper, and Redis on your local machine. After the build process has completed, you can start the Spring Cloud Data Flow server on your local machine, which will orchestrate each one of the microservice applications.

Before starting Spring Cloud Data Flow, make sure you're running each one of the backing services, as stated above. Also, you will need to start the Eureka server.

cd ./platform-services/discovery
mvn spring-boot:run

Once Eureka has started, you can start up the Spring Cloud Data Flow server, which will bootstrap the Spring Cloud Stream modules.

From the root of the project:

cd ./platform-services/data-flow-server
mvn spring-boot:run

The Spring Cloud Stream applications will be imported and the event streams will be available in the streams section of the data flow server. The load simulator stream module will begin to slowly pump events into the system and you'll be able to see the activity in the analytics section where you can see counters measuring the event load over time.

This project is working off of snapshots, so there may be issued running the data flow server. Please submit a issue or a pull request with a fix. (Thanks!)

Serverless Functions

Each bounded context in this reference architecture contains a set of action-mapped functions that can be deployed as Serverless functions. To learn more about how this works in practice, please take a look at the documentation for the account-worker application.

License

This project is an open source product licensed under GPLv3.