garystafford / virtual-vehicles-demo

Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.

PROJECT CODE UPDATED 11-09-2016 to v3.0.0

Note: All code available on GitHub. For the version of code that matches the details in this blog post, checkout v1.0 tag (after running git clone ..., run a 'git checkout tags/v1.0' command).

Previous Post

In Part One, we decided on the proper amount and responsibility of each microservice. We also determined the functionality of each microservice to meet the hypothetical functional and nonfunctional requirements of Virtual-Vehicles. To review, the four microservices we are building, are as follows:

All services also have an internal authenticateJwt function, to authenticate the JWT, passed in the HTTP request header, before performing any operation. Additionally, all services have a basic health-check function, ping (GET). There are only a few other functions required for our Virtual-Vehicles example, such as for creating JWTs.

 Part Two Introduction

In this post, we will accomplish the following:

• Create a default microservice project in NetBeans using RestExpress MongoDB Maven Archetype
• Understand the basic structure of a default RestExpress microservice project
• Review the changes made to the default RestExpress microservice project to create the Virtual-Vehicles example
• Compile and run the individual microservices directly from NetBeans

Using the RestExpress MongoDB Maven Archetype

Default Project Architecture

• Base Package (com.example.vehicle)
  • Configuration class reads in environment properties (see Files tab) and instantiates controllers
  • Constants class contains project constants
  • Relationships class defines linking resource which aids service discoverability (HATEOAS)
  • Main executable class
  • Routes class defines the routes (endpoints) exposed by the service and the corresponding controller class
• Model/Controllers Packages (com.example.vehicle.objectid and .uuid)
  • Entity class defines the data entity – a Vehicle in this case
  • Controller class contains the methods executed when the route (endpoint) is called
  • Repository class defines the connection details to MongoDB
  • Service class contains the calls to the persistence layer, validation, and business logic
• Serialization Package (com.example.vehicle.serialization)
  • XML and JSON serialization classes
  • Object ID and UUID serialization and deserialization classes

Again, I strongly recommend reviewing each of these package's classes. To understand the core functionality of RestExpress, you must understand the relationships between RestExpress microservice's Route, Controller, Service, Repository, Relationships, and Entity classes. In addition reviewing the default Maven project, there are limited materials available on the Internet. I would recommend the RestExpress Website on GitHub, RestExpress Google Group Forum, and the YouTube 3-part video series, Instant REST Services with RESTExpress.

Unit Tests? Disappointingly, the current RestExpress MongoDB Maven Archetype sample project does not come with sample JUnit unit tests. I am temped to start writing my own unit tests, if I decided to continue to use the RestExpress microservices framework for future projects.

Properties Files

Ports Within the 'dev' environment, each microservice is configured to start on separate ports  (i.e. port = 8581). Feel free to change the service's port mappings if they conflict with previously configured components running on your system. Be careful when changing the Authentication service's port, 8587, since this port is also mapped in all other microservices using the authentication.port property (authentication.port = 8587). Make sure you change both properties, if you change Authentication service's port mapping.

Base URL Also, in the properties files is the base.url property. This property defines the URL the microservice's endpoints will be expecting calls from, and making internal calls between services. In our post's example, this property in the 'dev' environment is set to localhost (base.url = http://localhost). You could map an alternate hostname from your hosts file (/etc/hosts). We will do this in a later post, in our 'prod' environment, mapping the base.url property to Virtual-Vehicles (base.url = http://virtual-vehicles.com). In the 'dev' environment properties file, MongoDB is also mapped to localhost (i.e. mongodb.uri = mongodb://virtual-vehicles.com:27017/virtual_vehicle).

Metrics Plugin and Graphite RestExpress also uses the properties file to hold configuration properties for Metrics Plugin and Graphite. The Metrics Plugin and Graphite are both first class citizens of RestExpress. Below is the copy of the Vehicles service environment.properties file for the 'dev' environment. Note, the Metrics Plugin and Graphite are both disabled in the 'dev' environment.

# Default is 8081
port = 8581

# Port used to call Authentication service endpoints
authentication.port = 8587

# The size of the executor thread pool
# (that can handle blocking back-end processing).
executor.threadPool.size = 20

# A MongoDB URI/Connection string
# see: http://docs.mongodb.org/manual/reference/connection-string/
mongodb.uri = mongodb://localhost:27017/virtual_vehicle

# The base URL, used as a prefix for links returned in data
base.url = http://localhost

#Configuration for the MetricsPlugin/Graphite
metrics.isEnabled = false
#metrics.machineName =
metrics.prefix = web1.example.com
metrics.graphite.isEnabled = false
metrics.graphite.host = graphite.example.com
metrics.graphite.port = 2003
metrics.graphite.publishSeconds = 60

Choosing a Data Identification Method

Project Modifications

• Remove all packages, classes, and code references to the UUID identification methods (example uses ObjectId)
• Rename several classes for convenience (dropped use of word 'Entity')
• Add the Utilities (com.example.utilities) and Authentication (com.example.authenticate) packages

MongoDB

From the MongoDB shell, we can observe the individual instances of the four microservice's databases.

In the case of the Vehicle microservice, the associated MongoDB database is virtual_vehicle. This database contains a single collection, vehicles. While the properties file defines the database name, the Vehicles entity class defines the collection name, using the org.mongodb.morphia.annotations classes annotation functionality.

@Entity("vehicles")
public class Vehicle
        extends AbstractMongodbEntity
        implements Linkable {

    private int year;
    private String make;
    private String model;
    private String color;
    private String type;
    private int mileage;
...

Looking at the virtual_vehicle database in the MongoDB shell, we see that the sample document's fields corresponds to the Vehicle entity classes properties.

Each of the microservice's MongoDB databases are configured in the environments.properties file, using the mongodb.uri property. In the 'dev' environment we use localhost as our host URL (i.e. mongodb.uri = mongodb://localhost:27017/virtual_vehicle).

Authentication and JSON Web Tokens

Why do we want authentication? We want to confirm that the requester using the Virtual-Vehicles REST API is the actual registered API client they are who they claim to be. JWT allows us to achieve this requirement with minimal effort.

According to jwt.io, 'a JSON Web Token is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).' I recommend reviewing the JWT draft standard to fully understand the structure, creation, and use of JWTs.

Virtual-Vehicles Authentication Process There are different approaches to implementing JWT. In our Virtual-Vehicles REST API example, we use the following process for JWT authentication:

1. Register the new API client by supplying the application name and a shared secret (one time only)
2. Receive an API key in response (one time only)
3. Obtain a JWT using the API key and the shared secret (each user session or renew when the previous JWT expires)
4. Include the JWT in each API call

Expiration and Issued date/time use the JWT standard recommended "Seconds Since the Epoch".  The default expiration for a Virtual-Vehicles JWT is set to an arbitrary 10 hours from time the JWT was issued (jwt.expire.length = 36000). That amount, 36,000, is equivalent to 10 hours x 60 minutes/hour x 60 seconds/minute.

Decoding a JWT Using the jwt.io site's JT Debugger tool, I have decoded a sample JWT issued by the Virtual-Vehicles REST API, generated by the Authentication service. Observe the three parts of the JWT, the JOSE Header, Claims Set, and the JSON Web Signature (JWS).

The JWT's header indicates that our JWT is a JWS that is MACed using the HMAC SHA-256 algorithm. The shared secret, passed by the API client, represents the HMAC secret cryptographic key. The secret is used in combination with the cryptographic hash function to calculate the message authentication code (MAC). In the example below, note how the API client's shared secret is used to validate our JWT's JWS.

Sequence Diagrams of Authentication Process Below are three sequence diagrams, which detail the following processes: API client registration process, obtaining a new JWT, and a REST call being authenticated using the JWT. The end-user of the API self-registers their application using the Authentication service, and receives back an API key. The API key is unique to that client.

The end-user application then uses the API key and the shared secret to receive a JWT from the Authentication service.

After receiving the JWT, the end-user application passes the JWT in header of each API request. The JWT is validated by calling the Authentication service. If the JWT is valid, the request is fulfilled. If not valid, a '401 Unauthorized' status is returned.

JWT Validation The JWT draft standard recommends how to validate a JWT. Our Virtual-Vehicles Authentication microservice uses many of those criteria to validate the JWT, which include:

1. API Key - Retrieve API client's shared secret from MongoDB, using API key contained in JWT's claims set (secret is returned, therefore API key is valid)
2. Algorithm - confirm the algorithm ('alg'), found in the JWT Header, which used to encode JWT, was 'HS256' (HMAC SHA-256)
3. Valid JWS - Use the client's shared secret from #1 above, decode HMAC SHA-256 encrypted JWS
4. Expiration - confirm JWT is not expired ('exp')

Inter-Service Communications

For the Virtual-Vehicles microservices example, we will use HTTP-based REST calls for inter-service communications. The primary service-to-service communications in our example, is between the three microservices, Vehicle, Valet, and Maintenance, and the Authentication microservice. The three services validate the JWT passed in each request to a CRUD operation, by calling the Authentication service and passing the JWT, as shown in the sequence diagram, above. This is done using an HTTP GET request to the Authentication service's .../jwts/{jwt} endpoint. The Authentication service's method, called by this endpoint, minus some logging and error handling, looks like the following:

public boolean authenticateJwt(Request request, String baseUrlAndAuthPort) {
    String jwt, output, valid = "";

    try {
        jwt = (request.getHeader("Authorization").split(" "))[1];
    } catch (NullPointerException | ArrayIndexOutOfBoundsException e) {
        return false;
    }

    try {
        URL url = new URL(baseUrlAndAuthPort + "/jwts/" + jwt);
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setRequestMethod("GET");
        conn.setRequestProperty("Accept", "application/json");

        if (conn.getResponseCode() != 200) {
            return false;
        }

        BufferedReader br = new BufferedReader(new InputStreamReader(
                (conn.getInputStream())));

        while ((output = br.readLine()) != null) {
            valid = output;
        }

        conn.disconnect();
    } catch (IOException e) {
        return false;
    }

    return Boolean.parseBoolean(valid);
}

Primarily, we are using the java.net and java.io packages, along with the org.restexpress.Request class to build and send our HTTP request to the Authentication service. Alternately, you could use just the org.restexpress package to construct request and handle the response. This same basic method structure shown above can be used to create unit tests for your service's endpoints.

Health Ping

public void ping(Request request, Response response) {
    response.setResponseStatus(HttpResponseStatus.OK);
    response.setResponseCode(200);
    response.setBody(true);
}

The ping health check can even be accessed with a simple curl command, curl localhost:8581/vehicles/utils/ping.

In a real-world application, we would add additional health checks to all services, providing additional insight into the health and performance of each microservice, as well as the service's dependencies.

API Documentation

An API should be documented before any code is written, similar to TDD. The API documentation is the result of a clear understanding of requirements. The API documentation should make the coding even easier, since the documentation serves as a further refinement of the requirements. The requirements are an architectural plan for the microservice's code structure.

Sample Documentation Below, is a sample of the Virtual-Vehicles REST API documentation. It details the function responsible for creating a new API client. The documentation provides a brief description of the function, the operation's endpoint (URI), HTTP method, request parameters, expected response body, expected response status, and even a view of the MongoDB collection's document for a new API client.

You can download a PDF version of the Virtual-Vehicles RESTful API documentation on GitHub or review the source document on Google Docs. It contains general notes about the API, and details about every one of the API's operations.

Running the Individual Microservices

Alternately, you can use the RestExpress project's recommended command from a terminal prompt to start each microservice from its root directory (mvn exec:java -Dexec.mainClass=test.Main -Dexec.args="dev"). You can also use this command to switch from the 'dev' to 'prod' environment properties (-Dexec.args="prod").

You may use a variety of commands to confirm all the microservices are running. I prefer something basic like sudo netstat -tulpn | grep 858[0-9]. This will find all the ports within the 'dev' port range. For more in-depth info, you can use a command like ps -aux | grep com.example | grep -v grep

Part Three: Testing our Services