Pravega and Analytics Connectors Examples

This repository contains code samples to demonstrate how developers can work with Pravega. We also provide code samples to connect analytics engines such as Flink and Hadoop with Pravega as a storage substrate for data streams. We also provide samples for using new pravega schema registry with pravega applications.

For more information on Pravega, we recommend to read the documentation and the developer guide.

Repository Structure

This repository is divided into sub-projects (pravega-client-examples, flink-connector-examples, hadoop-connector-examples, schema-registry-examples, and spark-connector-examples), each one addressed to demonstrate a specific component. In these sub-projects, we provide a battery of simple code examples aimed at illustrating how a particular feature or API works. Moreover, we also include a scenarios folder that contains more complex applications as sub-projects, which show use-cases exploiting one or multiple components.

Hint: Have a look to the terminology and concepts in Pravega.

Pravega Client Examples

The related documentation and instructions are here.

Flink Connector Examples

The related documentation and instructions are here.

Hadoop Connector Examples

The related documentation and instructions are here.

Scenarios

Schema Registry Examples

The prerequisite for running Schema Registry Examples is to deploy Pravega and Schema Registry Service. For instructions to run pravga schema registry, please see instructions here

The related documentation and instructions are here.

Spark Connector Examples

The related documentation and instructions are here.

Build Instructions

Next, we provide instructions for building the pravega-samples repository. There are two main options:

• Out-of-the-box: If you want a quick start, run the samples by building pravega-samples out-of-the-box (go straight to section Pravega Samples Build Instructions).
• Build from source: If you want to have fun building the different projects from source, please read section Building Pravega Components from Source (Optional) before building pravega-samples.

Pre-requisites

• Java 11
• Python 3.8 (if you wish to run the python examples)

Building Pravega Components from Source (Optional)

Pravega Build Instructions

If you want to build Pravega from source, you may need to generate the latest Pravega jar files and install them to your local Maven repository. To build Pravega from sources and use it here, please run the following commands:

$ git clone https://github.com/pravega/pravega.git
$ cd pravega
$ ./gradlew install

The above command should generate the required jar files into your local Maven repository.

Hint: For using in the sample applications the Pravega version you just built, you need to update the pravegaVersion=<local_maven_pravega_version> property in gradle.properties file of pravega-samples.

For more information, please visit Pravega.

Flink Connector Build Instructions

To build the Flink connector from source, follow the below steps to build and publish artifacts from source to local Maven repository:

$ git clone --recursive https://github.com/pravega/flink-connectors.git
$ cd flink-connectors
$ ./gradlew install

Hint: For using in the sample applications the Flink connector version you just built, you need to update the flinkConnectorVersion=<local_maven_flink_connector_version> property in gradle.properties file of pravega-samples.

For more information, please visit Flink Connectors.

Hadoop Connector Build Instructions

To build the Hadoop connector from source, follow the below steps to build and publish artifacts from source to local Maven repository:

$ git clone --recurse-submodules https://github.com/pravega/hadoop-connectors.git
$ cd hadoop-connectors
$ ./gradlew install

Hint: For using in the sample applications the Hadoop connector version you just built, you need to update the hadoopConnectorVersion=<local_maven_hadoop_connector_version> property in gradle.properties file of pravega-samples.

For more information, please visit Hadoop Connectors.

Schema Registry Build Instructions

Schema registry uses pravega, so make sure pravega is installed and running before installing schema registry. To build Schema Registry from source, follow the below steps to build and publish artifacts from source to local Maven repository:

$ git clone https://github.com/pravega/schema-registry.git
$ cd schema-registry
$ ./gradlew install
$ cd server/build/install/schema-registry
$ # edit conf/schema-registry.config.properties to point to pravega URI (hint: if you are running pravega standalone, it would be tcp://localhost:9090) 
$ ./bin/schema-registry

For more information, please visit Schema Registry.

Configuring Pravega Samples for Running with Source Builds

In the previous instructions, we noted that you will need to change the gradle.properties file in pravega-samples for using the Pravega components built from source. Here we provide an example of how to do so:

1. Imagine that we want to build Pravega from source. Let us assume that we executed git clone https://github.com/pravega/pravega.git and the last commit of master branch is 2990193xxx.

2. After executing ./gradlew install, we will see in our local Maven repository (e.g., ~/.m2/repository/io/pravega/*) artifacts that contain in their names that commit version such as 0.3.0-1889.2990193-SNAPSHOT. These artifacts are the result from building Pravega from source.

3. The only thing you have to do is to set pravegaVersion=0.3.0-1889.2990193-SNAPSHOT in the gradle.properties file of pravega-samples.

While this example is for Pravega, the same procedure applies for Flink and Hadoop connectors.

Pravega Samples Build Instructions

The pravega-samples project is prepared for working out-of-the-box with release artifacts of Pravega components, which are already available in Maven central. To build pravega-samples from source, use the built-in gradle wrapper as follows:

$ git clone https://github.com/pravega/pravega-samples.git
$ cd pravega-samples
$ ./gradlew clean installDist

That's it! You are good to go and execute the examples :)

To ease their execution, most examples can be run either using the gradle wrapper (gradlew) or scripts. The above gradle command automatically creates the execution scripts that can be found under:

pravega-samples/pravega-client-examples/build/install/pravega-client-examples/bin

There is a Linux/Mac script and a Windows (.bat) script for each separate executable.

Working with dev branch: If you are curious about the most recent sample applications, you may like to try the dev version of pravega-samples as well. To do so, just clone the dev branch instead of master (default):

$ git clone -b dev https://github.com/pravega/pravega-samples.git
$ cd pravega-samples
$ ./gradlew clean installDist

The dev branch works with Pravega snapshots artifacts published in our JFrog repository instead of using release versions.

Proposed Roadmap

We propose a roadmap to proceed with the execution of examples based on their complexity:

1. Pravega client examples: First step to understand the basics of Pravega and exercise the concepts presented in the documentation.
2. Flink connector examples: These examples show the basic functionality of the Flink connector for Pravega.
3. Hadoop connector examples: These examples show the basic functionality of the Hadoop connector for Pravega.
4. Scenarios: Applications that go beyond the basic usage of Pravega APIs, which may include complex interactions between Pravega and analytics engines (e.g., Flink, Hadoop, Spark) to demonstrate analytics use cases.

Where to Find Help

Documentation on Pravega and Analytics Connectors:

• Pravega.io, Pravega Wiki.
• Flink Connectors Wiki.

Did you find a problem or bug?

• First, check our FAQ.
• If the FAQ does not help you, create a new GitHub issue.

Do you want to contribute a new example application?

• Follow the guidelines for contributors.

Have fun!!