Flume Ng Pulsar Sink

This is a Flume Sink implementation that can publish data to a Pulsar topic

Compatibility

This sink is developed and tested using Apache Flume NG 1.9.0 and Apache Pulsar Client 2.3.0.

Requirements

• Docker

Clone the project

$ git clone https://github.com/streamnative/flume-ng-pulsar-sink.git

Start Pulsar Standalone

docker pull apachepulsar/pulsar:2.3.0
docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-flume-standalone apachepulsar/pulsar:2.3.0 bin/pulsar standalone

Start Pulsar Consumer

Start a consumer to consume messages from topic flume-test-topic.

docker cp src/test/python/pulsar-flume.py pulsar-flume-standalone:/pulsar
docker exec -it pulsar-flume-standalone /bin/bash
python pulsar-flume.py

Setup up Flume

Prepare Build Environment

Open a new terminal to start a docker instance flume of maven:3.6-jdk-8 in the same network as pulsar-flume-standalone we started at previous step. We will use this flume docker instace to install Flume and Flume-Ng-Pulsar-Sink.

docker pull maven:3.6-jdk-8
docker run -d -it --link pulsar-flume-standalone -p 44445:44445 --name flume maven:3.6-jdk-8 /bin/bash

Install Flume

Go to the docker instance flume

docker exec -it flume /bin/bash

At flume instance:

wget http://apache.01link.hk/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
tar -zxvf apache-flume-1.9.0-bin.tar.gz

Install Pulsar Sink

At flume instance:

git clone https://github.com/streamnative/flume-ng-pulsar-sink
cd flume-ng-pulsar-sink
mvn clean package
cd ..
cp flume-ng-pulsar-sink/target/flume-ng-pulsar-sink-1.9.0.jar apache-flume-1.9.0-bin/lib/
exit

Configure Flume

Copy the example configurations to flume:

• flume-example.conf
• flume-env.sh

docker cp src/test/resources/flume-example.conf flume:/apache-flume-1.9.0-bin/conf/
docker cp src/test/resources/flume-env.sh flume:/apache-flume-1.9.0-bin/conf/

Start Flume Ng Agent

docker exec -it flume /bin/bash

At flume instance:

apache-flume-1.9.0-bin/bin/flume-ng agent --conf apache-flume-1.9.0-bin/conf/ -f apache-flume-1.9.0-bin/conf/flume-example.conf -n a1

Send Data

Open another terminal, send data to port 44445 of flume

➜  ~ telnet localhost 44445
Trying ::1...
Connected to localhost.
Escape character is '^]'.
hello
OK
world
OK

At the terminal running pulsar-consumer.py, you will see following output:

'eceived message: 'hello
'eceived message: 'world

Cleanup

flume and pulsar-flume-standalone are running at background. Please remember to kill them at the end of this tutorial.

$ docker ps | grep pulsar-flume-standalone | awk '{ print $1 }' | xargs docker kill
$ docker ps | grep flume | awk '{ print $1 }' | xargs docker kill

Installation

Requirements

• JDK 1.8+
• Apache Maven 3.x

Build from Source

Clone the project from Github:

$ git clone https://github.com/streamnative/flume-ng-pulsar-sink.git

Building the Flume Ng Sink using maven:

$ cd flume-ng-pulsar-sink
$ mvn clean package

Once it is built successfully, you will find a jar flume-ng-pulsar-sink-<version>.jar generated under target directory. You can drop the built jar at your flume installation under lib directory.

Usage

Configurations

Client

Producer

License

This project is licensed under the Apache License 2.0.