twitter-archive / Bookkeeper

Build instructions for BookKeeper

Requirements:

• Unix System
• JDK 1.6
• Maven 3.0
• Autotools (if compiling native hedwig client)
• Internet connection for first build (to fetch all dependencies)

The BookKeeper project contains:

• bookkeeper-server (BookKeeper server and client)
• bookkeeper-benchmark (Benchmark suite for testing BookKeeper performance)
• hedwig-protocol (Hedwig network protocol)
• hedwig-client (Hedwig client library)
• hedwig-server (Hedwig server)

BookKeeper is a system to reliably log streams of records. It is designed to store write ahead logs, such as those found in database or database like applications.

Hedwig is a publish-subscribe system designed to carry large amounts of data across the internet in a guaranteed-delivery fashion from those who produce it (publishers) to those who are interested in it (subscribers).

How do I build?

BookKeeper uses maven as its build system. To build, run "mvn package" from the top-level directory, or from within any of the submodules.

Useful maven commands are:

• Clean : mvn clean
• Compile : mvn compile
• Run tests : mvn test
• Create JAR : mvn package
• Run findbugs : mvn compile findbugs:findbugs
• Install JAR in M2 cache : mvn install
• Deploy JAR to Maven repo : mvn deploy
• Run Rat : mvn apache-rat:check
• Build javadocs : mvn compile javadoc:aggregate
• Build distribution : mvn package assembly:single

Tests options:

• Use -DskipTests to skip tests when running the following Maven goals: 'package', 'install', 'deploy' or 'verify'
• -Dtest=,<TESTCLASSNAME#METHODNAME>,....
• -Dtest.exclude=
• -Dtest.exclude.pattern=/.java,/.java

NOTE:

BookKeeper uses maven-shade-plugin to build shade packages for old versions for backward compatibility testing. This shade plugin is only able to run at 'package' phase. So there are two ways to run bookkeeper tests:

• Run 'mvn clean package' under bookkeeper root directory
• Run 'mvn clean test' under bookkeeper/bookkeeper-server directory after you run 'mvn clean install -DskipTests' under bookkeeper directory

How do I run the services?

Running a Hedwig service requires a running BookKeeper service, which in turn requires a running ZooKeeper service (see http://zookeeper.apache.org). To start a bookkeeper service quickly for testing, run:

$ bookkeeper-server/bin/bookkeeper localbookie 10

This will start a standalone, ZooKeeper instance and 10 BookKeeper bookies. Note that this is only useful for testing. Data is not persisted between runs.

To start a real BookKeeper service, you must set up a ZooKeeper instance and run start a bookie on several machines. Modify bookkeeper-server/conf/bk_server.conf to point to your ZooKeeper instance. To start a bookie run:

$ bookkeeper-server/bin/bookkeeper bookie

Once you have at least 3 bookies runnings, you can start some Hedwig hubs. A hub is a machines which is responsible for a set of topics in the pubsub system. The service automatically distributes the topics among the hubs. To start a hedwig hub:

$ hedwig-server/bin/hedwig server

You can get more help on using these commands by running:

$ bookkeeper-server/bin/bookkeeper help and $ hedwig-server/bin/hedwig help