Intel-bigdata / Hpnl
Labels
Projects that are alternatives of or similar to Hpnl
HPNL
HPNL is a Fast, CPU-Efficient network library designed for modern network technology(#1 #2)
Contents
Introduction
Nowadays data is growing at a faster rate than ever before and large-scale data analytics present new challenge. HPNL is a light-weight network library built on Libfabric for big data application. It provides C/JAVA API and high level abstraction to let developer easily replace other TCP/IP based network library, like ASIO or Netty, without knowing the low level details of RDMA programing model. Ease to use API, functionality and performance is of primary design concern.
High level design
Installation
Build prerequisites
Library dependencies:
- C++ 11
- Libfabric 1.6+
- Boost 1.58+
- PMDK (optional)
- JDK 1.8+ (optional)
- FlatBuffers (optional)
HPNL depends on Libfabri. Please make sure the Libfabric is installed in your setup. Persistent Memory over Fabric is another targeting feature of HPNL with which you can use HPNL interface to communicate with Persistent Memory over different kinds of network fabrics. If you want to try the PMoF example, please install PMDK, a library to manage and access persistent memory devices.
On Ubuntu/Debian, install other requirements with:
sudo apt-get install cmake \
libboost-dev \
libboost-system-dev \
Build for C/C++
This project supports CMake out of box.
git clone https://github.com/Intel-bigdata/HPNL.git
cd HPNL
git submodule update --init --recursive
mkdir build; cd build;
By default, HPNL runs over TCP/IP protocol, in this mode, you may expect similar performance to other sockets based network library. Thanks to Libfabric, HPNL also supports RDMA, Omni-Path protocol, you can switch between different protocols by adding cmake build option as follows.
cmake -DWITH_VERBS=ON .. // for RDMA Protocol
cmake -DWITH_PSM2=ON .. // for Omni-Path Protocol
make && make install
If you install LibFabric or FlatBuffers in a custom directory you can point them using :
cmake -DLIBFABRIC_PREFIX=..... -DFLATBUFFERS_PREFIX=.....
Build for Java
The HPNL Java API comes with HPNL C/C++ shared library, the native code won't be included without explicit cmake option.
cmake -DWITH_JAVA=ON .. // to include native code for Java API
make && make install
cd ${project_root_path}/java/hpnl; mvn package
Test
cd ${project_root_path}/build
ctest // cmake ctest is binded to Catch2 unit-test framework
Usage
To get started, refer to the mini tutorial. Simple C/S examples can be found in HPNL/examples directory.