Tencent / Forward
Projects that are alternatives of or similar to Forward
Forward - A library for high performance deep learning inference on NVIDIA GPUs
[中文版]
Forward
Forward is a library for high performance deep learning inference on NVIDIA GPUs. It provides a well-designed scheme that directly parse Tensorflow/PyTorch/Keras models to high-performance engine based on TensorRT. Compared to TensorRT, it is easy-to-use and easy-to-expand. So far, Forward supports not only mainstream deep learning models in CV, NLP and Recommend fields, but also some advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
Features
- Utilize TensorRT API and customized operators for high-performance deep learning inference.
- Support not only mainstream deep learning models in CV, NLP and Recommend fields, but also advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
- Support FLOAT/HALF/INT8 infer modes.
- Easy to use: Load directly Tensorflow(.pb)/PyTorch(.pth)/Keras(.h5) models and then do inference with TensorRT.
- Easy to expand: Register customized layers refer to add_support_op.md.
- Provide C++ and Python interfaces.
Quick Start
Prerequisites
- NVIDIA CUDA >= 10.0, CuDNN >= 7 (Recommended version: CUDA 10.2 )
- TensorRT >= 7.0.0.11, (Recommended version: TensorRT-7.2.1.6)
- CMake >= 3.10.1
- GCC >= 5.4.0, ld >= 2.26.1
- (Pytorch) pytorch == 1.3.1
- (Tensorflow) TensorFlow == 1.15.0 (download Tensorflow 1.15.0 and unzip it to
source/third_party/tensorflow/lib) - (Keras) HDF 5
Build with CMake
Generate Makefiles or VS project (Windows) and build. Forward can be built for different framework, such as Fwd-Torch, Fwd-Python-Torch, Fwd-Tf, Fwd-Python-Tf, Fwd-Keras, Fwd-Python-Keras, which controlled by CMake options. For example, Fwd-Python-Tf is built as below.
mkdir build
cd build
cmake .. \
-DTensorRT_ROOT=/path/to/TensorRT \
-DENABLE_LOGGING=ON \
-DENABLE_PROFILING=ON \
-DENABLE_DYNAMIC_BATCH=ON \
-DBUILD_PTYHON_LIB=ON \
-DENABLE_TORCH=OFF \
-DENABLE_TENSORFLOW=ON \
-DENABLE_KERAS=OFF \
make -j
CMake build arguments
-
TensorRT_ROOT[Required]: Path to the TensorRT installation directory containing libraries - More CMake options refer to CMake Options
Unit Test
When the project is built, unit_test can be used to verify the project is successfully built.
cd build/bin
./unit_test --gtest_filter=TestTfNodes.*
Use Forward-Cpp
Refer to Demo for using Forward-Cpp in Linux
Use Forward-Python
Refer to Demo for using Forward-Python
More Usages
Notice: The name of INPUT in models can be viewed by model viewers, such as Netron.
FAQ
Models & Operators
Models
Operators
Contribution
Contributors
 Aster JIAN |
 Zexi YUAN |
 Ao LI |
 Paul LU |
 JettHu |
 Ryosuke1eep |
Any form of contribution is welcome. The above contributors have been officially released by Tencent.
We very much welcome developers to contribute to Tencent's open source, and we will also give them incentives to acknowledge and thank them. Here we provide an official description of Tencent's open source contribution. Specific contribution rules for each project are formulated by the project team. Developers can choose the appropriate project and participate according to the corresponding rules. The Tencent Project Management Committee will report regularly to qualified contributors and awards will be issued by the official contact.