Wizaron / Pytorch Cpp Inference
Serving PyTorch 1.0 Models as a Web Server in C++
Stars: ✭ 194
Programming Languages
cpp
1120 projects
Projects that are alternatives of or similar to Pytorch Cpp Inference
Bmw Labeltool Lite
This repository provides you with a easy to use labeling tool for State-of-the-art Deep Learning training purposes.
Stars: ✭ 145 (-25.26%)
Mutual labels: inference
Bmw Yolov4 Inference Api Cpu
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
Stars: ✭ 180 (-7.22%)
Mutual labels: inference
Mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Stars: ✭ 15,338 (+7806.19%)
Mutual labels: inference
Bmw Tensorflow Inference Api Cpu
This is a repository for an object detection inference API using the Tensorflow framework.
Stars: ✭ 158 (-18.56%)
Mutual labels: inference
Ml privacy meter
Machine Learning Privacy Meter: A tool to quantify the privacy risks of machine learning models with respect to inference attacks, notably membership inference attacks
Stars: ✭ 167 (-13.92%)
Mutual labels: inference
Forward
A library for high performance deep learning inference on NVIDIA GPUs.
Stars: ✭ 136 (-29.9%)
Mutual labels: inference
Hey Jetson
Deep Learning based Automatic Speech Recognition with attention for the Nvidia Jetson.
Stars: ✭ 161 (-17.01%)
Mutual labels: inference
Object Detection Api
Yolov3 Object Detection implemented as APIs, using TensorFlow and Flask
Stars: ✭ 177 (-8.76%)
Mutual labels: inference
Tensorflow template application
TensorFlow template application for deep learning
Stars: ✭ 1,851 (+854.12%)
Mutual labels: inference
Bert Ner Tf
Named Entity Recognition with BERT using TensorFlow 2.0
Stars: ✭ 155 (-20.1%)
Mutual labels: inference
Ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
Stars: ✭ 13,376 (+6794.85%)
Mutual labels: inference
Adversarial Robustness Toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
Stars: ✭ 2,638 (+1259.79%)
Mutual labels: inference
Onnxt5
Summarization, translation, sentiment-analysis, text-generation and more at blazing speed using a T5 version implemented in ONNX.
Stars: ✭ 143 (-26.29%)
Mutual labels: inference
Server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Stars: ✭ 2,994 (+1443.3%)
Mutual labels: inference
Sparktorch
Train and run Pytorch models on Apache Spark.
Stars: ✭ 195 (+0.52%)
Mutual labels: inference
Torch2trt
An easy to use PyTorch to TensorRT converter
Stars: ✭ 2,974 (+1432.99%)
Mutual labels: inference
Serving PyTorch Models in C++
- This repository contains various examples to perform inference using PyTorch C++ API.
- Run
git clone https://github.com/Wizaron/pytorch-cpp-inference
in order to clone this repository.
Environment
- Dockerfiles can be found at
docker
directory. There are two dockerfiles; one for cpu and the other for cuda10. In order to build docker image, you should go todocker/cpu
ordocker/cuda10
directory and rundocker build -t <docker-image-name> .
. - After creation of the docker image, you should create a docker container via
docker run -v <directory-that-this-repository-resides>:<target-directory-in-docker-container> -p 8181:8181 -it <docker-image-name>
(We will use 8181 to serve our PyTorch C++ model). - Inside docker container, go to the directory that this repository resides.
- Download
libtorch
from PyTorch Website (CPU :https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.3.1%2Bcpu.zip
- CUDA10 :https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.3.1.zip
). - Unzip libtorch via
unzip
. This will createlibtorch
directory that contains torch shared libraries and headers.
Code Structure
-
models
directory stores PyTorch models. -
libtorch
directory stores C++ torch headers and shared libraries to link the model against PyTorch. -
utils
directory stores various utility function to perform inference in C++. -
inference-cpp
directory stores codes to perform inference.
Exporting PyTorch ScriptModule
- In order to export
torch.jit.ScriptModule
of ResNet18 to perform C++ inference, go tomodels/resnet
directory and runpython3 resnet.py
. It will download pretrained ResNet18 model on ImageNet and createmodels/resnet_model_cpu.pth
and (optionally)models/resnet_model_gpu.pth
which we will use in C++ inference.
Serving the C++ Model
- We can either serve the model as a single executable or as a web server.
Single Executable
- In order to build a single executable for inference:
- Go to
inference-cpp/cnn-classification
directory. - Run
./build.sh
in order to build executable, named aspredict
. - Run the executable via
./predict <path-to-image> <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}>
. - Example:
./predict image.jpeg ../../models/resnet/resnet_model_cpu.pth ../../models/resnet/labels.txt false
- Go to
Web Server
- In order to build a web server for production:
- Go to
inference-cpp/cnn-classification/server
directory. - Run
./build.sh
in order to build web server, named aspredict
. - Run the binary via
./predict <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}>
(It will serve the model onhttp://localhost:8181/predict
). - Example:
./predict ../../../models/resnet/resnet_model_cpu.pth ../../../models/resnet/labels.txt false
- In order to make a request, open a new tab and run
python test_api.py
(It will make a request tolocalhost:8181/predict
).
- Go to
Acknowledgement
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].