FlexInfer
A flexible Python front-end inference SDK.
Features
-
Flexible
FlexInfer has a Python front-end, which makes it easy to build a computer vision product prototype.
-
Efficient
Most of time consuming part of FlexInfer is powered by C++ or CUDA, so FlexInfer is also efficient. If you are really hungry for efficiency and don't mind the trouble of C++, you can refer to CheetahInfer.
License
This project is released under Apache 2.0 license.
Installation
Requirements
- Linux
- Python 3.6 or higher
- TensorRT 7.1.3.4 or higher
- PyTorch 1.4.0 or higher
- CUDA 10.2 or higher
- volksdep 3.2.0 or higher
We have tested the following versions of OS and softwares:
- OS: Ubuntu 16.04.6 LTS
- Python 3.6.9
- TensorRT 7.1.3.4
- PyTorch 1.6.0
- CUDA: 10.2
- volksdep: 3.2.0
Install FlexInfer
- If your platform is x86 or x64, you can create a conda virtual environment and activate it.
conda create -n flexinfer python=3.6.9 -y
conda activate flexinfer-
Install volksdep following the official instructions
-
Setup
pip install "git+https://github.com/Media-Smart/flexinfer.git"Usage
We provide some examples for different tasks.
Throughput benchmark
- Device: Jetson AGX Xavier
- CUDA: 10.2
| Tasks | framework | version | input shape | data type | throughput(FPS) | latency(ms) | |
| Classification (ResNet18) | PyTorch | 1.5.0 | (1, 3, 224, 224) | FP16 | 172 | 6.01 | |
| TensorRT | 7.1.0.16 | (1, 3, 224, 224) | FP16 | 754 | 1.8 | ||
| Segmentation（U-Net） | PyTorch | 1.5.0 | (1, 3, 513, 513) | FP16 | 15 | 63.27 | |
| tensorrt | 7.1.0.16 | (1, 3, 513, 513) | FP16 | 29 | 34.03 | ||
| Object Detection | RetinaNet-R50 | PyTorch | 1.5.0 | (1, 3, 768, 1280) | FP16 | 8 | 118.79 |
| TensorRT | 7.1.0.16 | (1, 3, 768, 1280) | FP16 | 15 | 68.10 | ||
| TinaFace-R50-FPN-BN | PyTorch | 1.5.0 | (1, 3, 768, 1280) | FP16 | 3 | 273.60 | |
| TensorRT | 7.1.0.16 | (1, 3, 768, 1280) | FP16 | 6 | 159.70 | ||
| Scene Text Recognition (ResNet-CTC) | PyTorch | 1.5.0 | (1, 1, 32, 100) | FP16 | 113 | 10.75 | |
| TensorRT | 7.1.0.16 | (1, 1, 32, 100) | FP16 | 308 | 3.55 | ||
Media-Smart toolboxes
We provide some toolboxes of different tasks for training, testing and deploying.
-
Classification (vedacls)
-
Segmentation (vedaseg)
-
Object Detection (vedadet)
-
Scene Text Recognition (vedastr)
Contact
This repository is currently maintained by Yuxin Zou (@Yuxin Zou), Jun Sun(@ChaseMonsterAway), Hongxiang Cai (@hxcai) and Yichao Xiong (@mileistone).