triton-inference-server / Server

Licence: other
The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Programming Languages

C++
36643 projects - #6 most used programming language
python
139335 projects - #7 most used programming language
shell
77523 projects
CMake
9771 projects
Roff
2310 projects
Smarty
1635 projects

Projects that are alternatives of or similar to Server

Model server
A scalable inference server for models optimized with OpenVINO™
Stars: ✭ 431 (-85.6%)
Mutual labels:  cloud, inference, edge
Gpu Rest Engine
A REST API for Caffe using Docker and Go
Stars: ✭ 412 (-86.24%)
Mutual labels:  gpu, inference
Trainyourownyolo
Train a state-of-the-art yolov3 object detector from scratch!
Stars: ✭ 399 (-86.67%)
Mutual labels:  gpu, inference
Iot Edge V1
Azure IoT Edge
Stars: ✭ 522 (-82.57%)
Mutual labels:  cloud, edge
Bmw Tensorflow Inference Api Gpu
This is a repository for an object detection inference API using the Tensorflow framework.
Stars: ✭ 277 (-90.75%)
Mutual labels:  gpu, inference
Deep Learning In Cloud
List of Deep Learning Cloud Providers
Stars: ✭ 298 (-90.05%)
Mutual labels:  cloud, gpu
Practical Deep Learning Book
Official code repo for the O'Reilly Book - Practical Deep Learning for Cloud, Mobile & Edge
Stars: ✭ 441 (-85.27%)
Mutual labels:  cloud, edge
motor-defect-detector-python
Predict performance issues with manufacturing equipment motors. Perform local or cloud analytics of the issues found, and then display the data on a user interface to determine when failures might arise.
Stars: ✭ 24 (-99.2%)
Mutual labels:  inference, edge
Paperspace Python
Paperspace API for python
Stars: ✭ 38 (-98.73%)
Mutual labels:  cloud, gpu
People Counter Python
Create a smart video application using the Intel Distribution of OpenVINO toolkit. The toolkit uses models and inference to run single-class object detection.
Stars: ✭ 62 (-97.93%)
Mutual labels:  inference, edge
Keylime
A CNCF Project to Bootstrap & Maintain Trust on the Edge / Cloud and IoT
Stars: ✭ 128 (-95.72%)
Mutual labels:  cloud, edge
object-flaw-detector-python
Detect various irregularities of a product as it moves along a conveyor belt.
Stars: ✭ 17 (-99.43%)
Mutual labels:  inference, edge
concurrent-video-analytic-pipeline-optimization-sample-l
Create a concurrent video analysis pipeline featuring multistream face and human pose detection, vehicle attribute detection, and the ability to encode multiple videos to local storage in a single stream.
Stars: ✭ 39 (-98.7%)
Mutual labels:  inference, edge
K3sup
bootstrap Kubernetes with k3s over SSH < 1 min 🚀
Stars: ✭ 4,012 (+34%)
Mutual labels:  cloud, edge
safety-gear-detector-python
Observe workers as they pass in front of a camera to determine if they have adequate safety protection.
Stars: ✭ 54 (-98.2%)
Mutual labels:  inference, edge
Forward
A library for high performance deep learning inference on NVIDIA GPUs.
Stars: ✭ 136 (-95.46%)
Mutual labels:  gpu, inference
intruder-detector-python
Build an application that alerts you when someone enters a restricted area. Learn how to use models for multiclass object detection.
Stars: ✭ 16 (-99.47%)
Mutual labels:  inference, edge
object-size-detector-python
Monitor mechanical bolts as they move down a conveyor belt. When a bolt of an irregular size is detected, this solution emits an alert.
Stars: ✭ 26 (-99.13%)
Mutual labels:  inference, edge
Turbotransformers
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
Stars: ✭ 826 (-72.41%)
Mutual labels:  gpu, inference
Gpu monitor
Monitor your GPUs whether they are on a single computer or in a cluster
Stars: ✭ 133 (-95.56%)
Mutual labels:  cloud, gpu

License

Triton Inference Server

LATEST RELEASE: You are currently on the main branch which tracks under-development progress towards the next release.

Triton Inference Server provides a cloud and edge inferencing solution optimized for both CPUs and GPUs. Triton supports an HTTP/REST and GRPC protocol that allows remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton is available as a shared library with a C API that allows the full functionality of Triton to be included directly in an application.

The current release of the Triton Inference Server is 2.16.0 and corresponds to the 21.11 release of the tritonserver container on NVIDIA GPU Cloud (NGC). The branch for this release is r21.11.

Features

  • Deep learning frameworks. Triton supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch TorchScript and OpenVINO model formats. Both TensorFlow 1.x and TensorFlow 2.x are supported. Triton also supports TensorFlow-TensorRT, ONNX-TensorRT and PyTorch-TensorRT integrated models.

  • Machine learning frameworks. Triton supports popular machine learning frameworks such as XGBoost, LightGBM, Scikit-Learn and cuML using the RAPIDS Forest Inference Library.

  • Concurrent model execution. Triton can simultaneously run multiple models (or multiple instances of the same model) using the same or different deep-learning and machine-learning frameworks.

  • Dynamic batching. For models that support batching, Triton implements multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.

  • Extensible backends. In addition to deep-learning frameworks, Triton provides a backend API that allows Triton to be extended with any model execution logic implemented in Python or C++, while still benefiting from full CPU and GPU support, concurrent execution, dynamic batching and other features provided by Triton.

  • Model pipelines using Ensembling or Business Logic Scripting (BLS). A Triton ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. BLS allows a pipeline along with extra business logic to be represented in Python. In both cases a single inference request will trigger the execution of the entire pipeline.

  • HTTP/REST and GRPC inference protocols based on the community developed KServe protocol.

  • A C API allows Triton to be linked directly into your application for edge and other in-process use cases.

  • Metrics indicating GPU utilization, server throughput, and server latency. The metrics are provided in Prometheus data format.

Documentation

The master branch documentation tracks the upcoming, under-development release and so may not be accurate for the current release of Triton. See the r21.11 documentation for the current release.

Triton Architecture gives a high-level overview of the structure and capabilities of the inference server. There is also an FAQ. Additional documentation is divided into user and developer sections. The user documentation describes how to use Triton as an inference solution, including information on how to configure Triton, how to organize and configure your models, how to use the C++ and Python clients, etc. The developer documentation describes how to build and test Triton and also how Triton can be extended with new functionality.

The Triton Release Notes and Support Matrix indicate the required versions of the NVIDIA Driver and CUDA, and also describe supported GPUs.

User Documentation

The quickstart walks you through all the steps required to install and run Triton with an example image classification model and then use an example client application to perform inferencing using that model. The quickstart also demonstrates how Triton supports both GPU systems and CPU-only systems.

The first step in using Triton to serve your models is to place one or more models into a model repository. Optionally, depending on the type of the model and on what Triton capabilities you want to enable for the model, you may need to create a model configuration for the model. If your model has custom operations you will need to make sure they are loaded correctly by Triton.

After you have your model(s) available in Triton, you will want to send inference and other requests to Triton from your client application. The Python and C++ client libraries provide APIs to simplify this communication. There are also a large number of client examples that demonstrate how to use the libraries. You can also send HTTP/REST requests directly to Triton using the HTTP/REST JSON-based protocol or generate a GRPC client for many other languages.

Understanding and optimizing performance is an important part of deploying your models. The Triton project provides the Performance Analyzer and the Model Analyzer to help your optimization efforts. Specifically, you will want to optimize scheduling and batching and model instances appropriately for each model. You can also enable cross-model prioritization using the rate limiter which manages the rate at which requests are scheduled on model instances. You may also want to consider combining multiple models and pre/post-processing into a pipeline using ensembling or Business Logic Scripting (BLS). A Prometheus metrics endpoint allows you to visualize and monitor aggregate inference metrics.

NVIDIA publishes a number of deep learning examples that use Triton.

As part of your deployment strategy you may want to explicitly manage what models are available by loading and unloading models from a running Triton server. If you are using Kubernetes for deployment there are simple examples of how to deploy Triton using Kubernetes and Helm: GCP, AWS, and NVIDIA FleetCommand

The version 1 to version 2 migration information is helpful if you are moving to version 2 of Triton from previously using version 1.

Developer Documentation

Triton can be built using Docker or built without Docker. After building you should test Triton.

It is also possible to create a Docker image containing a customized Triton that contains only a subset of the backends.

The Triton project also provides client libraries for Python and C++ that make it easy to communicate with the server. There are also a large number of example clients that demonstrate how to use the libraries. You can also develop your own clients that directly communicate with Triton using HTTP/REST or GRPC protocols. There is also a C API that allows Triton to be linked directly into your application.

A Triton backend is the implementation that executes a model. A backend can interface with a deep learning framework, like PyTorch, TensorFlow, TensorRT or ONNX Runtime; or it can interface with a data processing framework like DALI; or you can extend Triton by writing your own backend in either C/C++ or Python.

A Triton repository agent extends Triton with new functionality that operates when a model is loaded or unloaded. You can introduce your own code to perform authentication, decryption, conversion, or similar operations when a model is loaded.

Papers and Presentation

Contributing

Contributions to Triton Inference Server are more than welcome. To contribute make a pull request and follow the guidelines outlined in CONTRIBUTING.md. If you have a backend, client, example or similar contribution that is not modifying the core of Triton, then you should file a PR in the contrib repo.

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:

  • minimal – use as little code as possible that still produces the same problem

  • complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it

  • verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].