Isaac ROS DNN Inference
Webinar Available
Learn how to use this package by watching our on-demand webinar: Accelerate YOLOv5 and Custom AI Models in ROS with NVIDIA Isaac
Overview
This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom models. One node uses the TensorRT SDK, while the other uses the Triton SDK. This repository also contains a node to preprocess images, and convert them into tensors for use by TensorRT and Triton.
TensorRT
TensorRT is a library that enables faster inference on NVIDIA GPUs; it provides an API for the user to load and execute inference with their own models. The TensorRT ROS2 node in this package integrates with the TensorRT API, so the user has no need to make any calls to or directly use TensorRT SDK. Instead, users simply configure the TensorRT node with their own custom models and parameters, and the node will make the necessary TensorRT API calls to load and execute the model. For further documentation on TensorRT, refer to the main page here.
Triton
Triton is a framework that brings up a generic inference server that can be configured with a model repository, which is a collection of various types of models (e.g. ONNX Runtime, TensorRT Engine Plan, TensorFlow, PyTorch). A brief tutorial on how to set up a model repository is included below, and further documentation on Triton is also available here.
For more details about the setup of TensorRT and Triton, look here.
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
Performance
The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:
Pipeline | AGX Orin | Orin Nano | x86_64 w/ RTX 3060 Ti |
---|---|---|---|
PeopleSemSegNet (544p) | 260 fps 3.7ms |
128 fps 6.7ms |
300 fps 2ms |
These data have been collected per the methodology described here.
Table of Contents
- Isaac ROS DNN Inference
Latest Update
Update 2022-10-19: Updated OSS licensing
Supported Platforms
This package is designed and tested to be compatible with ROS2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.
Note: Versions of ROS2 earlier than Humble are not supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.
Platform | Hardware | Software | Notes |
---|---|---|---|
Jetson | Jetson Orin Jetson Xavier |
JetPack 5.0.2 | For best performance, ensure that power settings are configured appropriately. |
x86_64 | NVIDIA GPU | Ubuntu 20.04+ CUDA 11.6.1+ |
Docker
To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
Quickstart with Triton
Note: The quickstart helps with getting raw inference (tensor) results from the two nodes. To use the packages in a useful context and get meaningful results from the package, please refer here.
-
Set up your development environment by following the instructions here.
-
Clone this repository and its dependencies under
~/workspaces/isaac_ros-dev/src
.cd ~/workspaces/isaac_ros-dev/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference
-
Launch the Docker container using the
run_dev.sh
script:cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
-
For this example, we will use
PeopleSemSegNet ShuffleSeg
. Download the ETLT file and theint8
inference mode cache file:mkdir -p /tmp/models/peoplesemsegnet_shuffleseg/1 && \ cd /tmp/models/peoplesemsegnet_shuffleseg && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_etlt.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_cache.txt
-
Convert the ETLT file to a TensorRT plan file:
/opt/nvidia/tao/tao-converter -k tlt_encode -d 3,544,960 -p input_2:0,1x3x544x960,1x3x544x960,1x3x544x960 -t int8 -c peoplesemsegnet_shuffleseg_cache.txt -e /tmp/models/peoplesemsegnet_shuffleseg/1/model.plan -o argmax_1 peoplesemsegnet_shuffleseg_etlt.etlt
-
Create a file named
/tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt
by copying the sample Triton config file:cp /workspaces/isaac_ros-dev/src/isaac_ros_dnn_inference/resources/peoplesemsegnet_shuffleseg_config.pbtxt /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt
-
Inside the container, build and source the workspace:
cd /workspaces/isaac_ros-dev && \ colcon build --symlink-install && \ source install/setup.bash
-
(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential
-
Run the following launch files to spin up a demo of this package: Launch Triton:
ros2 launch isaac_ros_triton isaac_ros_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1']
In another terminal, enter the Docker container:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then, run a test node that sends tensors to Triton:
source install/setup.bash && \ ros2 run isaac_ros_dnn_inference_test run_test_publisher --ros-args -p dimensions:='[1, 3, 544, 960]'
-
Visualize and validate the output of the package:
In a third terminal, enter the Docker container:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then echo the inference result:
source install/setup.bash && \ ros2 topic echo /tensor_sub
The expected result should look like this:
header: stamp: sec: 0 nanosec: 0 frame_id: '' tensors: - name: output_tensor shape: rank: 4 dims: - 1 - 544 - 960 - 1 data_type: 5 strides: - 2088960 - 3840 - 4 - 4 data: [...]
Quickstart with TensorRT
Note: The quickstart helps with getting raw inference (tensor) results from the two nodes. To use the packages in a useful context and get meaningful results from the package, please refer here.
-
Follow steps 1-5 of the Quickstart with Triton
-
Inside the container, build and source the workspace:
cd /workspaces/isaac_ros-dev && \ colcon build --symlink-install && \ source install/setup.bash
-
(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential
-
Run the following launch files to spin up a demo of this package: Launch TensorRT:
ros2 launch isaac_ros_tensor_rt isaac_ros_tensor_rt.launch.py engine_file_path:=/tmp/models/peoplesemsegnet_shuffleseg/1/model.plan input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1']
In another terminal, enter the Docker container:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then, run a test node that sends tensors to TensorRT:
source install/setup.bash && \ ros2 run isaac_ros_dnn_inference_test run_test_publisher --ros-args -p dimensions:='[1, 3, 544, 960]'
-
Visualize and validate the output of the package: In a third terminal, enter the Docker container:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then echo the inference result:
source install/setup.bash && \ ros2 topic echo /tensor_sub
The expected result should look like this:
header: stamp: sec: 0 nanosec: 0 frame_id: '' tensors: - name: output_tensor shape: rank: 4 dims: - 1 - 544 - 960 - 1 data_type: 5 strides: - 2088960 - 3840 - 4 - 4 data: [...]
Note: For both the Triton and the TensorRT Quickstarts, the data contents are omitted. The
data_type
field has a value of5
, corresponding to theuint32
data type, but the data array will be output in the terminal asuint8
bytes.For further details about the
TensorList
message type, see here.To view the results as a segmentation mask, please look at Isaac ROS Image Segmentation.
Next Steps
Use Different Models
Click here for more information about how to use NGC models.
We also natively support the following packages to perform DNN inference in a variety of contexts:
Package Name | Use Case |
---|---|
DNN Stereo Disparity | Deep learned stereo disparity estimation |
Image Segmentation | Hardware-accelerated, deep learned semantic image segmentation |
Object Detection | Deep learning model support for object detection including DetectNet |
Pose Estimation | Deep learned, hardware-accelerated 3D object pose estimation |
Proximity Segmentation | DNN-based proximity segmentation and obstacle field ranging using Bi3D |
Customize your Dev Environment
To customize your development environment, reference this guide.
Package Reference
isaac_ros_dnn_encoders
ROS Parameters
ROS Parameter | Type | Default | Description |
---|---|---|---|
network_image_width |
uint16_t |
0 |
The image width that the network expects. This will be used to resize the input image width |
network_image_height |
uint16_t |
0 |
The image height that the network expects. This will be used to resize the input image height |
image_mean |
double list |
[0.5, 0.5, 0.5] |
The mean of the images per channel that will be used for normalization |
image_stddev |
double list |
[0.5, 0.5, 0.5] |
The standard deviation of the images per channel that will be used for normalization |
Note: the following parameters are no longer supported:
- network_image_encoding
- maintain_aspect_ratio
- center_crop
- tensor_name
- network_normalization_type
ROS Topics Subscribed
ROS Topic | Interface | Description |
---|---|---|
image |
sensor_msgs/Image | The image that should be encoded into a tensor. |
Limitation: All input images are required to have height and width that are both an even number of pixels.
ROS Topics Published
ROS Topic | Interface | Description |
---|---|---|
encoded_tensor |
isaac_ros_tensor_list_interfaces/TensorList | The resultant tensor after converting the image . |
isaac_ros_triton
Usage
ros2 launch isaac_ros_triton isaac_ros_triton.launch.py model_name:=<model_name> model_repository_paths:=<model_repository_paths> max_batch_size:=<max_batch_size> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names>
ROS Parameters
ROS Parameter | Type | Default | Description |
---|---|---|---|
model_repository_paths |
string list |
[''] |
The absolute paths to your model repositories in your local file system (the structure should follow Triton requirements) E.g. ['/tmp/models'] |
model_name |
string |
"" |
The name of your model. Under model_repository_paths , there should be a directory with this name, and it should align with the model name in the model configuration under this directory E.g. peoplesemsegnet_shuffleseg |
max_batch_size |
uint16_t |
8 |
The maximum batch size allowed for the model. It should align with the model configuration |
num_concurrent_requests |
uint16_t |
10 |
The number of requests the Triton server can take at a time. This should be set according to the tensor publisher frequency |
input_tensor_names |
string list |
['input_tensor'] |
A list of tensor names to be bound to specified input bindings names. Bindings occur in sequential order, so the first name here will be mapped to the first name in input_binding_names |
input_binding_names |
string list |
[''] |
A list of input tensor binding names specified by model E.g. ['input_2:0'] |
input_tensor_formats |
string list |
[''] |
A list of input tensor nitros formats. This should be given in sequential order E.g. ['nitros_tensor_list_nchw_rgb_f32'] |
output_tensor_names |
string list |
['output_tensor'] |
A list of tensor names to be bound to specified output binding names |
output_binding_names |
string list |
[''] |
A list of tensor names to be bound to specified output binding names E.g. ['argmax_1'] |
output_tensor_formats |
string list |
[''] |
A list of input tensor nitros formats. This should be given in sequential order E.g. [nitros_tensor_list_nchw_rgb_f32] |
ROS Topics Subscribed
ROS Topic | Interface | Description |
---|---|---|
tensor_pub |
isaac_ros_tensor_list_interfaces/TensorList | The input tensor stream |
ROS Topics Published
ROS Topic | Interface | Description |
---|---|---|
tensor_sub |
isaac_ros_tensor_list_interfaces/TensorList | The tensor list of output tensors from the model inference |
isaac_ros_tensor_rt
Usage
ros2 launch isaac_ros_tensor_rt isaac_ros_tensor_rt.launch.py model_file_path:=<model_file_path> engine_file_path:=<engine_file_path> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> verbose:=<verbose> force_engine_update:=<force_engine_update>
ROS Parameters
ROS Parameter | Type | Default | Description |
---|---|---|---|
model_file_path |
string |
model.onnx |
The absolute path to your model file in the local file system (the model file must be .onnx) E.g. model.onnx |
engine_file_path |
string |
/tmp/trt_engine.plan |
The absolute path to either where you want your TensorRT engine plan to be generated (from your model file) or where your pre-generated engine plan file is located E.g. model.plan |
force_engine_update |
bool |
true |
If set to true, the node will always try to generate a TensorRT engine plan from your model file and needs to be set to false to use the pre-generated TensorRT engine plan |
input_tensor_names |
string list |
['input_tensor'] |
A list of tensor names to be bound to specified input bindings names. Bindings occur in sequential order, so the first name here will be mapped to the first name in input_binding_names |
input_binding_names |
string list |
[''] |
A list of input tensor binding names specified by model E.g. ['input_2:0'] |
input_tensor_formats |
string list |
[''] |
A list of input tensor nitros formats. This should be given in sequential order E.g. ['nitros_tensor_list_nchw_rgb_f32'] |
output_tensor_names |
string list |
['output_tensor'] |
A list of tensor names to be bound to specified output binding names |
output_binding_names |
string list |
[''] |
A list of tensor names to be bound to specified output binding names E.g. ['argmax_1'] |
output_tensor_formats |
string list |
[''] |
A list of input tensor nitros formats. This should be given in sequential order E.g. [nitros_tensor_list_nchw_rgb_f32] |
verbose |
bool |
true |
If set to true, the node will enable verbose logging to console from the internal TensorRT execution |
max_workspace_size |
int64_t |
67108864l |
The size of the working space in bytes |
max_batch_size |
int32_t |
1 |
The maximum possible batch size in case the first dimension is dynamic and used as the batch size |
dla_core |
int64_t |
-1 |
The DLA Core to use. Fallback to GPU is always enabled. The default setting is GPU only |
enable_fp16 |
bool |
true |
Enables building a TensorRT engine plan file which uses FP16 precision for inference. If this setting is false, the plan file will use FP32 precision |
relaxed_dimension_check |
bool |
true |
Ignores dimensions of 1 for the input-tensor dimension check |
ROS Topics Subscribed
ROS Topic | Type | Description |
---|---|---|
tensor_pub |
isaac_ros_tensor_list_interfaces/TensorList | The input tensor stream |
ROS Topics Published
ROS Topic | Type | Description |
---|---|---|
tensor_sub |
isaac_ros_tensor_list_interfaces/TensorList | The tensor list of output tensors from the model inference |
Troubleshooting
Isaac ROS Troubleshooting
For solutions to problems with Isaac ROS, please check here.
Deep Learning Troubleshooting
For solutions to problems with using DNN models, please check here.
Updates
Date | Changes |
---|---|
2022-10-19 | Updated OSS licensing |
2022-08-31 | Update to be compatible with JetPack 5.0.2 |
2022-06-30 | Added format string parameter in Triton/TensorRT, switched to NITROS implementation, removed parameters in DNN Image Encoder |
2021-11-03 | Split DOPE and U-Net into separate repositories |
2021-10-20 | Initial release |