PointRCNN

This is not the official implementation of PointRCNN. We add an image segmentation network to improve recall of point cloud segmentation. The 2-stage network is frustum pointNet. Any pull request is appreciated.

Introduction

A 3D object detector that takes point cloud and RGB image(optional) as input.

Results

Architecture

1. Perform foreground point segmentation on the whole point cloud
2. Output a 3D proposal box for every foreground point
3. Crop point cloud with proposal boxes and feed into the 2nd-stage classification and box refinement network

Usage

Dependencies

• python2.7
• tensorflow(1.10.0)
• shapely
• mayavi
• opencv-python
• Compile tensorflow operators for pointnet following to https://github.com/charlesq34/frustum-pointnets

Data Preparation

For trainning and validation, download KITTI 3d object detection dataset, and put the folders as

dataset/KITTI/object/
  training/
    calib/
    image_2/
    label_2/
    velodyne/

  testing/
    calib/
    image_2/
    velodyne/

For testing, download KITTI video sequence and calibration files, and put the folders as

2011_10_03/
  calib_cam_to_cam.txt
  calib_imu_to_velo.txt
  calib_velo_to_cam.txt
  image_02/
  velodyne_points/

[Optional] Scene Augmentation

Our implementation also supports using augmented scene point cloud for training RPN, please refer to the official implementation of PointRCNN. After generating the data, just put the aug_scene/ folder under dataset/KITTI/object. If you don't want to use it, just set use_aug_scene=False when using rpn_dataset.

Image segmentaion annotation

Our image segmentation network is DeelabV3+ official implementation. The senmantic segmentation annotation is obtained by the following steps:

• Pointcloud completion using ip_basic
• Project 3D points to image plane to get segmentation annotation Codes for finetuning can be founded at deeplab_kitti_object. Or you can just use your own image segmentation network.

Train

There are 3 sub-model to be trained.

Region Proposal Network

sh train_rpn.sh

Frustum Pointnet

Before training the 2-Stage Network, we need to save the output of RPN and Image Segmentaion Network to the disk first.

Image Segmentation Network

For now deeplabv3+ is used and finetune on KITTI 3D object dataset

Evaluate

Region Proposal Network

sh test_rpn.sh

This will save the output of RPN and Image segmentation network to ./rcnn_data_train for training the RCNN network.

Frustum Pointnet

test_frustum.sh

Test

End to end

sh test.sh

Evaluation

Point cloud segmentation

Coverage means the percentage of object that have at least one point being detected.

Recall of RPN

Setting: IoU >= 0.5, 100 proposal

AP on Val Set

Pretrained Models

Reference

• Frustum PointNets for 3D Object Detection from RGB-D Data
• PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud