overlappredator / OverlapPredator

PREDATOR: Registration of 3D Point Clouds with Low Overlap (CVPR 2021, Oral)

This repository represents the official implementation of the paper:

PREDATOR: Registration of 3D Point Clouds with Low Overlap

*Shengyu Huang, *Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, Konrad Schindler
|ETH Zurich | * Equal contribution

For implementation using MinkowskiEngine backbone, please check this

For more information, please see the project website

Contact

If you have any questions, please let us know:

• Shengyu Huang {[email protected]}
• Zan Gojcic {[email protected]}

News

• 2021-08-09: We've updated arxiv version of our paper with improved performance!
• 2021-06-02: Fix feature gathering bug in k-nn graph, please see improved performance in this issue. Stay tunned for updates on other experiments!
• 2021-05-31: Check our video and poster on project page!
• 2021-03-25: Camera ready is on arXiv! I also gave a talk on Predator(中文), you can find the recording here: Bilibili, Youtube
• 2021-02-28: MinkowskiEngine-based PREDATOR release
• 2020-11-30: Code and paper release

Instructions

This code has been tested on

• Python 3.8.5, PyTorch 1.7.1, CUDA 11.2, gcc 9.3.0, GeForce RTX 3090/GeForce GTX 1080Ti

Note: We observe random data loader crashes due to memory issues, if you observe similar issues, please consider reducing the number of workers or increasing CPU RAM. We now released a sparse convolution-based Predator, have a look here!

Requirements

To create a virtual environment and install the required dependences please run:

git clone https://github.com/overlappredator/OverlapPredator.git
virtualenv predator; source predator/bin/activate
cd OverlapPredator; pip install -r requirements.txt
cd cpp_wrappers; sh compile_wrappers.sh; cd ..

in your working folder.

Datasets and pretrained models

For KITTI dataset, please follow the instruction on KITTI Odometry website to download the KITTI odometry training set.

We provide

• preprocessed 3DMatch pairwise datasets (voxel-grid subsampled fragments together with their ground truth transformation matrices)
• raw dense 3DMatch datasets
• modelnet dataset
• pretrained models on 3DMatch, KITTI and Modelnet

The preprocessed data and models can be downloaded by running:

sh scripts/download_data_weight.sh

To download raw dense 3DMatch data, please run:

wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/pairwise_reg/3dmatch.zip
unzip 3dmatch.zip

The folder is organised as follows:

• 3dmatch
  • train
    • 7-scenes-chess
      • fragments
        • cloud_bin_*.ply
        • ...
      • poses
        • cloud_bin_*.txt
        • ...
    • ...
  • test

3DMatch(Indoor)

Train

After creating the virtual environment and downloading the datasets, Predator can be trained using:

python main.py configs/train/indoor.yaml

Evaluate

For 3DMatch, to reproduce Table 2 in our main paper, we first extract features and overlap/matachability scores by running:

python main.py configs/test/indoor.yaml

the features together with scores will be saved to snapshot/indoor/3DMatch. The estimation of the transformation parameters using RANSAC can then be carried out using:

for N_POINTS in 250 500 1000 2500 5000
do
  python scripts/evaluate_predator.py --source_path snapshot/indoor/3DMatch --n_points $N_POINTS --benchmark 3DMatch --exp_dir snapshot/indoor/est_traj --sampling prob
done

dependent on n_points used by RANSAC, this might take a few minutes. The final results are stored in snapshot/indoor/est_traj/{benchmark}_{n_points}_prob/result. To evaluate PREDATOR on 3DLoMatch benchmark, please also change 3DMatch to 3DLoMatch in configs/test/indoor.yaml.

Demo

We prepared a small demo, which demonstrates the whole Predator pipeline using two random fragments from the 3DMatch dataset. To carry out the demo, please run:

python scripts/demo.py configs/test/indoor.yaml

The demo script will visualize input point clouds, inferred overlap regions, and point cloud aligned with the estimated transformation parameters:

ModelNet(Synthetic)

Train

To train PREDATOR on ModelNet, please run:

python main.py configs/train/modelnet.yaml

We provide a small script to evaluate Predator on ModelNet test set, please run:

python main.py configs/test/modelnet.yaml

The rotation and translation errors could be better/worse than the reported ones due to randomness in RANSAC.

KITTI(Outdoor)

We provide a small script to evaluate Predator on KITTI test set, after configuring KITTI dataset, please run:

python main.py configs/test/kitti.yaml

the results will be saved to the log file.

Custom dataset

We have a few tips for train/test on custom dataset

• If it's similar indoor scenes, please run demo.py first to check the generalisation ability before retraining
• Remember to voxel-downsample the data in your data loader, see kitti.py for reference

Citation

If you find this code useful for your work or use it in your project, please consider citing:

@InProceedings{Huang_2021_CVPR,
    author    = {Huang, Shengyu and Gojcic, Zan and Usvyatsov, Mikhail and Wieser, Andreas and Schindler, Konrad},
    title     = {Predator: Registration of 3D Point Clouds With Low Overlap},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {4267-4276}
}

Acknowledgments

In this project we use (parts of) the official implementations of the followin works:

• FCGF (KITTI preprocessing)
• D3Feat (KPConv backbone)
• 3DSmoothNet (3DMatch preparation)
• MultiviewReg (3DMatch benchmark)
• SuperGlue (Transformer part)
• DGCNN (self-gnn)
• RPMNet (ModelNet preprocessing and evaluation)

We thank the respective authors for open sourcing their methods. We would also like to thank reviewers, especially reviewer 2 for his/her valuable inputs.