Silhouette-Guided-3D

PyTorch implementation of our WACV 2020 paper: "Silhouette Guided Point Cloud Reconstruction beyond Occlusion"

Our short introduction video

Network architecture:

Requirements

• Python 3
• PyTorch >= 0.4.0
• numpy, scipy, pickle, skimage, sklearn, random, re
• torchvision
• Matlab (for FSSR based post process)

Installation

• Install mve by following the instructions. This is for FSSR based point cloud refinement.
• Under ./matlab folder, install gptoolbox for Matlab based Poisson-Disc Sampling
• [Optional] Install Pix3D evaluation toolkit under the current folder. Note that this requires Tensorflow.
• [Optional] Install PCN evaluation toolkit under the current folder. Note that this requires Tensorflow. PCN toolkit is for object-centered evaluation.
• [Optional] Install Mask-RCNN benchmark under the current folder. This is for getting visible silhouette for completion in Pix3D (We've included pre-processed results as below).

Download Data and Pre-trained Model

• Download pre-trained models and put them under the ./model/ folder.
• Download pre-processed DYCE dataset and put them under the ./data/ folder.
• Download pre-processed Pix3D dataset and put them under the ./data/ folder. This includes pre-computed complete silhouette and ground truth point clouds rotated w.r.t. camera position. We've excluded examples with incorrect Mask-RCNN detections.
• Download ShapeNet dataset and put them under the ./data/ folder.
• Download pre-processed LSUN dataset and put them under the ./data/ folder
• Download pre-computed result. and put them under the ./result/ folder. This includes point clouds prediction on ShapeNet and Pix3D after FSSR refinement.
• [Optional] Download ShapeNet rendered images, and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.
• [Optional] Download ShapeNet object-centered point cloud ground truth, , and put them under ./data/ShapeNet/ folder. This is for comparing to object-centered point cloud reconstruction approach.

Training

• Point cloud reconstruction

  python train.py
  python test.py
  

  This will save network predictions for the downstream FSSR post-refinement step.

• Silhouette completion First train on DYCE dataset:

  python train_sc.py
  

  Then finetune on Pix3D dataset, using 5-fold cross validation ( you will need to run it 5 times by changing the fold number in L32-35 ):

  python train_sc_ft.py
  python test_sc_pix3d.py
  

• Silhouette guidede point cloud reconstruction

  python train_occ.py
  python test_rec_pix3d.py
  

  Then perform FSSR post-refinement step as describe below

Surface Based Point Clouds Refinement

• Start matlab

  cd matlab
  ./matlab
  

• pre-compute FSSR params (per-pixel normal and scale), change folder name based on your saved network predictions path

  FssrPostRefine
  

• FSSR Here we provide the sample batch-process code (need to go back to the main folder):

  cd ..
  python fssr_batch_process.py
  

• smoothing

  cd matlab
  preComputeFssrParam
  cd ..
  

  This produces the refined point clouds for evaluation.

Evaluation

• Pix3D
  • ICP-based fitting since Pix3D ground truth is object-centered (you can skip this step since we've included pre-computered ground truth and predictions). Code is derived from 3D-LMNET, which also includes the ground truth point cloud.

    cd pcn
    python metrics_pix3d.py
    cd ..
    

  • You need to use TensorFlow 3.0+ to run the evaluation:

    cd pix3d/eval/
    python eval_pix3d.py
    cd ../../
    

• ShapeNet
  • You need to use TensorFlow 3.0+ to run the evaluation:

    cd pcn
    python eval_shapenet.py
    cd ..
    

  • To compare with object-centered point cloud prediction appraoch (3D-LMNET), you need to perform ICP-based fitting first:

    cd pcn
    python metrics_shapenet.py
    cd ../pix3d/eval/
    python eval_shapenet_object_centered.py
    cd ../../
    

• DYCE (silhouette completion)
  • This is PyTorch based

    python test_sc_DYCE.py
    

Citation

Please cite our paper for any purpose of usage.

@inproceedings{zou2020silhouette,
  title={Silhouette Guided Point Cloud Reconstruction beyond Occlusion},
  author={Zou, Chuhang and Hoiem, Derek},
  booktitle={The IEEE Winter Conference on Applications of Computer Vision},
  pages={41--50},
  year={2020}
}