GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → gist-ailab → uoais

gist-ailab / uoais

Licence: other
Codes of paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling", ICRA 2022

Programming Languages

python
139335 projects - #7 most used programming language
Cuda
1817 projects
C++
36643 projects - #6 most used programming language
CMake
9771 projects

Labels

deep-learningpytorchsegmentationinstance-segmentationsynthetic-datarobot-visionunseen-object

Projects that are alternatives of or similar to uoais

Multi Human Parsing
🔥🔥Official Repository for Multi-Human-Parsing (MHP)🔥🔥
Stars: ✭ 507 (+558.44%)
Mutual labels:  segmentation, instance-segmentation
Seg By Interaction
Unsupervised instance segmentation via active robot interaction
Stars: ✭ 78 (+1.3%)
Mutual labels:  segmentation, instance-segmentation
Entity
EntitySeg Toolbox: Towards Open-World and High-Quality Image Segmentation
Stars: ✭ 313 (+306.49%)
Mutual labels:  segmentation, instance-segmentation
Tfvos
Semi-Supervised Video Object Segmentation (VOS) with Tensorflow. Includes implementation of *MaskRNN: Instance Level Video Object Segmentation (NIPS 2017)* as part of the NIPS Paper Implementation Challenge.
Stars: ✭ 151 (+96.1%)
Mutual labels:  segmentation, instance-segmentation
Tfwss
Weakly Supervised Segmentation with Tensorflow. Implements instance segmentation as described in Simple Does It: Weakly Supervised Instance and Semantic Segmentation, by Khoreva et al. (CVPR 2017).
Stars: ✭ 212 (+175.32%)
Mutual labels:  segmentation, instance-segmentation
BCNet
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]
Stars: ✭ 434 (+463.64%)
Mutual labels:  segmentation, instance-segmentation
Data Science Bowl 2018
End-to-end one-class instance segmentation based on U-Net architecture for Data Science Bowl 2018 in Kaggle
Stars: ✭ 56 (-27.27%)
Mutual labels:  segmentation, instance-segmentation
Keraspersonlab
Keras-tensorflow implementation of PersonLab (https://arxiv.org/abs/1803.08225)
Stars: ✭ 163 (+111.69%)
Mutual labels:  segmentation, instance-segmentation
Deepmask Pytorch
PyTorch re-implementation of DeepMask
Stars: ✭ 201 (+161.04%)
Mutual labels:  segmentation, instance-segmentation
Ffn
Flood-Filling Networks for instance segmentation in 3d volumes.
Stars: ✭ 252 (+227.27%)
Mutual labels:  segmentation, instance-segmentation
CAP augmentation
Cut and paste augmentation for object detection and instance segmentation
Stars: ✭ 93 (+20.78%)
Mutual labels:  segmentation, instance-segmentation
VNet
Prostate MR Image Segmentation 2012
Stars: ✭ 54 (-29.87%)
Mutual labels:  segmentation
DeepSegmentor
Sequence Segmentation using Joint RNN and Structured Prediction Models (ICASSP 2017)
Stars: ✭ 17 (-77.92%)
Mutual labels:  segmentation
Robotics-Object-Pose-Estimation
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.
Stars: ✭ 153 (+98.7%)
Mutual labels:  synthetic-data
rt-mrcnn
Real time instance segmentation with Mask R-CNN, live from webcam feed.
Stars: ✭ 47 (-38.96%)
Mutual labels:  instance-segmentation
ML IDCard Segmentation-TF-Keras
Machine Learning Project to identify an ID Card on an image
Stars: ✭ 38 (-50.65%)
Mutual labels:  segmentation
Three-Filters-to-Normal
Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator (RAL+ICRA'21)
Stars: ✭ 41 (-46.75%)
Mutual labels:  synthetic-data
prostate segmentation
NCI-ISBI 2013 Challenge - Automated Segmentation of Prostate Structures
Stars: ✭ 28 (-63.64%)
Mutual labels:  instance-segmentation
nicMSlesions
Easy multiple sclerosis white matter lesion segmentation using convolutional deep neural networks.
Stars: ✭ 33 (-57.14%)
Mutual labels:  segmentation
Basic-Image-Processing
Implementation of Basic Digital Image Processing Tasks in Python / OpenCV
Stars: ✭ 102 (+32.47%)
Mutual labels:  segmentation
View All Similar Projects ➔

Unseen Object Amodal Instance Segmentation (UOAIS)

Seunghyeok Back, Joosoon Lee, Taewon Kim, Sangjun Noh, Raeyoung Kang, Seongho Bak, Kyoobin Lee

This repository contains source codes for the paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling." (Accepted at ICRA 2022)

[Paper] [Project Website] [Video]

Updates & TODO Lists

  • (2021.09.26) UOAIS-Net has been released
  • (2021.11.15) Inference codes for kinect azure and OSD dataset.
  • (2021.11.22) ROS nodes for kinect azure and realsense D435
  • (2021.12.22) Train and evaluation codes on OSD and OCID dataset + OSD-Amodal annotation

Getting Started

Environment Setup

Tested on Titan RTX with python 3.7, pytorch 1.8.0, torchvision 0.9.0, CUDA 10.2 / 11.1 and detectron2 v0.5 / v0.6

  1. Download source codes and checkpoints
git clone https://github.com/gist-ailab/uoais.git
cd uoais
mkdir output

  1. Download checkpoints at GDrive

  2. Move the R50_depth_mlc_occatmask_hom_concat and R50_rgbdconcat_mlc_occatmask_hom_concat to the output folder.

  3. Move the rgbd_fg.pth to the foreground_segmentation folder

  4. Set up a python environment

conda create -n uoais python=3.8
conda activate uoais
pip install torch torchvision 
pip install shapely torchfile opencv-python pyfastnoisesimd rapidfuzz termcolor

  1. Install detectron2

  2. Build custom AdelaiDet

python setup.py build develop

Run on sample OSD dataset

# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/run_on_OSD.py --use-cgnet --dataset-path ./sample_data --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/run_on_OSD.py --use-cgnet --dataset-path ./sample_data  --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/run_on_OSD.py --dataset-path ./sample_data --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/run_on_OSD.py --dataset-path ./sample_data --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Run with ROS

  1. Realsense D-435 (realsense-ros is required.)
# launch realsense2 driver
roslaunch realsense2_camera rs_aligned_depth.launch
# launch uoais node
roslaunch uoais uoais_rs_d435.launch 
# or you can use rosrun
rosrun uoais uoais_node.py _mode:="topic"
  1. Azure Kinect (Azure_kinect_ROS_Driver is required)
# launch azure kinect driver
roslaunch azure_kinect_ros_driver driver.launch
# launch uoais node
roslaunch uoais uoais_k4a.launch

Topics & service

  • /uoais/vis_img (sensor_msgs/Image): visualization results
  • /uoais/results (uoais/UOAISResults): UOAIS-Net predictions (mode:=topic)
  • /get_uoais_results (uoais/UOAISRequest): UOAIS-Net predictions (mode:=service)

Parameters

  • mode (string): running mode of ros node (topic or service)
  • rgb (string): topic name of the input rgb
  • depth (string): topic name of the input depth
  • camera_info (string): topic name of the input camera info
  • use_cgnet (bool): use CG-Net [1] for foreground segmentation or not
  • use_planeseg (bool): use RANSAC for plane segmentation or not
  • ransac_threshold (float): max distance a point can be from the plane model

Run without ROS

  1. Realsense D-435 (librealsense and pyrealsense2 are required.)
# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/rs_demo.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/rs_demo.py --use-cgnet --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/rs_demo.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/rs_demo.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
  1. Azure Kinect (Azure-Kinect-Sensor-SDK and pyk4a are required.)
# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/k4a_demo.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/k4a_demo.py --use-cgnet --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/k4a_demo.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/k4a_demo.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Train & Evaluation

Dataset Preparation

  1. Download UOAIS-Sim.zip and OSD-Amodal-annotations.zip at GDrive
  2. Download OSD-0.2-depth.zip at OSD. [2]
  3. Download OCID dataset at OCID. [3]
  4. Extract the downloaded datasets and organize the folders as follows
uoais
├── output
└── datasets
       ├── OCID-dataset # for evaluation on indoor scenes
       │     └──ARID10
       │     └──ARID20
       │     └──YCB10
       ├── OSD-0.20-depth # for evaluation on tabletop scenes
       │     └──amodal_annotation # OSD-amodal
       │     └──annotation
       │     └──disparity
       │     └──image_color
       │     └──occlusion_annotation # OSD-amodal
       └── UOAIS-Sim # for training
              └──annotations
              └──train
              └──val

Train on UOAIS-Sim

# UOAIS-Net (RGB-D) 
python train_net.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth) 
python train_net.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Evaluation on OSD dataset

# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python eval/eval_on_OSD.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml --use-cgnet
# UOAIS-Net (depth) + CG-Net (foreground segmentation)
python eval/eval_on_OSD.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml --use-cgnet

This code evaluates the UOAIS-Net that was trained on a single seed (7), thus the metrics from this code and the paper (an average of seeds 7, 77, 777) can be different.

Evaluation on OCID dataset

# UOAIS-Net (RGB-D)
python eval/eval_on_OCID.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python eval/eval_on_OCID.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Visualization on OSD dataset

python tools/run_on_OSD.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml

License

The source code of this repository is released only for academic use. See the license file for details.

Notes

The codes of this repository are built upon the following open sources. Thanks to the authors for sharing the code!

Citation

If you use our work in a research project, please cite our work:

@misc{back2021unseen,
      title={Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling}, 
      author={Seunghyeok Back and Joosoon Lee and Taewon Kim and Sangjun Noh and Raeyoung Kang and Seongho Bak and Kyoobin Lee},
      year={2021},
      eprint={2109.11103},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

References

[1] SUN, Yao, et al. Cg-net: Conditional gis-aware network for individual building segmentation in vhr sar images. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-15.
[2] Richtsfeld, Andreas, et al. "Segmentation of unknown objects in indoor environments." 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.
[3] Suchi, Markus, et al. "EasyLabel: a semi-automatic pixel-wise object annotation tool for creating robotic RGB-D datasets." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
[4] Xiang, Yu, et al. "Learning rgb-d feature embeddings for unseen object instance segmentation." Conference on Robot Learning (CoRL). 2020.
[5] Xiao, Yuting, et al. "Amodal Segmentation Based on Visible Region Segmentation and Shape Prior." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.
[6] DENNINGER, Maximilian, et al. Blenderproc. arXiv preprint arXiv:1911.01911, 2019.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].