wkentaro / Morefusion
Licence: other
MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion
Stars: ✭ 138
Programming Languages
python
139335 projects - #7 most used programming language
MoreFusion
Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion
Kentaro Wada,
Edgar Sucar,
Stephen James,
Daniel Lenton,
Andrew J. Davison
Dyson Robotics Laboratory , Imperial College London
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
Dyson Robotics Laboratory , Imperial College London
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
MoreFusion is an object-level reconstruction system that builds a map with known-shaped objects, exploiting volumetric reconstruction of detected objects in a real-time, incremental scene reconstruction senario. The key components are:
- Occupancy-based volumetric reconstruction of detected objects for model alignment in the later stage;
- Volumetric pose prediction that exploits volumetric reconstruction and CNN feature extraction from the image observation;
- Joint pose refinement of objects based on geometric consistency among objects and impenetrable space.
Installation
There're several options for installation:
- Python project only: Python framework for pose estimation (e.g., training, inference, refinement).
- ROS project for camera demonstration: ROS framework for object-level mapping with live cameras.
- ROS project for robotic demonstration: ROS framework for robotic demonstration with object-level mapping.
NOTE: We have developed this project on Ubuntu 16.04 (and ROS Kinetic, CUDA 10.1), so several code changes may be needed to adapt to other OS (and ROS, CUDA versions).
Python project only
make install
ROS project for camera demonstration
mkdir -p ~/ros_morefusion/src
cd ~/ros_morefusion/src
git clone https://github.com/wkentaro/morefusion.git
cd morefusion
make install
cd ~/ros_morefusion
ln -s src/ros/*.sh .
./rosdep_install.sh
./catkin_build.robot_agent.sh
source .autoenv.zsh
ROS project for robotic demonstration
-
robot-agent
: A computer for visual processing. -
robot-node
: A computer with real-time OS for Panda robot.
@robot-agent
Same as above instruction: ROS project for camera demonstration.
@robot-node
mkdir -p ~/ros_morefusion/src
cd ~/ros_morefusion/src
git clone https://github.com/wkentaro/morefusion.git
cd ~/ros_morefusion
ln -s src/ros/*.sh .
./catkin_build.robot_node.sh
source devel/setup.bash
rosrun morefusion_panda create_udev_rules.sh
Usage
Training & Inference
Pre-trained models are provided in the demos as following, so this process is optional to run the demos.
Instance Segmentation
cd examples/ycb_video/instance_segm
./download_dataset.py
mpirun -n 4 python train_multi.py # 4-gpu training
./image_demo.py --model logs/XXX/XXX.npz
6D pose prediction
# baseline model (point-cloud-based)
cd examples/ycb_video/singleview_pcd
./download_dataset.py
./train.py --gpu 0 --centerize-pcd --pretrained-resnet18 # 1-gpu
mpirun -n 4 ./train.py --multi-node --centerize-pcd --pretrained-resnet18 # 4-gpu
# volumetric prediction model (3D-CNN-based)
cd examples/ycb_video/singleview_3d
./download_dataset.py
./train.py --gpu 0 --pretrained-resnet18 --with-occupancy # 1-gpu
mpirun -n 4 ./train.py --multi-node --pretrained-resnet18 --with-occupancy # 4-gpu
mpirun -n 4 ./train.py --multi-node --pretrained-resnet18 # w/o occupancy
# inference
./download_pretrained_model.py # for downloading pretrained model
./demo.py logs/XXX/XXX.npz
./evaluate.py logs/XXX
Joint pose refinement
cd examples/ycb_video/pose_refinement
./check_icp_vs_icc.py # press [s] to start
Camera demonstration
Static Scene
# using orb-slam2 for camera tracking
roslaunch morefusion_panda_ycb_video rs_rgbd.launch
roslaunch morefusion_panda_ycb_video rviz_static.desk.launch
roslaunch morefusion_panda_ycb_video setup_static.desk.launch
# using robotic kinematics for camera tracking
roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
roslaunch morefusion_panda_ycb_video rviz_static.robot.launch
roslaunch morefusion_panda_ycb_video setup_static.robot.launch
Dynamic Scene
roslaunch morefusion_panda_ycb_video rs_rgbd.launch
roslaunch morefusion_panda_ycb_video rviz_dynamic.desk.launch
roslaunch morefusion_panda_ycb_video setup_dynamic.desk.launch
roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
roslaunch morefusion_panda_ycb_video rviz_dynamic.robot.launch
roslaunch morefusion_panda_ycb_video setup_dynamic.robot.launch
Robotic Demonstration
Robotic Pick-and-Place
robot-agent $ sudo ntpdate 0.uk.pool.ntp.org # for time synchronization
robot-node $ sudo ntpdate 0.uk.pool.ntp.org # for time synchronization
robot-node $ roscore
robot-agent $ roslaunch morefusion_panda panda.launch
robot-node $ roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
robot-node $ roslaunch morefusion_panda_ycb_video rviz_static.launch
robot-node $ roslaunch morefusion_panda_ycb_video setup_static.robot.launch TARGET:=2
robot-node $ rosrun morefusion_panda_ycb_video robot_demo_node.py
>>> ri.run()
Figure 4. Targetted Object Pick-and-Place. (a) Scanning the Scene; (b) Removing Distractor Objects; (c) Picking Target Object.
Citation
If you find MoreFusion useful, please consider citing the paper as:
@inproceedings{Wada:etal:CVPR2020,
title={{MoreFusion}: Multi-object Reasoning for {6D} Pose Estimation from Volumetric Fusion},
author={Kentaro Wada and Edgar Sucar and Stephen James and Daniel Lenton and Andrew J. Davison},
booktitle={Proceedings of the {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2020},
}
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].