All Projects → CurryYuan → InstanceRefer

CurryYuan / InstanceRefer

Licence: other
[ICCV 2021] InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to InstanceRefer

cx leaderboard
Elixir library for fast, customizable leaderboards
Stars: ✭ 18 (-71.87%)
Mutual labels:  leaderboard
Highway
✈️~🎢A Java game to drive your truck against other 2 trucks throughout the highway and compete to stay on the top of the leaderboard
Stars: ✭ 13 (-79.69%)
Mutual labels:  leaderboard
SnowflakeNet
(TPAMI 2022) Snowflake Point Deconvolution for Point Cloud Completion and Generation with Skip-Transformer
Stars: ✭ 74 (+15.63%)
Mutual labels:  iccv2021
MSMARCO-MRC-Analysis
Analysis on the MS-MARCO leaderboard regarding the machine reading comprehension task.
Stars: ✭ 20 (-68.75%)
Mutual labels:  leaderboard
games services
A Flutter plugin to support game center and google play games services.
Stars: ✭ 67 (+4.69%)
Mutual labels:  leaderboard
Quizzie
Open Sourced Quiz Portal which can be used for any event / competition with a custom leaderboard.
Stars: ✭ 31 (-51.56%)
Mutual labels:  leaderboard
PlaneTR3D
[ICCV'21] PlaneTR: Structure-Guided Transformers for 3D Plane Recovery
Stars: ✭ 58 (-9.37%)
Mutual labels:  iccv2021
MSRGCN
Official implementation of MSR-GCN (ICCV2021 paper)
Stars: ✭ 42 (-34.37%)
Mutual labels:  iccv2021
gci-leaders
A website showing Google Code-in information 🏆
Stars: ✭ 40 (-37.5%)
Mutual labels:  leaderboard
Opensource-Contribution-Leaderboard
Open Source project contributors tracking leaderboard built with ❤️ in NodeJS 😉
Stars: ✭ 30 (-53.12%)
Mutual labels:  leaderboard
discord-paginationembed
A pagination utility for MessageEmbed in Discord.JS
Stars: ✭ 93 (+45.31%)
Mutual labels:  leaderboard
laravel-gamify
Laravel Gamify: Gamification System with Points & Badges support
Stars: ✭ 35 (-45.31%)
Mutual labels:  leaderboard
atari-leaderboard
A leaderboard of human and machine performance on the Arcade Learning Environment (ALE).
Stars: ✭ 22 (-65.62%)
Mutual labels:  leaderboard
Scavenger
A virtual "scavenger hunt" game for mobile devices using Unity, Azure, and PlayFab
Stars: ✭ 19 (-70.31%)
Mutual labels:  leaderboard
rclc
Rich Context leaderboard competition, including the corpus and current SOTA for required tasks.
Stars: ✭ 20 (-68.75%)
Mutual labels:  leaderboard
Awesome-ICCV2021-Low-Level-Vision
A Collection of Papers and Codes for ICCV2021 Low Level Vision and Image Generation
Stars: ✭ 163 (+154.69%)
Mutual labels:  iccv2021
RPC-Leaderboard
RPC Dataset Leaderboard
Stars: ✭ 63 (-1.56%)
Mutual labels:  leaderboard
Deep-Matching-Prior
Official implementation of deep matching prior
Stars: ✭ 21 (-67.19%)
Mutual labels:  iccv2021
LLVIP
LLVIP: A Visible-infrared Paired Dataset for Low-light Vision
Stars: ✭ 438 (+584.38%)
Mutual labels:  iccv2021
G-SFDA
code for our ICCV 2021 paper 'Generalized Source-free Domain Adaptation'
Stars: ✭ 88 (+37.5%)
Mutual labels:  iccv2021

InstanceRefer

InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring

This repository is for the ICCV 2021 paper and 1st method on ScanRefer benchmark [paper].

Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li*, Shuguang Cui

If you find our work useful in your research, please consider citing:

@InProceedings{yuan2021instancerefer,
  title={Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring},
  author={Yuan, Zhihao and Yan, Xu and Liao, Yinghong and Zhang, Ruimao and Li, Zhen and Cui, Shuguang},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={1791-1800},
  year={2021}
}

News

  • 2021-07-23 InstanceRefer is accepted at ICCV 2021 🔥!
  • 2021-04-22 We release evaluation codes and pre-trained model!
  • 2021-03-31 We release InstanceRefer v1 🚀!
  • 2021-03-18 We achieve 1st place in ScanRefer leaderboard 🔥.

Getting Started

Setup

The code is tested on Ubuntu 16.04 LTS & 18.04 LTS with PyTorch 1.6 CUDA 10.1 and PyTorch 1.8 CUDA 10.2 installed.

conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch

Install the necessary packages listed out in requirements.txt:

pip install -r requirements.txt

After all packages are properly installed, please run the following commands to compile the torchsaprse 1.2:

pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git
sudo apt-get install libsparsehash-dev

Before moving on to the next step, please don't forget to set the project root path to the CONF.PATH.BASE in lib/config.py.

Data preparation

  1. Download the ScanRefer dataset and unzip it under data/.
  2. Downloadand the preprocessed GLoVE embeddings (~990MB) and put them under data/.
  3. Download the ScanNetV2 dataset and put (or link) scans/ under (or to) data/scannet/scans/ (Please follow the ScanNet Instructions for downloading the ScanNet dataset). After this step, there should be folders containing the ScanNet scene data under the data/scannet/scans/ with names like scene0000_00
  4. Used official and pre-trained PointGroup generate panoptic segmentation in PointGroupInst/. We provide pre-processed data in Baidu Netdisk [password: 0nxc].
  5. Pre-processed instance labels, and new data should be generated in data/scannet/pointgroup_data/
cd data/scannet/
python prepare_data.py --split train --pointgroupinst_path [YOUR_PATH]
python prepare_data.py --split val   --pointgroupinst_path [YOUR_PATH]
python prepare_data.py --split test  --pointgroupinst_path [YOUR_PATH]

Finally, the dataset folder should be organized as follows.

InstanceRefer
├── data
│   ├── glove.p
│   ├── ScanRefer_filtered.json
│   ├── ...
│   ├── scannet
│   │  ├── meta_data
│   │  ├── pointgroup_data
│   │  │  ├── scene0000_00_aligned_bbox.npy
│   │  │  ├── scene0000_00_aligned_vert.npy
│   │  ├──├──  ... ...

Training

Train the InstanceRefer model. You can change hyper-parameters in config/InstanceRefer.yaml:

python scripts/train.py --log_dir instancerefer

Evaluation

You need specific the use_checkpoint with the folder that contains model.pth in config/InstanceRefer.yaml and run with:

python scripts/eval.py

Pre-trained Models

Input [email protected] [email protected] Checkpoints
xyz+rgb 37.6 30.7 Baidu Netdisk [password: lrpb]

TODO

  • Updating to the best version.
  • Release codes for prediction on benchmark.
  • Release pre-trained model.
  • Merge PointGroup in an end-to-end manner.

Acknowledgement

This project is not possible without multiple great opensourced codebases.

License

This repository is released under MIT License (see LICENSE file for details).

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].