HOI-Learning-List

Some recent (2015-now) Human-Object Interaction Learing studies. If you find any errors or problems, please feel free to comment.

A list of Transfomer-based vision works: https://github.com/DirtyHarryLYL/Transformer-in-Vision.

Dataset

• PaStaNet-HOI (TPAMI2021) [Benchmark]

• HAKE (CVPR2020) [YouTube] [bilibili] [Website] [Paper] [HAKE-Action-Torch] [HAKE-Action-TF]

• Ambiguous-HOI (CVPR2020) [Website] [Paper]

• HICO-DET (WACV2018) [Website] [Paper]

• HCVRD (AAAI2018) [Website] [Paper]

• V-COCO (May 2015) [Website] [Paper]

• HICO (ICCV2015) [Website] [Paper]

• OpenImage [Website] [Paper]

• PIC [Website]

More...

Method

HOI Recognition: Image-based, to recognize all the HOIs in one image.

• PaStaNet: Toward Human Activity Knowledge Engine (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• Pairwise (ECCV2018) [Paper]

• Attentional Pooling for Action Recognition (NIPS2017) [Code] [Paper]

• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (ECCV2016) [Code] [Paper]

• Contextual Action Recognition with R*CNN (ICCV2015) [Code] [Paper]

• HOCNN (ICCV2015) [Code] [Paper]

• SGAP-Net (AAAI2020) [Paper]

More...

Unseen or zero-shot learning (image-level recognition).

• Compositional Learning for Human Object Interaction (ECCV2018) [Paper]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

More...

HOI Detection: Instance-based, to detect the human-object pairs and classify the interactions.

• FCL (CVPR2021) [Paper], [Code]

• AS-Net (CVPR2021) [Paper], [Code]

• End-to-End Human Object Interaction Detection with HOI Transformer (CVPR2021), [Paper], [Code]

• QPIC (CVPR2021) [Paper], [Code]

• TIN (TPAMI2021) [Paper], [Code]

• IDN (NeurIPS2020) [Paper] [Code]

• DIRV (AAAI2021) [Paper]

• DecAug (AAAI2021) [Paper]

• OSGNet (IEEE Access) [Paper]

• PFNet (CVM) [Paper]

• UniDet (ECCV2020) [Paper]

• DRG (ECCV2020) [Paper] [Code]

• FCMNet (ECCV2020) [Paper]

• Contextual Heterogeneous Graph Network for Human-Object Interaction Detection (ECCV2020) [Paper]

• PD-Net (ECCV2020) [Paper-1] [Paper-2] [Code]

• VCL (ECCV2020) [Paper] [Code]

• ACP (ECCV2020) [Paper] [Code]

• ConsNet (ACMMM2020) [Paper] [Code]

• Action-Guided Attention Mining and Relation Reasoning Network for Human-Object Interaction Detection (IJCAI2020) [Paper]

• PaStaNet (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• DJ-RN (CVPR2020) [Code] [Paper]

• Cascaded Human-Object Interaction Recognition (CVPR2020) [Code] [Paper]

• PPDM (CVPR2020) [Code] [Paper]

• IP-Net (CVPR2020) [Code] [Paper]

• VSGNet (CVPR2020) [Code] [Paper]

• HOID (CVPR2020) [Code] [Paper]

• Diagnosing Rarity in Human-Object Interaction Detection (CVPRW2020) [Paper]

• MLCNet (ICMR2020) [Paper]

• SIGN (ICME2020) [Paper]

• In-GraphNet (IJCAI-PRICAI 2020) [Paper]

• PMFNet(ICCV2019) [Code] [Paper]

• No-Frills (ICCV2019) [Code] [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• RPNN (ICCV2019) [Paper]

• Deep Contextual Attention for Human-Object Interaction Detection (ICCV2019) [Paper]

• Interactiveness (CVPR2019) [Code] [Paper]

• Turbo (AAAI2019) [Paper]

• GPNN (ECCV2018) [Code] [Paper]

• iCAN (BMVC2018) [Code] [Paper]

• InteractNet (CVPR2018) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

• HO-RCNN (WACV2018) [Code] [Paper]

• VS-GATs (Mar. 2020) [Paper]

• Classifying All Interacting Pairs in a Single Shot (Jan. 2020) [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• PMN (Jul. 2020) [Paper] [Code]

• SAG (Dec 2020) [Paper] [Code]

• SABRA (Dec 2020) [Paper]

More...

Unseen or zero-shot learning (instance-level detection).

• FCL (CVPR2021) [Paper], [Code]

• Detecting Human-Object Interaction with Mixed Supervision (WACV 2021) [Paper]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

• VCL (ECCV2020) [Paper] [Code]

• HOID (CVPR2020) [Code] [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• Functional (AAAI2020) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

More...

Video HOI methods

• LIGHTEN (ACMMM2020) [Paper] [Code]

• Generating Videos of Zero-Shot Compositions of Actions and Objects (Jul 2020), HOI GAN, [Paper]

• Grounded Human-Object Interaction Hotspots from Video (ICCV2019) [Code] [Paper]

• GPNN (ECCV2018) [Code] [Paper]

More...

Result

PaStaNet-HOI:

Proposed by TIN (TPAMI version, Transferable Interactiveness Network). It is built on HAKE data, includes 110K+ images and 520 HOIs (without the 80 "no_interaction" HOIs of HICO-DET to avoid the incomplete labeling). It has a more severe long-tailed data distribution thus is more difficult.

Detector: COCO pre-trained

HICO-DET:

1) Detector: COCO pre-trained

2) Detector: pre-trained on COCO, fine-tuned on HICO-DET train set (with GT human-object pair boxes) or one-stage detector

Finetuned detector would learn to only detect the interactive humans and objects (with interactiveness), thus suppress many wrong pairings (non-interactive human-object pairs) and boost the performance. |Method| Pub|Full(def) | Rare(def) | None-Rare(def)| Full(ko) | Rare(ko) | None-Rare(ko) | |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |UniDet|ECCV2020|17.58 |11.72 |19.33 |19.76 |14.68 |21.27| |IP-Net | CVPR2020| 19.56 |12.79| 21.58 |22.05 |15.77 |23.92| |PPDM (paper) |CVPR2020|21.10 |14.46| 23.09| -|-|-| |PPDM (github-hourglass104) |CVPR2020|21.73/21.94 |13.78/13.97 |24.10/24.32 |24.58/24.81| 16.65/17.09| 26.84/27.12| |Functional |AAAI2020|21.96 |16.43|23.62| -|-|-| |SABRA-Res50| arXiv| 23.48| 16.39| 25.59| 28.79| 22.75| 30.54| |VCL|ECCV2020|23.63 |17.21 |25.55 |25.98 |19.12 |28.03| |SABRA-Res50FPN| arXiv| 24.12 |15.91| 26.57| 29.65| 22.92| 31.65| |ConsNet|ACMMM2020|24.39 |17.10 |26.56|-|-|-| |DRG|ECCV2020|24.53 |19.47 |26.04 |27.98 |23.11 |29.43| |SABRA-Res152| arXiv| 26.09 |16.29| 29.02| 31.08| 23.44| 33.37| |IDN|NeurIPS2020|26.29|22.61|27.39|28.24|24.47|29.37| |Zou et al.|CVPR2021|26.61 |19.15| 28.84| 29.13| 20.98| 31.57| |AS-Net|CVPR2021|28.87 |24.25 |30.25 |31.74 |27.07 |33.14| |QPIC-Res50|CVPR2021| 29.07 |21.85 |31.23 |31.68 |24.14 |33.93| |FCL|CVPR2021|29.12 |23.67 |30.75 |31.31 |25.62 |33.02| |QPIC-Res101|CVPR2021|29.90 |23.92 |31.69 |32.38 |26.06 |34.27|

3) Ground Truth human-object pair boxes (only evaluating HOI recognition)

4) Enhanced with HAKE:

Ambiguous-HOI

Detector: COCO pre-trained

V-COCO: Scenario1

1) Detector: COCO pre-trained or one-stage detector

2) Enhanced with HAKE:

HICO

1) Default

2) Enhanced with HAKE: