Fast and accurate Human Pose Estimation using ShelfNet with PyTorch

This repository is the result of my curiosity to find out whether ShelfNet is an efficient CNN architecture for computer vision tasks other than semantic segmentation, and more specifically for the human pose estimation task. The answer is a clear yes, with 74.6 mAP and 127 FPS on the MS COCO Keypoints data set which represents a 3.5x boost in FPS compared to HRNet for a similar accuracy.

This repository includes:

• Source code of ShelfNet modified from the authors' repository

• Code to prepare the MS COCO keypoints dataset

• Training and evaluation code for MS COCO keypoints modified from the HRNet authors' repository

• Pre-trained weights for ShelfNet50

If you use it in your projects, please consider citing this repository (bibtex below).

ShelfNet Architecture Overview

The ShelfNet architecture was introduced by J. Zhuang, J. Yang, L. Gu and N. Dvornek through a paper available on arXiv. The paper evaluates the network only on the semantic segmentation task. The authors' contribution is to have created a fast architecture with a performance similar to the state of the art (PSPNet & EncNet at the time of publishing this repository) on PASCAL VOC and better performance on Cityscapes. Therefore, ShelfNet is presently one of the most suitable architectures for real-world applications with resource constraints.

As depicted above, ShelfNet uses a ResNet backbone combined with 2 encoder/decoder branches. The first encoder (in green?) reduces channel complexity by a factor 4 for faster inference speed. The S-block is a residual block with shared-weights to significantly reduce the number of parameters. The network uses strided convolutions for down-sampling and transpose convolutions for up-sampling. The structure can be seen as an ensemble of FCN where the information flows through many different paths, resulting in increased accuracy.

Results on Microsoft COCO KeyPoints

This section reports test results for ShelfNet50 on the famous MS COCO KeyPoints dataset, and makes a comparison with the state of the art HRNet. All experiments use the same person detector which has AP of 56.4 on COCO val2017 dataset. You can find the download link on the HRNet repository. A single Titan RTX with 24GB RAM was used for the ShelfNet50 experiments. The batch size is 128 for an input size of 256x192 and 72 for 384x288.

Training on Your Own

I'm providing pre-trained weights for ShelfNet50 to make it easier to start. The test accuracies are obtained without providing the ground truth bounding boxes.

You can train and evaluate directly from the command line as such:

# Train ShelfNet on COCO
python train.py --cfg coco/shelfnet/shelfnet50_384x288_adam_lr1e-3.yaml

# Test ShelfNet on COCO
python test.py --cfg coco/shelfnet/shelfnet50_384x288_adam_lr1e-3.yaml TEST.MODEL_FILE ../output/coco/shelfnet/shelf_384x288_adam_lr1e-3/model_best.pth TEST.USE_GT_BBOX False
| Arch       | AP    | Ap .5 | AP .75| AP (M)| AP (L)| AR    | AR .5 | AR .75| AR (M)| AR (L)|
|------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| shelfnet   | 0.746 | 0.901 | 0.814 | 0.706 | 0.818 | 0.797 | 0.938 | 0.858 | 0.752 | 0.862 |

Requirements

Python 3.7, Torch 1.3.1 or greater, requests, tqdm, yacs, json_tricks, and pycocotools. Contrary to the ShelfNet repository, this repository is not based on torch-encoding.

Citation

Use this bibtex to cite this repository:

@misc{fmahoudeau_shelfnet_human_pose_2020,
  title={ShelfNet for Human Pose Estimation},
  author={Florent Mahoudeau},
  year={2020},
  publisher={GitHub},
  journal={GitHub repository},
  howpublished={\url{https://github.com/fmahoudeau/ShelfNet-Human-Pose-Estimation}},
}