PyTorch implementation of popular two-stream frameworks for video action recognition

Current release is the PyTorch implementation of the "Towards Good Practices for Very Deep Two-Stream ConvNets". You can refer to paper for more details at Arxiv.

If you find this implementation useful in your work, please acknowledge it appropriately and cite the paper or code accordingly:

@article{zhu_arxiv2020_comprehensiveVideo,
  title={A Comprehensive Study of Deep Video Action Recognition},
  author={Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, Mu Li},
  journal={arXiv preprint arXiv:2012.06567},
  year={2020}
}

@inproceedings{wang_eccv2016_tsn,
  title={Temporal Segment Networks: Towards Good Practices for Deep Action Recognition},
  author={Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2016}
}

@inproceedings{zhu_accv2018_hidden,
  title={Hidden Two-Stream Convolutional Networks for Action Recognition},
  author={Yi Zhu, Zhenzhong Lan, Shawn Newsam, Alexander G. Hauptmann},
  booktitle={Asian Conference on Computer Vision (ACCV)},
  url={https://arxiv.org/abs/1704.00389}
  year={2018}
}

If you are looking for a good-to-use codebase with a large model zoo, please checkout the video toolkit at GluonCV. We have SOTA model implementations (TSN, I3D, NLN, SlowFast, etc.) for popular datasets (Kinetics400, UCF101, Something-Something-v2, etc.) in both PyTorch and MXNet. We also have accompaning survey paper and video tutorial.

Installation

Tested on PyTorch:

OS: Ubuntu 16.04
Python: 3.5
CUDA: 8.0
OpenCV3
dense_flow

To successfully install dense_flow(branch opencv-3.1), you probably need to install opencv3 with opencv_contrib. (For opencv-2.4.13, dense_flow will be installed more easily without opencv_contrib, but you should run code of this repository under opencv3 to avoid error)

Code also works for Python 2.7.

Data Preparation

Download data UCF101 and use unrar x UCF101.rar to extract the videos.

Convert video to frames and extract optical flow

python build_of.py --src_dir ./UCF-101 --out_dir ./ucf101_frames --df_path <path to dense_flow>

build file lists for training and validation

python build_file_list.py --frame_path ./ucf101_frames --out_list_path ./settings

Training

For spatial stream (single RGB frame), run:

python main_single_gpu.py DATA_PATH -m rgb -a rgb_resnet152 --new_length=1
--epochs 250 --lr 0.001 --lr_steps 100 200

For temporal stream (10 consecutive optical flow images), run:

python main_single_gpu.py DATA_PATH -m flow -a flow_resnet152
--new_length=10 --epochs 350 --lr 0.001 --lr_steps 200 300

DATA_PATH is where you store RGB frames or optical flow images. Change the parameters passing to argparse as you need.

Testing

Go into "scripts/eval_ucf101_pytorch" folder, run python spatial_demo.py to obtain spatial stream result, and run python temporal_demo.py to obtain temporal stream result. Change those label files before running the script.

For ResNet152, I can obtain a 85.60% accuracy for spatial stream and 85.71% for temporal stream on the split 1 of UCF101 dataset. The result looks promising. Pre-trained RGB_ResNet152 Model Pre-trained Flow_ResNet152 Model

For VGG16, I can obtain a 78.5% accuracy for spatial stream and 80.4% for temporal stream on the split 1 of UCF101 dataset. The spatial result is close to the number reported in original paper, but flow result is 5% away. There are several reasons, maybe the pretained VGG16 model in PyTorch is differnt from Caffe, maybe there are subtle bugs in my VGG16 flow model. Welcome any comments if you found the reason why there is a performance gap. Pre-trained RGB_VGG16 Model Pre-trained Flow_VGG16 Model

I am experimenting with memory efficient DenseNet now, will release the code in a couple of days. Stay tuned.

Related Projects

TSN: Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Hidden Two-Stream: Hidden Two-Stream Convolutional Networks for Action Recognition