Implementation of our CVPR 2019 paper Distilling Object Detectors with Fine-grained Feature Imitation

We have proposed a general distillation approach for anchor based object detection model to get enhanced small student model with the knowledge of large teacher model, which is othorgonal and can be further combined with other model compression method like quantization and pruning. The key observation of vanilla knowledge distillation technique is that the inter-class discrepancy of perdiction confidence reveals how curmbersome model tends to genearlize (e.g., how much confidence the model would put on cat label when the input is actually a dog). While our idea is the inter-location discrepancy of feature response near object also reveals how large detector tends to generalize (e.g., how is the model's response different for different near object anchor locations).

We release the code for distilling shufflenet based detector and VGG11 based Faster R-CNN, this code repository implements Faster R-CNN imitation based on pytorch-faster-rcnn. Check Distilling-ShuffleDet for tensorflow code of Shufflenet based detector imitation.

🔥Updating🔥

TODO

We have accumulated the following to-do list, which we hope to complete in the near future

• Still to come:
  • [ ] Add more models(ResNet-FRCNN, FPN-FRCNN).
  • [ ] Implement SSD model distillation.

Distilling VGG11-FRCNN

pytorch 0.4.0 python2

Preparation

1 Clone the repository

First of all, clone the code

git clone https://github.com/twangnh/Distilling-Object-Detectors

Then, create a folder:

cd Distilling-Object-Detectors && mkdir data

2 Requirements and Compilation

pip install -r requirements.txt

Compile the cuda dependencies using following commands:

cd lib
sh make.sh

To use the .py file in 'lib', you should add the 'lib' to PYTHONPATH, so open bashrc and add the path to the bashrc file:

vim ~/.bashrc

export PYTHONPATH="${PYTHONPATH}:/the/lib/path/of/the/project"

After changing the bashrc file, update the bashrc by login again or directly update it by:

source ~/.bashrc

3 Data preparation

• PASCAL_VOC 07+12: Please follow the instructions in py-faster-rcnn to prepare VOC datasets. Actually, you can refer to any others. After downloading the data, create softlinks in the folder data/. The prepaired direcoty is like data/VOCdevkit2007/VOC2007/...

4 download imagenet pretrained model and trained VGG16-FRCNN teacher model

download imagenet pretrained VGG11 model at GoogleDrive and put it into data/pretrained_model/

download trained VGG16-FRCNN model at GoogleDrive and put it into data/VGG16-FRCNN/

Train

currently only batch size of 1 is supported

python trainval_net_sup.py --dataset pascal_voc --net vgg11 --bs 1 --nw 2 --lr 3e-3 --lr_decay_step 5 --cuda --s 1 --gpu 0

[session 1][epoch  1][iter    0/10022] loss: 13.4238, loss_sup: 0.0000, lr: 3.00e-03
			fg/bg=(15/241), time cost: 0.307381
			rpn_cls: 0.7839, rpn_box: 0.4312, rcnn_cls: 12.0130, rcnn_box 0.1957 
[session 1][epoch  1][iter  100/10022] loss: 2.1172, loss_sup: 0.0000, lr: 3.00e-03
			fg/bg=(15/241), time cost: 17.871297
			rpn_cls: 0.2372, rpn_box: 0.0492, rcnn_cls: 0.5382, rcnn_box 0.1475 
[session 1][epoch  1][iter  200/10022] loss: 2.3993, loss_sup: 0.0000, lr: 3.00e-03
			fg/bg=(27/229), time cost: 17.885193
			rpn_cls: 0.0451, rpn_box: 0.3003, rcnn_cls: 2.5547, rcnn_box 0.5216 
[session 1][epoch  1][iter  300/10022] loss: 1.6754, loss_sup: 0.0000, lr: 3.00e-03
			fg/bg=(21/235), time cost: 17.856990
			rpn_cls: 0.2837, rpn_box: 0.2542, rcnn_cls: 1.1131, rcnn_box 0.2073 
[session 1][epoch  1][iter  400/10022] loss: 1.6178, loss_sup: 0.1145, lr: 3.00e-03
			fg/bg=(23/233), time cost: 17.976755
			rpn_cls: 0.3597, rpn_box: 0.0106, rcnn_cls: 0.7343, rcnn_box 0.2363 
[session 1][epoch  1][iter  500/10022] loss: 1.4362, loss_sup: 9.6434, lr: 3.00e-03
			fg/bg=(32/224), time cost: 17.911143
			rpn_cls: 0.1783, rpn_box: 0.0235, rcnn_cls: 0.4522, rcnn_box 0.3731 
[session 1][epoch  1][iter  600/10022] loss: 1.3638, loss_sup: 8.4568, lr: 3.00e-03
			fg/bg=(18/238), time cost: 18.024369
			rpn_cls: 0.4774, rpn_box: 0.1143, rcnn_cls: 0.4781, rcnn_box 0.1663 

training should progress as above, where loss_sup is the imitation loss and firts 400 steps are warmup steps with no imitation (i.e., loss_sup = 0.) models will be saved in ./temp/vgg11/pascal_voc/xxx.pth

Train without imitation (baseline training)

currently only batch size of 1 is supported

python trainval_net_sup.py --dataset pascal_voc --net vgg11 --bs 1 --nw 2 --lr 3e-3 --lr_decay_step 5 --cuda --s 1 --gpu 0 --tfi True

models will be saved in ./temp/vgg11/pascal_voc/xxx.pth Note the imitation loss weight and warm up step can be further tuned with --ilw and --ws

Test

python test_net.py --dataset pascal_voc --net vgg11 --checksession 1 --checkepoch 2 --checkpoint 10021 --cuda --gpu 0

change checksession, checkepoch, checkpoint to test specific model

models at max_epoch are reported

the numbers are different from the paper as they are independent running of the algorithm.

Test with trained model

download the trained model at the GoogleDrive link, run

python test_net.py --dataset pascal_voc --net vgg11 --load_name ./path_to/xxx.pth --cuda --gpu 0

Distilling ShuffleDet

...

We have implemented a single layer one-stage toy object detector with tensorflow, and mutli-gpu training with cross-gpu batch normalization, check Distilling-ShuffleDet for codes

models with highest mAP are reported for both baseline and distilled model

Note the numbers are different from the paper as they are independent running of the algorithm and we have migrated from single GPU training to multi-gpu training with larger batch size.

Distilling YoloV2

Third party implementation of distilling YOLOV2 on Widerface(codes not available yet, but very easy to implement)

Citation

@inproceedings{wang2019distilling,
  title={Distilling Object Detectors With Fine-Grained Feature Imitation},
  author={Wang, Tao and Yuan, Li and Zhang, Xiaopeng and Feng, Jiashi},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={4933--4942},
  year={2019}
}

License

The code and the models are MIT licensed, as found in the LICENSE file.