Selective Joint Fine-tuning

By [Weifeng Ge], Yizhou Yu

Department of Computer Science, The University of Hong Kong

Table of Contents

0. Introduction
1. Citation
2. Pipeline
3. Codes and Installation
4. Models
5. Results

Introduction

This repository contains the codes and models described in the paper "Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning"(https://arxiv.org/abs/1702.08690). These models are those used in Stanford Dogs 120, Oxford Flowers 102, Caltech 256 and MIT Indoor 67.

Note

0. All algorithms are implemented based on the deep learning framework Caffe.
1. Please add the additional layers used into your own Caffe to run the training codes.

Citation

If you use these codes and models in your research, please cite:

   @InProceedings{Ge_2017_CVPR,
           author = {Ge, Weifeng and Yu, Yizhou},
           title = {Borrowing Treasures From the Wealthy: Deep Transfer Learning Through Selective Joint Fine-Tuning},
           booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
           month = {July},
           year = {2017}
   }

Pipeline

0. Pipeline of the proposed selective joint fine-tuning:

Codes and Installation

0. Add new layers into Caffe:

   • caffe.proto
   • MergeData
   • SplitData
   • RandomCropBoostedData
   • FeatureStatistics
   • NormalKnnMatch
   • RefinedHistFeature
   • Residual

1. Image Retrieval:

   • get the features statistics
   • extract histogram features
   • online nearest neighbor searching

2. Selective Joint Fine-tuning:

   • joint fine-tuning

Models

0. Visualizations of network structures (tools from ethereon):

   • [Selective Joint Fine-tuning: ResNet-152] (http://ethereon.github.io/netscope/#/gist/8bdda026e3391eacfa43cc24f4f4a9ff)

1. Model files:

   • ResNet 152 Net pretrained on ImageNet ILSVRC 2012: deploy, model.
   • ResNet 152 Net pretrained on Places 205: deploy, model.
   • Stanford Dogs 120: deploy, model.
   • Oxford Flowers 102: deploy, model.
   • Caltech 256: deploy, model.
   • Mit Indoor 67: deploy, model.

Results

0. Multi crop testing accuracy on Stanford Dogs 120 (in the same manner with that in VGG-net):

   Method	mean Accuracy(%)
   HAR-CNN	49.4
   Local Alignment	57.0
   Multi Scale Metric Learning	70.3
   MagNet	75.1
   Web Data + Original Data	85.9
   Target Only Training from Scratch	53.8
   Selective Joint Training from Scratch	83.4
   Fine-tuning w/o source domain	80.4
   Selective Joint FT with all source samples	85.6
   Selective Joint FT with random source samples	85.5
   Selective Joint FT w/o iterative NN retrieval	88.3
   Selective Joint FT with Gabor filter bank	87.5
   Selective Joint FT	90.2
   Selective Joint FT with Model Fusion	90.3

1. Multi crop testing accuracy on Oxford Flowers 102 (in the same manner with that in VGG-net):

   Method	mean Accuracy(%)
   MPP	91.3
   Multi-model Feature Concat	91.3
   MagNet	91.4
   VGG-19 + GoogleNet + AlexNet	94.5
   Target Only Training from Scratch	58.2
   Selective Joint Training from Scratch	80.6
   Fine-tuning w/o source domain	90.2
   Selective Joint FT with all source samples	93.4
   Selective Joint FT with random source samples	93.2
   Selective Joint FT w/o iterative NN retrieval	94.2
   Selective Joint FT with Gabor filter bank	93.8
   Selective Joint FT	94.7
   Selective Joint FT with Model Fusion	95.8
   VGG-19 + Part Constellation Model	95.3
   Selective Joint FT with val set	97.0

2. Multi crop testing accuracy on Caltech 256 (in the same manner with that in VGG-net):

   Method	mean Acc(%) 15/class	mean Acc(%) 30/class	mean Acc(%) 45/class	mean Acc(%) 60/class
   M-HMP	40.5 ± 0.4	48.0 ± 0.2	51.9 ± 0.2	55.2 ± 0.3
   Z.&F. Net	65.7 ± 0.2	70.6 ± 0.2	72.7 ± 0.4	74.2 ± 0.3
   VGG-19	-	-	-	85.1 ± 0.3
   VGG-19 + GoogleNet + AlexNet	-	-	-	86.1
   VGG-19 + VGG-16	-	-	-	86.2 ± 0.3
   Fine-tuning w/o source domain	76.4 ± 0.1	81.2 ± 0.2	83.5 ± 0.2	86.4 ± 0.3
   Selective Joint FT	80.5 ± 0.3	83.8 ± 0.5	87.0 ± 0.1	89.1 ± 0.2

3. Multi crop testing accuracy on MIT Indoor 67 (in the same manner with that in VGG-net):

   Method	mean Accuracy(%)
   MetaObject-CNN	78.9
   MPP + DFSL	80.8
   VGG-19 + FV	81.0
   VGG-19 + GoogleNet	84.7
   Multi Scale + Multi Model Ensemble	86.0
   Fine-tuning w/o source domain	81.7
   Selective Joint FT with ImageNet	82.8
   Selective Joint FT with Places	85.8
   Selective Joint FT with hybrid data	85.5
   Average the output of Places and hybrid data	86.9