TFusion

CVPR2018: Unsupervised Cross-dataset Person Re-identification by Transfer Learning of Spatio-temporal Patterns

• We present a novel method to learn pedestrians' spatio-temporal patterns in unlabeled target datsets by transferring the visual classifier from the source dataset. The algorithm does not require any prior knowledge about the spatial distribution of cameras nor any assumption about how people move in the target environment.

• We propose a Bayesian fusion model, which combines the spatio-temporal patterns learned and the visual features to achieve high performance of person Re-ID in the unlabeled target datasets.

• We propose a learning-to-rank based mutual promotion procedure, which uses the fusion classifier to teach the weaker visual classifier by the ranking results on unlabeled dataset. This mutual learning mechanism can be applied to many domain adaptation problems.

This code is ONLY released for academic use.

How to use

We split TFusion into two components:

• rank-reid
  • Framework: Keras and Tensorflow
  • Training Resnet based Siamese network on source dataset
  • Learning to rank on target dataset
• TrackViz
  • Dependencies: Some traditional libraries, including numpy, pickle, matplotlib, seaborn  - Building spatial temporal model with visual classification results
  • Bayesian Fusion

Components communicate by ranking results(those *.log files ). We use this results for visualization and logical analysis in our experiments, thus we save them on file system in TrackViz/data.

Written and tested in python2, keras2.1.5, tensorflow 1.4.

Attention: make sure you are using the repos specified in TFusion, corresponding to [email protected] and [email protected] You are possible to meet some errors if you use other version repos.

Dataset

Download

• CUHK01
• VIPeR
• Market-1501
• GRID

Pre-process

• CUHK01

we only use CUHK01 as source dataset, so we use all images for pretrain, place all images in a directory.

• VIPeR

the same as CUHK01.

• GRID as Source dataset

we use all labeled images in GRID for pretraining as source dataset, so place all labeled images in a directory, for example "grid_label"

• Market-1501

  • download
  • rename training directory to 'train', rename probe directory to 'probe', renmae gallery directory to 'test'

• GRID as Target Dataset

  • follow dataset instruction, split the dataset to ten cross-validation sets
  • in each cross-validation set, rename training directory to 'train', rename probe directory to 'probe', renmae gallery directory to 'test'
  • you can also refer to 'TrackViz/data/grid' for more details about GRID cross validation.

Finally, your data will look like this:

Market-1501
├── probe
│   ├── 0003_c1s6_015971_00.jpg
│   ├── 0003_c3s3_064744_00.jpg
│   ├── 0003_c4s6_015641_00.jpg
│   ├── 0003_c5s3_065187_00.jpg
│   └── 0003_c6s3_088392_00.jpg
├── test
│   ├── 0003_c1s6_015971_02.jpg
│   ├── 0003_c1s6_015996_02.jpg
│   ├── 0003_c4s6_015716_03.jpg
│   ├── 0003_c5s3_065187_01.jpg
│   ├── 0003_c6s3_088392_04.jpg
│   └── 0003_c6s3_088442_04.jpg
└── train
    ├── 0002_c1s1_000451_03.jpg
    ├── 0002_c1s1_000551_01.jpg
    ├── 0002_c1s1_000776_01.jpg
    ├── 0002_c1s1_000801_01.jpg
    ├── 0002_c1s1_069056_02.jpg
    └── 0002_c6s1_073451_02.jpg

grid_train_probe_gallery
├── cross0
│   ├── probe
│   │   ├── 0002_1_25008_169_19_94_224.jpeg
│   │   ├── 0003_1_25008_57_44_97_265.jpeg
│   │   ├── 0004_1_25072_204_72_106_277.jpeg
│   │   └── 0005_1_25120_210_22_84_215.jpeg
│   ├── test
│   │   ├── 0000_1_25698_101_16_87_246.jpeg
│   │   ├── 0000_1_26113_116_13_72_212.jpeg
│   │   ├── 0000_1_26207_113_25_69_172.jpeg
│   │   └── gallery.txt
│   └── train
│       ├── 0001_1_25004_107_32_106_221.jpeg
│       ├── 0001_2_25023_116_134_128_330.jpeg
│       ├── 0009_1_25208_126_19_71_215.jpeg
│       ├── 0009_2_25226_176_72_87_246.jpeg
│       └── 0248_5_33193_101_100_90_308.jpeg
├── cross1
├── cross2
├── cross3
├── cross4
├── cross5
├── cross6
├── cross7
├── cross8
└── cross9

Place all datasets in the same directory, like this:

dataset
├── cuhk01
├── grid_train_probe_gallery
├── Market-1501
└── source

Configuration

• Pretrain Config: Modify all path containing '/home/cwh' appearing in rank-reid/pretrain/pair_train.py to your corresponding path.
• Fusion Config
  • Modify all path containing '/home/cwh' appearing in TrackViz/ctrl/transfer.py to your corresponding path.
  • Modify all path containing '/home/cwh' appearing in rank-reid/rank-reid.py to your corresponding path.

Pretrain

Pretrain Resnet52 and Siamese Network using source datasets.

cd rank-reid/pretrain && python pair_train.py

This code will save pretrained model in pair-train directory:

pretrain
├── cuhk_pair_pretrain.h5
├── cuhk_softmax_pretrain.h5
├── eval.py
├── grid-cv-0_pair_pretrain.h5
├── grid-cv-0_softmax_pretrain.h5
├── grid-cv-1_pair_pretrain.h5
├── grid-cv-1_softmax_pretrain.h5
├── grid-cv-2_pair_pretrain.h5
├── grid-cv-2_softmax_pretrain.h5
├── grid-cv-3_pair_pretrain.h5
├── grid-cv-3_softmax_pretrain.h5
├── grid-cv-4_pair_pretrain.h5
├── grid-cv-4_softmax_pretrain.h5
├── grid-cv-5_pair_pretrain.h5
├── grid-cv-5_softmax_pretrain.h5
├── grid-cv-6_pair_pretrain.h5
├── grid-cv-6_softmax_pretrain.h5
├── grid-cv-7_pair_pretrain.h5
├── grid-cv-7_softmax_pretrain.h5
├── grid-cv-8_pair_pretrain.h5
├── grid-cv-8_softmax_pretrain.h5
├── grid-cv-9_pair_pretrain.h5
├── grid-cv-9_softmax_pretrain.h5
├── grid_pair_pretrain.h5
├── grid_softmax_pretrain.h5
├── __init__.py
├── market_pair_pretrain.h5
├── market_softmax_pretrain.h5
├── pair_train.py
├── pair_transfer.py
├── source_pair_pretrain.h5
└── source_softmax_pretrain.h5

TFusion

include directly vision transfering, fusion, learning to rank

cd TrackViz && python ctrl/transfer.py

Results will be saved in TrackViz/data

TrackViz/data
├── source_target-r-test # transfer after learning to rank on test set
│   ├── cross_filter_pid.log
│   ├── cross_filter_score.log
│   ├── renew_ac.log
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
├── source_target-r-train # transfer after learning to rank on training set
│   ├── cross_filter_pid.log
│   ├── cross_filter_score.log
│   ├── cross_mid_score.log
│   ├── renew_ac.log
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
├── source_target-r-train_diff # ST model built by random classifier minus visual classfier after learning to rank
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
├── source_target-r-train_rand  # ST model built by random classifier after learning to rank
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
├── source_target-test # directly transfer from source to target test set
│   ├── cross_filter_pid_32.log
│   ├── cross_filter_pid.log
│   ├── cross_filter_score.log
│   ├── renew_ac.log
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
├── source_target-train # directly transfer from source to  target training set
│   ├── cross_filter_pid.log # sorted pids by fusion scores
│   ├── cross_filter_score.log # sorted fusion scores corresponding to pids
│   ├── cross_mid_score.log # can be use to generate pseudo lable, ignore it 
│   ├── renew_ac.log #  sorted vision scores corresponding to pids
│   ├── renew_pid.log # sorted pids by vision scores
│   └── sorted_deltas.pickle # store time deltas, so called ST model built by visual classifier
├── source_target-train_diff # store time deltas, ST model built by random classifier minus visual classifier
│   ├── renew_pid.log
│   └── sorted_deltas.pickle
└── source_target-train_rand # store time deltas, built by random visual classifier
    ├── renew_pid.log
    └── sorted_deltas.pickle

Evaluation

Evaluation result will be automatically saved in the log_path, as you specified in rank-reid/rank-reid.py predict_eval(), default location is TrackViz/market_result_eval.log, TrackViz/grid_eval.log

• GRID evaluation includes rank1, rank5, rank-10 accuracy
• Market-1501 evaluation includes rank1 accuracy and mAP. Rank5 and rank10 should be computed by code in MATLAB provided by Liang Zheng.

Citation

Please cite this paper in your publications if it helps your research:

@inproceedings{DBLP:conf/cvpr/LvCLY18,
  author    = {Jianming Lv and
               Weihang Chen and
               Qing Li and
               Can Yang},
  title     = {Unsupervised Cross-Dataset Person Re-Identification by Transfer Learning
               of Spatial-Temporal Patterns},
  booktitle = {2018 {IEEE} Conference on Computer Vision and Pattern Recognition,
               {CVPR} 2018, Salt Lake City, UT, USA, June 18-22, 2018},
  pages     = {7948--7956},
  year      = {2018},
  crossref  = {DBLP:conf/cvpr/2018},
  url       = {http://openaccess.thecvf.com/content\_cvpr\_2018/html/Lv\_Unsupervised\_Cross-Dataset\_Person\_CVPR\_2018\_paper.html},
  doi       = {10.1109/CVPR.2018.00829},
  timestamp = {Mon, 07 Jan 2019 17:17:41 +0100},
  biburl    = {https://dblp.org/rec/bib/conf/cvpr/LvCLY18},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@proceedings{DBLP:conf/cvpr/2018,
  title     = {2018 {IEEE} Conference on Computer Vision and Pattern Recognition,
               {CVPR} 2018, Salt Lake City, UT, USA, June 18-22, 2018},
  publisher = {{IEEE} Computer Society},
  year      = {2018},
  url       = {http://openaccess.thecvf.com/CVPR2018.py},
  timestamp = {Mon, 07 Jan 2019 12:43:48 +0100},
  biburl    = {https://dblp.org/rec/bib/conf/cvpr/2018},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}