Saliency Weighted Convolutional features for Instance Search

Eva Mohedano	Kevin McGuinness	Xavier Giro-i-Nieto	Noel O'Connor

A joint collaboration between:

Insight Centre for Data Analytics	Dublin City University (DCU)	Universitat Politecnica de Catalunya (UPC)	UPC ETSETB TelecomBCN	UPC Image Processing Group

Publication

Find our paper at arXiv

@article{mohedano2017saliency,
  title={Saliency Weighted Convolutional Features for Instance Search},
  author={Mohedano, Eva and McGuinness, Kevin and Giro-i-Nieto, Xavier and O'Connor, Noel E},
  journal={arXiv preprint arXiv:1711.10795},
  year={2017}
}

Abstract

This work explores attention models to weight the contribution of local convolutional representations for the instance search task. We present a retrieval framework based on bags of local convolutional features (BLCF) that benefits from saliency weighting to build an efficient image representation. The use of human visual attention models (saliency) allows significant improvements in retrieval performance without the need to conduct region analysis or spatial verification, and without requiring any feature fine tuning. We investigate the impact of different saliency models, finding that higher performance on saliency benchmarks does not necessarily equate to improved performance when used in instance search tasks. The proposed approach outperforms the state-of-the-art on the challenging INSTRE benchmark by a large margin, and provides similar performance on the Oxford and Paris benchmarks compared to more complex methods that use off-the-shelf representations.

Code Instructions

This repo contains python scripts to build Bag of Visual Words based on local CNN features to perform instance search in three different datasets:

• Instre, following the evaluation protocol from Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations.
• Oxford Buildings.
• Paris Buildings.

Prerequisits

• Create a virtual enviroment how to create a virtual enviroment.

 virtualenv ~/salbow
 source ~/salbow/bin/activate

• The code runs with CUDA Version 7.5.18. For python dependencies run:

 pip install --upgrade pip
 pip install -r requirements.txt

• Then, install the custom python-vlfeat library by running:

 python install_vlfeat.py

• Lastly, modify config.py file to set custom paths: Dataset images are automatically downloaded in:

PATH_DATASET='custom/dataset/path'

Precomputed data contained saliency predictions for the three datasets, and BLCF models, assignment maps and raw features.

saliency masks for each of the datasets are stored in:

PATH_SALIENCY='custom/saliency/path'

BLCF models, features and assignment maps are stored in:

PATH_OUTPUT='custom/output/path'

How to run it

Usage: python evaluation.py [OPTIONS]

Options:
  --dataset TEXT     Selected dataset for extraction  (availables 'instre' (default), 'oxford', 'paris')
  --layer TEXT       layer from vgg16                 (default 'conv5_1')
  --max_dim INTEGER  Max dimension of images          (default '340')
  --weighting TEXT   Spatial weighting scheme         (availables None (default), 'gaussian', 'l2norm', 'SALGAN')
  --global_search    Flag to apply global search for queries        
  --query_expansion  Flag to apply Average Query Expansion    

Example:

python evaluation.py --dataset 'instre' --query_expansion --weighting 'SALGAN'

ret:
mAP = 0.697773325515
+QE mAP = 0.757181174096

The command above applies saliency weighting from SalGAN to the assignment maps of Instre, reporting mAP when performing query expansion (top 10 retrieved images). Results using precomputed data.

Acknowledgements

This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289 and SFI/15/SIRG/3283.
The Image ProcessingGroup at the UPC is a SGR14 Consolidated Research Group recognized and sponsored by the Catalan Government (Generalitat de Catalunya) through its AGAUR office.
This work has been developed in the framework of the project BigGraph TEC2013-43935-R, funded by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF).

Contact

If you have any general doubt about our work or code which may be of interest for other researchers, please use the public issues section on this github repo. Alternatively, drop us an e-mail at mailto:[email protected].