Tensorflow Retrieval Baseline
This repository provides a retrieval/space embedding baseline using multiple retrieval datasets and ranking losses. This code is based on triplet-reid repos.
Evaluation Metrics
- Normalized Mutual Information (NMI)
- Recall@K
Deep Fashion In-shop Retrieval Evaluation
All the following experiments assume a training mini-batch of size 60. The architecture employed is the one used in In Defense of the Triplet Loss for Person Re-Identification but ResNet is replaced by a DenseNet169. Optimizer: Adam, Number of iterations = 25K
| Method | Normalized | Margin | NMI | R@1 | R@4 | # of classes | #samples per class |
|---|---|---|---|---|---|---|---|
| Semi-Hard | Yes | 0.2 | 0.902 | 87.43 | 95.42 | 10 | 6 |
| Hard-Negative | No | 1.0 | 0.904 | 88.38 | 95.74 | 10 | 6 |
| Lifted Structured | No | 1.0 | 0.903 | 87.32 | 95.59 | 10 | 6 |
| N-Pair Loss | No | N/A | 0.903 | 89.12 | 96.13 | 30 | 2 |
| Angular Loss | Yes | N/A | 0.8931 | 84.70 | 92.32 | 30 | 2 |
| Custom Contrastive Loss | Yes | 1.0 | 0.826 | 44.09 | 67.17 | 15 | 4 |
CUB200-2011 Retrieval Evaluation
Mini-batch size=120. Architecture: Inception_Net V1. Optimizer: Momentum. Number of iterations = 10K
| Method | Normalized | Margin | NMI | R@1 | R@4 | # of classes | #samples per class |
|---|---|---|---|---|---|---|---|
| Semi-Hard | Yes | 0.2 | 0.587 | 49.03 | 73.43 | 20 | 6 |
| Hard Negatives | No | 1.0 | 0.561 | 46.55 | 71.03 | 20 | 6 |
| Lifted Structured | No | 1.0 | 0.502 | 35.26 | 59.82 | 20 | 6 |
| N-Pair Loss | No | N/A | 0.573 | 46.52 | 59.26 | 60 | 2 |
| Angular Loss | Yes | N/A | 0.546 | 45.50 | 68.43 | 60 | 2 |
| Custom Contrastive Loss | Yes | 1.0 | 0.476 | 37.27 | 62.39 | 30 | 4 |
Stanford Online Products Retrieval Evaluation
Mini-batch size=120. Architecture: Inception_Net V1. Optimizer: Adam. Number of iterations = 30K
| Method | Normalized | Margin | NMI | R@1 | R@4 | # of classes | #samples per class |
|---|---|---|---|---|---|---|---|
| Semi-Hard | Yes | 0.2 | 0.893 | 71.22 | 81.77 | 20 | 6 |
| Hard Negatives | No | 1.0 | 0.895 | 72.03 | 82.55 | 20 | 6 |
| Lifted Structured | No | 1.0 | 0.889 | 68.26 | 79.72 | 20 | 6 |
| N-Pair Loss | No | N/A | 0.893 | 72.60 | 82.59 | 60 | 2 |
| Angular Loss | Yes | N/A | 0.878 | 60.30 | 72.78 | 60 | 2 |
| Custom Contrastive Loss | Yes | 1.0 | 0.825 | 19.05 | 32.28 | 30 | 4 |
Requirements
- Python 3+ [Tested on 3.4.7 / 3.7]
- Tensorflow 1 and TF 2.0 [Tested on 1.8 / 1.14 / 2.0]
Code Setup
- Update the directories' paths in constants.py.
- Use train.py and train_tf2.py for TF 1.X and TF 2.X, respectively.
- Use embed.py and embed_tf2.py for TF 1.X and TF 2.X, respectively.
- eval.py.
Supported Ranking losses
- Triplet Loss with hard mining - 'hard_triplet'
- Triplet Loss with semi-hard mining - 'semi_hard_triplet'
- Lifted Structure Loss - 'lifted_loss'
- N-pairs loss - 'npairs_loss'
- Angular loss - 'angular_loss'
- Contrastive loss - 'contrastive_loss'
Keep an eye on ranking/__init__.py for new ranking loss
Recommeneded Setting for each loss
| Method | Setting |
|---|---|
| Semi-Hard | L2-Norm Yes, Margin =0.2 |
| Hard Negatives | L2-Norm No , Margin =1.0 |
| Lifted Structured | L2-Norm No , Margin =1.0 |
| N-Pair Loss | L2-Norm No , Margin =N/A |
| Angular Loss | L2-Norm Yes, Margin =N/A |
| Custom Contrastive Loss | L2-Norm Yes, Margin =1.0 |
Wiki
- [Done] Explain the fast contrastive loss sampling procedure
- [Done] The contrastive loss in the repos is customized to avoid nan during training. When the anchor and positive belong to the same class and the distance between their embeddings is near zero, the derivative turns into nan. Lei Mao provides a nice detailed mathematical explanation for this issue.
TODO
- [TODO] bash script for train, embed and then eval
- [TODO] Evaluate space embedding during training.
- [TODO] After supporting TF 2.0 (eager execution), It become easier to support more losses -- Maybe add Margin loss.
Misc Notes
- I noticed that some methods depend heavily on training parameters like the optimizer and number of iterations. For example, the semi-hard negative performance drops significantly on CUB-dataset if Adam optimizer is used instead of Momentum! The number of iterations seems also matter for this small dataset.
- The Tensorflow 2.0 implementation uses more memory even when disabling the eager execution. I tested the code with a smaller batch size -- ten classes and five samples per class. After training for 10K iterations, the performance achieved is NMI=0.54, R@1=42.64, R@4=66.52.
Release History
- 0.0.1
- CHANGE: Jan 8, 2020. Update code to support Tensorflow 2.0
- CHANGE: Dec 31, 2019. Update code to support Tensorflow 1.14
- First Commit: May 24, 2019. Code tested on Tensorflow 1.8