ducha-aiki / Caffenet Benchmark

Welcome to evaluation of CNN design choises performance on ImageNet-2012. Here you can find prototxt's of tested nets and full train logs.

upd.: Here is technical report version of this benchmark

If you use results from this benchmark, please cite

@Article{CaffeNetBench2017,
  Title                    = {Systematic evaluation of convolution neural network advances on the Imagenet },
  Author                   = {Dmytro Mishkin and Nikolay Sergievskiy and Jiri Matas},
  Journal                  = {Computer Vision and Image Understanding },
  Year                     = {2017},
  Doi                      = {https://doi.org/10.1016/j.cviu.2017.05.007},
  ISSN                     = {1077-3142},
  Keywords                 = {CNN},
  Url                      = {http://www.sciencedirect.com/science/article/pii/S1077314217300814}
}

**upd2.: Some of the pretrained models are in Releases section. They are licensed for unrestricted use.

***upd3.: Nice paper on noise sensitiveness: Fine-grained Recognition in the Noisy Wild: Sensitivity Analysis of Convolutional Neural Networks Approaches

The basic architecture is similar to CaffeNet, but has several differences:

1. Images are resized to small side = 128 for speed reasons. Therefore pool5 spatial size is 3x3 instead of 6x6.
2. fc6 and fc7 layers have 2048 neurons instead of 4096.
3. Networks are initialized with LSUV-init (code)
4. Because LRN layers add nothing to accuracy (validated here), they were removed for speed reasons in most experiments.

Taking into account Neural Network Training Variations in Speech and Subsequent Performance Evaluation, results can vary from run to run (data order is the same, but random seeds are different). However, I haven`t experienced results difference for several CaffeNet-ReLU training runs.

On-going evaluations with graphs:

• activations
• pooling
• solvers
• lr_policy
• architectures
• First layer parameters
• Conv1 depth
• classfier architectures
• augmentation
• batchnorm
• colorspace
• regularization
• resnets, not yet successfull
• batch size
• dataset size
• Network width
• other mix

Activations

The above analyses show that the bottom layers seem to waste a large portion of the additional parametrisation (figure 2 (a,e)) thus could be replaced, for example, by smaller ReLU layers. Similarly, maxout units in higher layers seem to use piecewise-linear components in a more active way suggesting the use of larger pools._

Prototxt, logs

Pooling type

Pooling window/stride

Prototxt, logs

CLF architecture

pool5pad following nets mistakenly were trained with ELU non-linearity instead of default ReLU

Prototxt, logs

Conv1 parameters

prototxt, logs

Squeezing representation

For example, for using activations in image retrieval.

Prototxt, logs

Solvers

Not converge at all:

ADAM: lr=0.001 m=0.9 m2=0.999 delta=1e-8 lr=0.001 m=0.95 m2=0.999 delta=1e-8 lr=0.001 m=0.95 m2=0.999 delta=1e-7 lr=0.01 m=0.9 m2=0.999 delta=1e-8 lr=0.01 m=0.9 m2=0.999 delta=1e-7 lr=0.01 m=0.9 m2=0.999 delta=1e-9 lr=0.01 m=0.9 m2=0.99 delta=1e-8 lr=0.01 m=0.9 m2=0.999 delta=1e-8 lr=0.01 m=0.95 m2=0.999 delta=1e-9

AdaDelta: delta: 1e-5

RMSProp, lr=0.01, rms_decay=0.5 lr=0.01, rms_decay=0.9 lr=0.01, rms_decay=0.95 lr=0.01, rms_decay=0.98 lr=0.001, rms_decay=0.9 lr=0.001, rms_decay=0.98

Converge, but much worse that SGD: Adagrad, lr=0.01, lr=0.02 AdaDelta: delta: 1e-6, delta: 1e-7, delta: 1e-8 RMSProp, lr=0.01, rms_decay=0.99

Prototxt, logs

LR-policy

LR-policy-BatchNorm-Dropout = 0.2

Prototxt, logs

Regularization

Prototxt, logs

Dropout and width

Hypothesis about "same effective neurons = same performance" looks unvalidated

Prototxt, logs

Architectures

CaffeNet only

Others

ResNet attempts are moved to ResNets.md

ResNets, good attempts

Train augmentation

Prototxt, logs

Colorspace

Prototxt, logs

Batch normalization

BN-paper, caffe-PR Note, that results are obtained without mentioned in paper y=kx+b additional layer.

BN -- before or after ReLU?

So in all next experiments, BN is put after non-linearity

BN and activations

BN and dropout

ReLU non-linearity, fc6 and fc7 layer only

Prototxt, logs

BN-arch-init

Prototxt, logs

Batch size, ReLU

Tanh results are moved [here] (https://github.com/ducha-aiki/caffenet-benchmark/blob/master/BatchSize.md)

Prototxt, logs

So general recommendation: too big batch_sizes leads to a bit inferior results, but in general batch_size should be selected based computation speed. If learning rate is adjusted, than no practial differenc e between different batch sizes.

From contributors

Base net is caffenet+BN+ReLU+drop=0.2 There difference in filters (main, 5x5 -> 3x3 + 3x3 or 1x5+5x1) and solver.

Prototxt, logs

Residual experiments

Prototxt, logs

Network width

logs

Dataset size

Dataset size, no RGB scaling

Or why input var=1 for LSUV is so important

logs

Input image size

logs

Dataset quality

logs

Conv1 depth

logs

Other

ReLU non-linearity, fc6 and fc7 layer only

Prototxt, logs

The PRs with test are welcomed

P.S. Logs are merged from lots of "save-resume", because were trained at nights, so plot "Anything vs. seconds" will give weird results.