Stochastic Delta Rule implementation using DenseNet in TensorFlow

THIS REPOSITORY IS NO LONGER IN USE. THE NEW SDR REPOSITORY CAN BE FOUND HERE.

NOTE This is repository is based off of Illarion Khlestov's DenseNet implementation. Check out his blog post about implementing DenseNet in TensorFlow here.

Check out @lifeiteng's results from implementing SDR with WaveNet.

UPDATE: Due to a bug found by @basveeling which has now been corrected, the testing errors are being recalculated. Here are the preliminary results, which I will continue to update as the results come out. "-----" indicates results that have not yet been redone.

This repository holds the code for the paper

'Dropout is a special case of the stochastic delta rule: faster and more accurate deep learning' (submitted to NIPS; on arXiv)

Noah Frazier-Logue, Stephen Jose Hanson

Stochastic Delta Rule (SDR) is a weight update mechanism that assigns to each weight a standard deviation that changes as a function of the gradients every training iteration. At the beginning of each training iteration, the weights are re-initialized using a normal distribution bound by their standard deviations. Over the course of the training iterations and epochs, the standard deviations converge towards zero as the network becomes more sure of what the values of each of the weights should be. For a more detailed description of the method and its properties, have a look at the paper [link here].

Two types of Densely Connected Convolutional Networks (DenseNets) are available:

• DenseNet - without bottleneck layers
• DenseNet-BC - with bottleneck layers

Each model can be tested on such datasets:

• CIFAR-10
• CIFAR-10+ (with data augmentation)
• CIFAR-100
• CIFAR-100+ (with data augmentation)
• SVHN

A number of layers, blocks, growth rate, image normalization and other training params may be changed trough shell or inside the source code.

Usage

Example run:

    python run_dense_net.py --depth=40 --train --test --dataset=C10 --sdr

This run uses SDR instead of dropout. To use dropout, run something like

    python run_dense_net.py --depth=40 --train --test --dataset=C10 --keep_prob=0.8

where keep_prob is the probability (in this case 80%) that a neuron is kept during dropout.

NOTE: the --sdr argument will override the --keep_prob argument. For example:

    python run_dense_net.py --depth=40 --train --test --dataset=C10 --keep_prob=0.8 --sdr

will use SDR and not dropout.

List all available options:

    python run_dense_net.py --help

There are also many other implementations - they may be useful.

Citation:

     @article{Huang2016Densely,
            author = {Huang, Gao and Liu, Zhuang and Weinberger, Kilian Q.},
            title = {Densely Connected Convolutional Networks},
            journal = {arXiv preprint arXiv:1608.06993},
            year = {2016}
     }

KNOWN ISSUES

• model will not save due to graph definiton being larger than 2GB

If you see anything wrong, feel free to open an issue!

Results from SDR paper

This table shows the results on CIFAR shown in the paper. Parameters are all the same as what are used in the paper, except for a batch size of 100 and an epoch size of 100. SDR's beta value was 0.1 and zeta was 0.01. The augmented datasets were not tested on because dropout was not used on these datasets in the original paper, however they may be added in the future (as will the SVHN results and results with higher layer counts).

Epochs to error rate

The below tables show the number of training epochs required to reach a training error of 15, 10, and 5, respectively. For example, the dropout version of DenseNet-40 on CIFAR-10 took 8 epochs to reach a training error of 15, 16 epochs to reach a training error of 10, and 94 epochs to reach a training error of 5. In contrast, the SDR version of DenseNet-40 on CIFAR-10 took 5 epochs to reach a training error of 15, 5 epochs to reach a training error of 10, and 15 epochs to reach a training error of 5. Best results for each value, across both dropout and SDR, are bolded.

Dropout

SDR

Comparison to original DenseNet implementation with dropout

Test results on various datasets. Image normalization per channels was used. Results reported in paper provided in parenthesis. For Cifar+ datasets image normalization was performed before augmentation. This may cause a little bit lower results than reported in paper.

Difference compared to the original implementation

The existing model should use identical hyperparameters to the original code.

Dependencies

• Model was tested with Python 3.4.3+ and Python 3.5.2 with and without CUDA.
• Model should work as expected with TensorFlow >= 0.10 FOR DROPOUT ONLY. SDR was added using a development environment with TensorFlow 1.7 so it may require 1.0+.

Repo supported with requirements files - so the easiest way to install all just run:

• in case of CPU usage pip install -r requirements/cpu.txt.
• in case of GPU usage pip install -r requirements/gpu.txt.