Optimizers-for-Tensorflow
Adam, NAdam and AAdam (See below for details about this optimizer) optimizers
UPDATE July 2019: The accelerated solutions have been updated here , also the full paper explaining the solutions is available here.
Requirements
- Tensorflow (Last version)
- Python (3 or higher)
- Your computer :-)
How to use
# training
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
train_step = AAdamOptimizer(2e-3).minimize(cross_entropy)Results
UPDATE: I included 3 files to test the optimizers on mnist data. I do a small grid search for the learning rate so it can takes some time to execute. I recommand to use a cloud service if you do not have a GPU on your computer (mainly for the MLP and the CNN models).
I also included a small test file which implements a simple neural net and 8 optimizers including AAdam (the old version) and its variant. To run the code, simply go to command line an put python neuralnets-testing-optimizers.py. You don't need tensorflow to run this file and to quickly view the difference between optimizers.
However, I recommand for more interesting cases that you use the separated pyhton files.
The optimizers are tested on the make_moons toy data set by sklearn, availaible here
Here are the results so far ... (leraning rate = 1e-2)
rmsprop => mean accuracy: 0.8698666666666667, std: 0.007847434117099816
sgd => mean accuracy: 0.8794666666666666, std: 0.001359738536958064
adam => mean accuracy: 0.872, std: 0.009074506414492586
aadam1 => mean accuracy: 0.8741333333333333, std: 0.006607739569794042 <-- without the sign of the gradient
nesterov => mean accuracy: 0.864, std: 0.021496046148071032
aadam2 => mean accuracy: 0.8784000000000001, std: 0.0011313708498984561 <-- with the sign of the gradient
adagrad => mean accuracy: 0.7981333333333334, std: 0.11408036153908735
momentum => mean accuracy: 0.7970666666666667, std: 0.025884529914388783
Enjoy !