All Projects → ilija139 → Hord

ilija139 / Hord

Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Hord

Awesome Automl And Lightweight Models
A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering.
Stars: ✭ 691 (+597.98%)
Mutual labels:  hyperparameter-optimization
Deeplearning
This repository will contain the example detailed codes of Tensorflow and Keras, This repository will be useful for Deep Learning staters who find difficult to understand the example codes
Stars: ✭ 49 (-50.51%)
Mutual labels:  deep-learning-algorithms
Xcessiv
A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.
Stars: ✭ 1,255 (+1167.68%)
Mutual labels:  hyperparameter-optimization
Image Text Localization Recognition
A general list of resources to image text localization and recognition 场景文本位置感知与识别的论文资源与实现合集 シーンテキストの位置認識と識別のための論文リソースの要約
Stars: ✭ 788 (+695.96%)
Mutual labels:  deep-learning-algorithms
Letslearnai.github.io
Lets Learn AI
Stars: ✭ 33 (-66.67%)
Mutual labels:  deep-learning-algorithms
Bayeso
Simple, but essential Bayesian optimization package
Stars: ✭ 57 (-42.42%)
Mutual labels:  hyperparameter-optimization
Hyperparameter hunter
Easy hyperparameter optimization and automatic result saving across machine learning algorithms and libraries
Stars: ✭ 648 (+554.55%)
Mutual labels:  hyperparameter-optimization
360sd Net
Pytorch implementation of ICRA 2020 paper "360° Stereo Depth Estimation with Learnable Cost Volume"
Stars: ✭ 94 (-5.05%)
Mutual labels:  deep-learning-algorithms
Deep traffic
MIT DeepTraffic top 2% solution (75.01 mph) 🚗.
Stars: ✭ 47 (-52.53%)
Mutual labels:  hyperparameter-optimization
My Journey In The Data Science World
📢 Ready to learn or review your knowledge!
Stars: ✭ 1,175 (+1086.87%)
Mutual labels:  deep-learning-algorithms
Pytorch Deform Conv
PyTorch implementation of Deformable Convolution
Stars: ✭ 832 (+740.4%)
Mutual labels:  deep-learning-algorithms
Mljar Supervised
Automated Machine Learning Pipeline with Feature Engineering and Hyper-Parameters Tuning 🚀
Stars: ✭ 961 (+870.71%)
Mutual labels:  hyperparameter-optimization
Mgo
Purely functional genetic algorithms for multi-objective optimisation
Stars: ✭ 63 (-36.36%)
Mutual labels:  hyperparameter-optimization
Gradient Free Optimizers
Simple and reliable optimization with local, global, population-based and sequential techniques in numerical discrete search spaces.
Stars: ✭ 711 (+618.18%)
Mutual labels:  hyperparameter-optimization
Cmaes
Python library for CMA Evolution Strategy.
Stars: ✭ 88 (-11.11%)
Mutual labels:  hyperparameter-optimization
Test Tube
Python library to easily log experiments and parallelize hyperparameter search for neural networks
Stars: ✭ 663 (+569.7%)
Mutual labels:  hyperparameter-optimization
Tpot
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
Stars: ✭ 8,378 (+8362.63%)
Mutual labels:  hyperparameter-optimization
Hyperopt Keras Cnn Cifar 100
Auto-optimizing a neural net (and its architecture) on the CIFAR-100 dataset. Could be easily transferred to another dataset or another classification task.
Stars: ✭ 95 (-4.04%)
Mutual labels:  hyperparameter-optimization
Machine Learning Algorithms
A curated list of almost all machine learning algorithms and deep learning algorithms grouped by category.
Stars: ✭ 92 (-7.07%)
Mutual labels:  deep-learning-algorithms
Determined
Determined: Deep Learning Training Platform
Stars: ✭ 1,171 (+1082.83%)
Mutual labels:  hyperparameter-optimization

Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

Code for reproducing results published in the paper "Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates" (AAAI-17) by Ilija Ilievski, Taimoor Akhtar, Jiashi Feng, and Christine Annette Shoemaker.

arXiv -- PDF -- Supplement -- Poster

About

The paper presents an algorithm for Hyperparameter Optimization via RBF and Dynamic coordinate search (HORD). HORD searches a surrogate model of the expensive function (full training and validation of deep learning algorithm) for the most promising hyperparameter values through dynamic coordinate search and thus requires many fewer expensive function evaluations. HORD does well in low dimensions but it is exceptionally better in higher dimensions. Extensive evaluations on MNIST and CIFAR-10 for four deep learning algorithms demonstrate HORD significantly outperforms the well-established Bayesian optimization methods such as GP, SMAC, and TPE. For instance, on average, HORD is more than 6 times faster than GP-EI in obtaining the best configuration of 19 hyperparameters.

Left: Efficiency comparison of HORD with baselines. The methods are used for optimizing an MLP network with 6 hyperparameters on the MNIST dataset (6-MLP). We plot validation error curves of the compared methods against number of the function evaluations (averaged over 10 trials).

Right: Mean validation error v.s. number of function evaluations of different methods for optimizing 6 hyperparameters of MLP (6-MLP) network on MNIST (averaged over 10 trials). One dot represents validation error of an algorithm at the corresponding evaluation instance.

Left: Efficiency comparison of HORD with baselines. The methods are used for optimizing a CNN with 8 hyperparameters on the MNIST dataset (8-CNN). We plot validation error curves of the compared methods against number of the function evaluations (averaged over 5 trials).

Right: Mean validation error v.s. number of function evaluations of different methods for optimizing 8 hyperparameters of CNN (8-CNN) on MNIST (averaged over 5 trials). One dot represents validation error of an algorithm at the corresponding evaluation instance.

Left: Efficiency comparison of HORD and HORD-ISP with baselines. The methods are used for optimizing a CNN with 15 hyperparameters on the MNIST dataset (15-CNN). We plot validation error curves of the compared methods against number of the function evaluations (averaged over 5 trials). HORD and HORD-ISP show to be significantly more efficient than other methods.

Right: Mean validation error v.s. number of function evaluations of different methods for optimizing 15 hyperparameters of CNN (15-CNN) on MNIST (averaged over 5 trials). One dot represents validation error of an algorithm at the corresponding evaluation instance.

Left: Efficiency comparison of HORD and HORD-ISP with baselines for optimizing a CNN with 19 hyperparameters on the CIFAR-10 dataset (19-CNN). We plot validation error curves of the compared methods against number of the function evaluations (averaged over 5 trials). HORD and HORD-ISP show to be significantly more efficient than other methods. HORD-ISP only takes 54 function evaluations to achieve the lowest validation error that the best baseline (SMAC) achieves after 200 evaluations.

Right: Mean validation error v.s. number of function evaluations of different methods for optimizing 19 hyperparameters of CNN (19-CNN) on CIFAR-10. One dot represents validation error of an algorithm at the corresponding evaluation instance. After n0 evaluations, the searching of HORD and HORD-ISP starts to focus on the hyperparameters with smaller validation error (<35%), in stark contrast with other methods.

For more details download the paper from arxiv.

Installation

The HORD algorithm presented in the paper uses the open-source pySOT toolbox by David Eriksson, David Bindel, and Christine Shoemaker. To install pySOT go to pySOT and follow the directions there. We use version 0.1.23

The implementation of the deep learning algorithms is in torch, so you would also need to install torch from torch. We ran the deep learning algorithms on a cluster of GPU devices but you can modify the code and run them on CPU.

Reproducing

After successfully installing pySOT and torch you can run the experiments by executing:

cd 6-MLP
python pySOT_runner.py 1 200 139 exp-1

6-MLP is the folder containing the code for experiment 6-MLP (as referenced in the paper). Other experiments are 8-CNN, 15-CNN, or 19-CNN.

The four arguments to pySOT_runner.py are the following:

  • number of threads: we only tested the serial version of pySOT i.e. only one thread in all our experiments
  • function evaluation budget: we limit the number of function evaluations (full training and validation of the deep learning algorithms) to 200 in all our experiments.
  • seed number
  • experiment name

Citing the HORD algorithm

To cite the paper use the following BibTeX entry:

@inproceedings{ilievski2017efficient,
  title={Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates},
  author={Ilievski, Ilija and Akhtar, Taimoor and Feng, Jiashi and Shoemaker, Christine},
  booktitle={31st AAAI Conference on Artificial Intelligence (AAAI-17)},
  year={2017}
}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].