All Projects → dut-media-lab → Boml

dut-media-lab / Boml

Licence: mit
Bilevel Optimization Library in Python for Multi-Task and Meta Learning

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Boml

Auto Sklearn
Automated Machine Learning with scikit-learn
Stars: ✭ 5,916 (+4830%)
Mutual labels:  meta-learning, hyperparameter-optimization
Meta-SAC
Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient - 7th ICML AutoML workshop 2020
Stars: ✭ 19 (-84.17%)
Mutual labels:  hyperparameter-optimization, meta-learning
Hyperactive
A hyperparameter optimization and data collection toolbox for convenient and fast prototyping of machine-learning models.
Stars: ✭ 182 (+51.67%)
Mutual labels:  meta-learning, hyperparameter-optimization
Awesome Automl And Lightweight Models
A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering.
Stars: ✭ 691 (+475.83%)
Mutual labels:  meta-learning, hyperparameter-optimization
Maxl
The implementation of "Self-Supervised Generalisation with Meta Auxiliary Learning" [NeurIPS 2019].
Stars: ✭ 101 (-15.83%)
Mutual labels:  meta-learning
Pytorch Meta
A collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch
Stars: ✭ 1,239 (+932.5%)
Mutual labels:  meta-learning
Determined
Determined: Deep Learning Training Platform
Stars: ✭ 1,171 (+875.83%)
Mutual labels:  hyperparameter-optimization
Memory Efficient Maml
Memory efficient MAML using gradient checkpointing
Stars: ✭ 60 (-50%)
Mutual labels:  meta-learning
Deephyper
DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks
Stars: ✭ 117 (-2.5%)
Mutual labels:  hyperparameter-optimization
Deep architect
A general, modular, and programmable architecture search framework
Stars: ✭ 110 (-8.33%)
Mutual labels:  hyperparameter-optimization
Hord
Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
Stars: ✭ 99 (-17.5%)
Mutual labels:  hyperparameter-optimization
Xcessiv
A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.
Stars: ✭ 1,255 (+945.83%)
Mutual labels:  hyperparameter-optimization
Talos
Hyperparameter Optimization for TensorFlow, Keras and PyTorch
Stars: ✭ 1,382 (+1051.67%)
Mutual labels:  hyperparameter-optimization
Learn2learn
A PyTorch Library for Meta-learning Research
Stars: ✭ 1,193 (+894.17%)
Mutual labels:  meta-learning
Chocolate
A fully decentralized hyperparameter optimization framework
Stars: ✭ 112 (-6.67%)
Mutual labels:  hyperparameter-optimization
Mgo
Purely functional genetic algorithms for multi-objective optimisation
Stars: ✭ 63 (-47.5%)
Mutual labels:  hyperparameter-optimization
Gnn Meta Attack
Implementation of the paper "Adversarial Attacks on Graph Neural Networks via Meta Learning".
Stars: ✭ 99 (-17.5%)
Mutual labels:  meta-learning
Meta Blocks
A modular toolbox for meta-learning research with a focus on speed and reproducibility.
Stars: ✭ 110 (-8.33%)
Mutual labels:  meta-learning
R2d2
[ICLR'19] Meta-learning with differentiable closed-form solvers
Stars: ✭ 96 (-20%)
Mutual labels:  meta-learning
Hyperopt Keras Cnn Cifar 100
Auto-optimizing a neural net (and its architecture) on the CIFAR-100 dataset. Could be easily transferred to another dataset or another classification task.
Stars: ✭ 95 (-20.83%)
Mutual labels:  hyperparameter-optimization

BOML - A Bilevel Optimization Library in Python for Meta Learning

PyPI version Build Status codecov Documentation Status Language Python version license Code style: black

BOML is a modularized optimization library that unifies several ML algorithms into a common bilevel optimization framework. It provides interfaces to implement popular bilevel optimization algorithms, so that you could quickly build your own meta learning neural network and test its performance.

ReadMe.md contains brief introduction to implement meta-initialization-based and meta-feature-based methods in few-shot classification field. Except for algorithms which have been proposed, various combinations of lower level and upper level strategies are available.

Meta Learning

Meta learning works fairly well when facing incoming new tasks by learning an initialization with favorable generalization capability. And it also has good performance even provided with a small amount of training data available, which gives birth to various solutions for different application such as few-shot learning problem.

We present a general bilevel optimization paradigm to unify different types of meta learning approaches, and the mathematical form could be summarized as below:

Bilevel Optimization Model

Generic Optimization Routine

Here we illustrate the generic optimization process and hierarchically built strategies in the figure, which could be quikcly implemented in the following example.

Optimization Routine

Documentation

For more detailed information of basic function and construction process, please refer to our Documentation orProject Page. Scripts in the directory named test_script are useful for constructing general training process.

Here we give recommended settings for specific hyper paremeters to quickly test performance of popular algorithms.

Running examples

Start from loading data

import boml
from boml import utils
from test_script.script_helper import *

dataset = boml.load_data.meta_omniglot(
    std_num_classes=args.classes,
    examples_train=args.examples_train,
    examples_test=args.examples_test,
)
# create instance of BOMLExperiment for ong single task
ex = boml.BOMLExperiment(dataset)

Build network structure and define parameters for meta-learner and base-learner

boml_ho = boml.BOMLOptimizer(
    method="MetaInit", inner_method="Simple", outer_method="Simple"
)
meta_learner = boml_ho.meta_learner(_input=ex.x, dataset=dataset, meta_model="V1")
ex.model = boml_ho.base_learner(_input=ex.x, meta_learner=meta_learner)

Define LL objectives and LL calculation process

loss_inner = utils.cross_entropy(pred=ex.model.out, label=ex.y)
accuracy = utils.classification_acc(pred=ex.model.out, label=ex.y)
inner_grad = boml_ho.ll_problem(
    inner_objective=loss_inner,
    learning_rate=args.lr,
    T=args.T,
    experiment=ex,
    var_list=ex.model.var_list,
)

Define UL objectives and UL calculation process

loss_outer = utils.cross_entropy(pred=ex.model.re_forward(ex.x_).out, label=ex.y_) # loss function
boml_ho.ul_problem(
    outer_objective=loss_outer,
    meta_learning_rate=args.meta_lr,
    inner_grad=inner_grad,
    meta_param=tf.get_collection(boml.extension.GraphKeys.METAPARAMETERS),
)

Aggregate all the defined operations

# Only need to be called once after all the tasks are ready
boml_ho.aggregate_all()

Meta training iteration

with tf.Session() as sess:
    tf.global_variables_initializer().run(session=sess)
    for itr in range(args.meta_train_iterations):
        # Generate the feed_dict for calling run() everytime
        train_batch = BatchQueueMock(
            dataset.train, 1, args.meta_batch_size, utils.get_rand_state(1)
        )
        tr_fd, v_fd = utils.feed_dict(train_batch.get_single_batch(), ex)
        # Meta training step
        boml_ho.run(tr_fd, v_fd)
        if itr % 100 == 0:
            print(sess.run(loss_inner, utils.merge_dicts(tr_fd, v_fd)))

Related Methods

License

MIT License

Copyright (c) 2020 Yaohua Liu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].