All Projects → xiaomi-automl → Fairdarts

xiaomi-automl / Fairdarts

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Programming Languages

python
139335 projects - #7 most used programming language

Labels

Projects that are alternatives of or similar to Fairdarts

Neural-Architecture-Search
This repo is about NAS
Stars: ✭ 26 (-82.07%)
Mutual labels:  nas, automl
Nas Bench 201
NAS-Bench-201 API and Instruction
Stars: ✭ 537 (+270.34%)
Mutual labels:  nas, automl
BossNAS
(ICCV 2021) BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
Stars: ✭ 125 (-13.79%)
Mutual labels:  nas, automl
Nni
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
Stars: ✭ 10,698 (+7277.93%)
Mutual labels:  nas, automl
Awesome Autodl
A curated list of automated deep learning (including neural architecture search and hyper-parameter optimization) resources.
Stars: ✭ 1,819 (+1154.48%)
Mutual labels:  nas, automl
Hypernets
A General Automated Machine Learning framework to simplify the development of End-to-end AutoML toolkits in specific domains.
Stars: ✭ 221 (+52.41%)
Mutual labels:  nas, automl
Autodeeplab
AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch
Stars: ✭ 269 (+85.52%)
Mutual labels:  nas, automl
Awesome Nas Papers
Awesome Neural Architecture Search Papers
Stars: ✭ 213 (+46.9%)
Mutual labels:  nas, automl
Shape Adaptor
The implementation of "Shape Adaptor: A Learnable Resizing Module" [ECCV 2020].
Stars: ✭ 59 (-59.31%)
Mutual labels:  nas, automl
Autodl
Automated Deep Learning without ANY human intervention. 1'st Solution for AutoDL [email protected]
Stars: ✭ 854 (+488.97%)
Mutual labels:  nas, automl
nas-encodings
Encodings for neural architecture search
Stars: ✭ 29 (-80%)
Mutual labels:  nas, automl
Autodl Projects
Automated deep learning algorithms implemented in PyTorch.
Stars: ✭ 1,187 (+718.62%)
Mutual labels:  nas, automl
Awesome Automl And Lightweight Models
A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering.
Stars: ✭ 691 (+376.55%)
Mutual labels:  nas, automl
Once For All
[ICLR 2020] Once for All: Train One Network and Specialize it for Efficient Deployment
Stars: ✭ 1,127 (+677.24%)
Mutual labels:  nas, automl
Nas Benchmark
"NAS evaluation is frustratingly hard", ICLR2020
Stars: ✭ 126 (-13.1%)
Mutual labels:  nas, automl
Synology
Cheatsheet and bash scripts sripts for Synology Nas Stations cheet cheat sheet nas networkdisk
Stars: ✭ 125 (-13.79%)
Mutual labels:  nas
Pnasnet.tf
TensorFlow implementation of PNASNet-5 on ImageNet
Stars: ✭ 102 (-29.66%)
Mutual labels:  automl
Advisor
Open-source implementation of Google Vizier for hyper parameters tuning
Stars: ✭ 1,359 (+837.24%)
Mutual labels:  automl
Sgas
SGAS: Sequential Greedy Architecture Search (CVPR'2020) https://www.deepgcns.org/auto/sgas
Stars: ✭ 137 (-5.52%)
Mutual labels:  automl
Auto ml
[UNMAINTAINED] Automated machine learning for analytics & production
Stars: ✭ 1,559 (+975.17%)
Mutual labels:  automl

[ECCV'20] Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

*: This is the official implementation of the FairDARTS paper.

Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, there are two fundamental weaknesses remain untackled. First, we observe that the well-known aggregation of skip connections during optimization is caused by an unfair advantage in an exclusive competition. Second, there is a non-negligible incongruence when discretizing continuous architectural weights to a one-hot representation. Because of these two reasons, DARTS delivers a biased solution that might not even be suboptimal. In this paper, we present a novel approach to curing both frailties. Specifically, as unfair advantages in a pure exclusive competition easily induce a monopoly, we relax the choice of operations to be collaborative, where we let each operation have an equal opportunity to develop its strength. We thus call our method Fair DARTS. Moreover, we propose a zero-one loss to directly reduce the discretization gap. Experiments are performed on two mainstream search spaces, in which we achieve new state-of-the-art networks on ImageNet.

User Guide

Prerequisites

Python 3

pip install -r requirements.txt

The fairdarts folder includes our search, train and evaluation code. The darts folder consists of random and noise experiments on the original DARTS.

Run Search

python train_search.py --aux_loss_weight 10 --learning_rate 0.005 --batch_size 128 --parse_method threshold_sparse --save 'EXP-lr_0005_alw_10'

Default batch-size is 128

Single Model Training

python train.py --auxiliary --cutout --arch FairDARTS_a --parse_method threshold --batch_size 128 --epoch 600

Single Model Evaluation

python evaluate_model.py  --arch FairDARTS_b --model_path ../best_model/FairDARTS-b.tar --parse_method threshold

Searched Architectures by FairDARTS

Note that we select architecture by barring with threshold σ, and |edge| <= 2 per node.

FairDARTS_a:

DCO_SPARSE_normal DCO_SPARSE_reduce

FairDARTS_b

DCO_SPARSE_3_normal DCO_SPARSE_3_reduce

FairDARTS_c

DCO_SPARSE_1_normal DCO_SPARSE_1_reduce

FairDARTS_d

DCO_SPARSE_2_normal DCO_SPARSE_2_reduce

FairDARTS_e

DCO_SPARSE_4_normal DCO_SPARSE_4_reduce

FairDARTS_f

DCO_SPARSE_5_normal DCO_SPARSE_5_reduce

FairDARTS_g

DCO_SPARSE_6_normal DCO_SPARSE_6_reduce

The isolated nodes (in gray) are ignored after parsing the genotypes.

Evaluation Results on CIFAR-10

Performance Stability

We run FairDARTS 7 times, all searched architectures have close performance.

Model Flops Params Performance
FairDARTS_a 373M 2.83M 97.46
FairDARTS_b 536M 3.88M 97.49
FairDARTS_c 400M 2.59M 97.50
FairDARTS_d 532M 3.84M 97.51
FairDARTS_e 414M 3.12M 97.47
FairDARTS_f 497M 3.62M 97.35
FairDARTS_g 453M 3.38M 97.46
mean,var ~457.85M ~3.32M 97.46±0.049

Note: We remove batch normalization for FLOPs' calculation in thop package. This is to follow status quo treamtment.

Comparison with Other State-of-the-art Results (CIFAR-10)

Model FLOPs Params Batch size lr DP Optimizer Performance
FairDARTS-a 373M 2.83 96 0.025 0.2 SGD+CosineAnnealingLR 97.46
FairDARTS-b 536M 3.88 96 0.025 0.2 SGD+CosineAnnealingLR 97.49
DARTS_V2 522M 3.36 96 0.025 0.2 SGD+CosineAnnealingLR 96.94*
PC-DARTS 558M 3.63 96 0.025 0.2 SGD+CosineAnnealingLR 97.31*
PDARTS 532M 3.43 96 0.025 0.2 SGD+CosineAnnealingLR 97.53*

*: Results obtained by training their published code.

Citation

@inproceedings{chu2019fairdarts,
    title={{Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search}},
    author={Chu, Xiangxiang and Zhou, Tianbao and Zhang, Bo and Li, Jixiang},
    booktitle={16th Europoean Conference On Computer Vision},
    url={https://arxiv.org/abs/1911.12126.pdf},
    year={2020}
}

Acknowledgement

This code is based on the implementation of DARTS.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].