rahulvigneswaran / Lottery Ticket Hypothesis In Pytorch
This repository contains a Pytorch implementation of the paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle and Michael Carbin that can be easily adapted to any model/dataset.
Stars: ✭ 162
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Lottery Ticket Hypothesis In Pytorch
Distiller
Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research. https://intellabs.github.io/distiller
Stars: ✭ 3,760 (+2220.99%)
Mutual labels: pruning
Awesome Pruning
A curated list of neural network pruning resources.
Stars: ✭ 1,017 (+527.78%)
Mutual labels: pruning
Micronet
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
Stars: ✭ 1,232 (+660.49%)
Mutual labels: pruning
Awesome Emdl
Embedded and mobile deep learning research resources
Stars: ✭ 554 (+241.98%)
Mutual labels: pruning
Ridurre Network Filter Pruning Keras
Keras model convolutional filter pruning package
Stars: ✭ 35 (-78.4%)
Mutual labels: pruning
Filter Pruning Geometric Median
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration (CVPR 2019 Oral)
Stars: ✭ 338 (+108.64%)
Mutual labels: pruning
Awesome Edge Machine Learning
A curated list of awesome edge machine learning resources, including research papers, inference engines, challenges, books, meetups and others.
Stars: ✭ 139 (-14.2%)
Mutual labels: pruning
Model Optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Stars: ✭ 992 (+512.35%)
Mutual labels: pruning
Adventures In Tensorflow Lite
This repository contains notebooks that show the usage of TensorFlow Lite for quantizing deep neural networks.
Stars: ✭ 79 (-51.23%)
Mutual labels: pruning
Paddleslim
PaddleSlim is an open-source library for deep model compression and architecture search.
Stars: ✭ 677 (+317.9%)
Mutual labels: pruning
Tf Keras Surgeon
Pruning and other network surgery for trained TF.Keras models.
Stars: ✭ 25 (-84.57%)
Mutual labels: pruning
Grasp
Code for "Picking Winning Tickets Before Training by Preserving Gradient Flow" https://openreview.net/pdf?id=SkgsACVKPH
Stars: ✭ 58 (-64.2%)
Mutual labels: pruning
Aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
Stars: ✭ 453 (+179.63%)
Mutual labels: pruning
Filter Grafting
Filter Grafting for Deep Neural Networks(CVPR 2020)
Stars: ✭ 110 (-32.1%)
Mutual labels: pruning
Keras Surgeon
Pruning and other network surgery for trained Keras models.
Stars: ✭ 339 (+109.26%)
Mutual labels: pruning
Delve
PyTorch and Keras model training and layer saturation monitor
Stars: ✭ 49 (-69.75%)
Mutual labels: pruning
Cen
[NeurIPS 2020] Code release for paper "Deep Multimodal Fusion by Channel Exchanging" (In PyTorch)
Stars: ✭ 112 (-30.86%)
Mutual labels: pruning
Model Compression And Acceleration Progress
Repository to track the progress in model compression and acceleration
Stars: ✭ 63 (-61.11%)
Mutual labels: pruning
Lottery Ticket Hypothesis in Pytorch
This repository contains a Pytorch implementation of the paper The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks by Jonathan Frankle and Michael Carbin that can be easily adapted to any model/dataset.
Requirements
pip3 install -r requirements.txt
How to run the code ?
Using datasets/architectures included with this repository :
python3 main.py --prune_type=lt --arch_type=fc1 --dataset=mnist --prune_percent=10 --prune_iterations=35
-
--prune_type
: Type of pruning- Options :
lt
- Lottery Ticket Hypothesis,reinit
- Random reinitialization - Default :
lt
- Options :
-
--arch_type
: Type of architecture- Options :
fc1
- Simple fully connected network,lenet5
- LeNet5,AlexNet
- AlexNet,resnet18
- Resnet18,vgg16
- VGG16 - Default :
fc1
- Options :
-
--dataset
: Choice of dataset- Options :
mnist
,fashionmnist
,cifar10
,cifar100
- Default :
mnist
- Options :
-
--prune_percent
: Percentage of weight to be pruned after each cycle.- Default :
10
- Default :
-
--prune_iterations
: Number of cycle of pruning that should be done.- Default :
35
- Default :
-
--lr
: Learning rate- Default :
1.2e-3
- Default :
-
--batch_size
: Batch size- Default :
60
- Default :
-
--end_iter
: Number of Epochs- Default :
100
- Default :
-
--print_freq
: Frequency for printing accuracy and loss- Default :
1
- Default :
-
--valid_freq
: Frequency for Validation- Default :
1
- Default :
-
--gpu
: Decide Which GPU the program should use- Default :
0
- Default :
Using datasets/architectures that are not included with this repository :
- Adding a new architecture :
- For example, if you want to add an architecture named
new_model
withmnist
dataset compatibility.- Go to
/archs/mnist/
directory and create a filenew_model.py
. - Now paste your Pytorch compatible model inside
new_model.py
. -
IMPORTANT : Make sure the input size, number of classes, number of channels, batch size in your
new_model.py
matches with the corresponding dataset that you are adding (in this case, it ismnist
). - Now open
main.py
and go toline 36
and look for the comment# Data Loader
. Now find your corresponding dataset (in this case,mnist
) and addnew_model
at the end of the linefrom archs.mnist import AlexNet, LeNet5, fc1, vgg, resnet
. - Now go to
line 82
and add the following to it :
Here,elif args.arch_type == "new_model": model = new_model.new_model_name().to(device)
new_model_name()
is the name of the model that you have given insidenew_model.py
.
- Go to
- For example, if you want to add an architecture named
- Adding a new dataset :
- For example, if you want to add a dataset named
new_dataset
withfc1
architecture compatibility.- Go to
/archs
and create a directory namednew_dataset
. - Now go to /archs/new_dataset/
and add a file named
fc1.py` or copy paste it from existing dataset folder. -
IMPORTANT : Make sure the input size, number of classes, number of channels, batch size in your
new_model.py
matches with the corresponding dataset that you are adding (in this case, it isnew_dataset
). - Now open
main.py
and gotoline 58
and add the following to it :
Note that as of now, you can only add dataset that are natively available in Pytorch.elif args.dataset == "cifar100": traindataset = datasets.new_dataset('../data', train=True, download=True, transform=transform) testdataset = datasets.new_dataset('../data', train=False, transform=transform)from archs.new_dataset import fc1
- Go to
- For example, if you want to add a dataset named
prune_type
?
How to combine the plots of various - Go to
combine_plots.py
and add/remove the datasets/archs who's combined plot you want to generate (Assuming that you have already executed themain.py
code for those dataset/archs and produced the weights). - Run
python3 combine_plots.py
. - Go to
/plots/lt/combined_plots/
to see the graphs.
Kindly raise an issue if you have any problem with the instructions.
Datasets and Architectures that were already tested
fc1 | LeNet5 | AlexNet | VGG16 | Resnet18 | |
---|---|---|---|---|---|
MNIST | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
CIFAR10 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
FashionMNIST | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
CIFAR100 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Repository Structure
Lottery-Ticket-Hypothesis-in-Pytorch
├── archs
│ ├── cifar10
│ │ ├── AlexNet.py
│ │ ├── densenet.py
│ │ ├── fc1.py
│ │ ├── LeNet5.py
│ │ ├── resnet.py
│ │ └── vgg.py
│ ├── cifar100
│ │ ├── AlexNet.py
│ │ ├── fc1.py
│ │ ├── LeNet5.py
│ │ ├── resnet.py
│ │ └── vgg.py
│ └── mnist
│ ├── AlexNet.py
│ ├── fc1.py
│ ├── LeNet5.py
│ ├── resnet.py
│ └── vgg.py
├── combine_plots.py
├── dumps
├── main.py
├── plots
├── README.md
├── requirements.txt
├── saves
└── utils.py
Interesting papers that are related to Lottery Ticket Hypothesis which I enjoyed
Acknowledgement
Parts of code were borrowed from ktkth5.
Issue / Want to Contribute ? :
Open a new issue or do a pull request incase you are facing any difficulty with the code base or if you want to contribute to it.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].