polo5 / Zeroshotknowledgetransfer
Accompanying code for the paper "Zero-shot Knowledge Transfer via Adversarial Belief Matching"
Stars: ✭ 93
Labels
Projects that are alternatives of or similar to Zeroshotknowledgetransfer
Car Damage Detector
Detect dents and scratches in cars. Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow.
Stars: ✭ 91 (-2.15%)
Mutual labels: jupyter-notebook
One Shot Siamese
A PyTorch implementation of "Siamese Neural Networks for One-shot Image Recognition".
Stars: ✭ 91 (-2.15%)
Mutual labels: jupyter-notebook
Juliaworkshop19
Advanced Julia for undergraduate physicists
Stars: ✭ 92 (-1.08%)
Mutual labels: jupyter-notebook
Stepik Dl Nlp
Материалы мини-курса на Stepik "Нейронные сети и обработка текста"
Stars: ✭ 93 (+0%)
Mutual labels: jupyter-notebook
Cognoma
Putting machine learning in the hands of cancer biologists
Stars: ✭ 92 (-1.08%)
Mutual labels: jupyter-notebook
Resnet cnn mri adni
Code for Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification paper
Stars: ✭ 92 (-1.08%)
Mutual labels: jupyter-notebook
Ds With Pysimplegui
Data science and Machine Learning GUI programs/ desktop apps with PySimpleGUI package
Stars: ✭ 93 (+0%)
Mutual labels: jupyter-notebook
Sprint gan
Privacy-preserving generative deep neural networks support clinical data sharing
Stars: ✭ 92 (-1.08%)
Mutual labels: jupyter-notebook
Programming Collective Intelligence
《集体智慧编程》Python代码(基于Python3.6)和数据集
Stars: ✭ 92 (-1.08%)
Mutual labels: jupyter-notebook
Zero Shot Knowledge Transfer
This is accompanying code for the NeurIPS 2019 spotlight paper "Zero-shot Knowledge Transfer via Adversarial Belief Matching" see arxiv
What is this work about? (TLDR)
Our task is to compress a large neural network (teacher) into a smaller one (student), but we assume that the data used to train the teacher is not available anymore. We thus generate pseudo points adversarially (yellow markers above) and use those to match the student (right) to the teacher (left).
Environment
- Python 3.6
- pytorch 1.0.0 (both cpu and gpu version tested)
- tensorboard 1.7.0 (for logging, + needs tensorflow)
- scipy <= 1.2 (otherwise scipy.misc.toimage complains in logger)
Run zero shot knowledge transfer
- Pretrain a teacher for the dataset/architecture you want (or download some of mine here)
- Make sure you have the same folder structure as in the link above, i.e. Pretrained/{dataset}/{architecture}/last.pth.tar
- Edit the paths in e.g. scripts/ZeroShot/main0.sh and run it
Make transition curves
- Pretrain a zero-shot student or a student distilled with KD+AT (or download some of mine here)
- Edit the paths in e.g. scripts/TransitionCurves/transition_curves0.sh and run it
- This saves .pickle file with all the transition curves
- You can import that file in notebooks/transition_curves_and_error.ipynb to plot the transition curves and calculate MTE scores
Cite
If you use this work please consider citing:
@incollection{Micaelli2019ZeroShotKT,
title = {Zero-shot Knowledge Transfer via Adversarial Belief Matching},
author = {Micaelli, Paul and Storkey, Amos J},
booktitle = {Advances in Neural Information Processing Systems 32},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {9551--9561},
year = {2019},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/9151-zero-shot-knowledge-transfer-via-adversarial-belief-matching.pdf}
}
Baselines
A few people have asked for the KD+AT few shot baselines so I've made them available in this repo
Notes
- Version 1 and 2 on arxiv have lower accuracy scores because I hadn't trained batchnorm layers properly.
- Attention only gives you an average of 2% boost across all architectures tested so you can delete that code if you want to save on memory/compute time.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].