TropComplique / Trained Ternary Quantization
Licence: mit
Reducing the size of convolutional neural networks
Stars: ✭ 90
Projects that are alternatives of or similar to Trained Ternary Quantization
Keras model compression
Model Compression Based on Geoffery Hinton's Logit Regression Method in Keras applied to MNIST 16x compression over 0.95 percent accuracy.An Implementation of "Distilling the Knowledge in a Neural Network - Geoffery Hinton et. al"
Stars: ✭ 59 (-34.44%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Gtsrb
Convolutional Neural Network for German Traffic Sign Recognition Benchmark
Stars: ✭ 65 (-27.78%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Cnn graph
Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
Stars: ✭ 1,110 (+1133.33%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Accurate Binary Convolution Network
Binary Convolution Network for faster real-time processing in ASICs
Stars: ✭ 49 (-45.56%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Age Gender Estimation
Keras implementation of a CNN network for age and gender estimation
Stars: ✭ 1,195 (+1227.78%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Very Deep Convolutional Networks For Natural Language Processing In Tensorflow
implement the paper" Very Deep Convolutional Networks for Natural Language Processing"(https://arxiv.org/abs/1606.01781 ) in tensorflow
Stars: ✭ 54 (-40%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Audio classification
CNN 1D vs 2D audio classification
Stars: ✭ 65 (-27.78%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Teacher Student Training
This repository stores the files used for my summer internship's work on "teacher-student learning", an experimental method for training deep neural networks using a trained teacher model.
Stars: ✭ 34 (-62.22%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Cnn Interpretability
🏥 Visualizing Convolutional Networks for MRI-based Diagnosis of Alzheimer’s Disease
Stars: ✭ 68 (-24.44%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Equivariant Transformers
Equivariant Transformer (ET) layers are image-to-image mappings that incorporate prior knowledge on invariances with respect to continuous transformations groups (ICML 2019). Paper: https://arxiv.org/abs/1901.11399
Stars: ✭ 68 (-24.44%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Svhn Cnn
Google Street View House Number(SVHN) Dataset, and classifying them through CNN
Stars: ✭ 44 (-51.11%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Automatic Image Captioning
Generating Captions for images using Deep Learning
Stars: ✭ 84 (-6.67%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Computervision Recipes
Best Practices, code samples, and documentation for Computer Vision.
Stars: ✭ 8,214 (+9026.67%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Convisualize nb
Visualisations for Convolutional Neural Networks in Pytorch
Stars: ✭ 57 (-36.67%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Yann
This toolbox is support material for the book on CNN (http://www.convolution.network).
Stars: ✭ 41 (-54.44%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Pneumonia Detection From Chest X Ray Images With Deep Learning
Detecting Pneumonia in Chest X-ray Images using Convolutional Neural Network and Pretrained Models
Stars: ✭ 64 (-28.89%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Deep learning projects
Stars: ✭ 28 (-68.89%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Keras Faster Rcnn
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Stars: ✭ 28 (-68.89%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Sru Deeplearning Workshop
دوره 12 ساعته یادگیری عمیق با چارچوب Keras
Stars: ✭ 66 (-26.67%)
Mutual labels: jupyter-notebook, convolutional-neural-networks
Pyepr
Powerful, automated analysis and design of quantum microwave chips & devices [Energy-Participation Ratio and more]
Stars: ✭ 81 (-10%)
Mutual labels: jupyter-notebook, quantization
Trained Ternary Quantization
pytorch implementation of Trained Ternary Quantization, a way of replacing full precision weights of a neural network by ternary values. I tested it on Tiny ImageNet dataset. The dataset consists of 64x64 images and has 200 classes.
The quantization roughly proceeds as follows.
- Train a model of your choice as usual (or take a trained model).
- Copy all full precision weights that you want to quantize. Then do the initial quantization:
in the model replace them by ternary values {-1, 0, +1} using some heuristic. - Repeat until convergence:
- Make the forward pass with the quantized model.
- Compute gradients for the quantized model.
- Preprocess the gradients and apply them to the copy of full precision weights.
- Requantize the model using the changed full precision weights.
- Throw away the copy of full precision weights and use the quantized model.
Results
I believe that this results can be made better by spending more time on hyperparameter optimization.
| model | accuracy, % | top5 accuracy, % | number of parameters |
|---|---|---|---|
| DenseNet-121 | 74 | 91 | 7151176 |
| TTQ DenseNet-121 | 66 | 87 | ~7M 2-bit, 88% are zeros |
| small DenseNet | 49 | 75 | 440264 |
| TTQ small DenseNet | 40 | 67 | ~0.4M 2-bit, 38% are zeros |
| SqueezeNet | 52 | 77 | 827784 |
| TTQ SqueezeNet | 36 | 63 | ~0.8M 2-bit, 66% are zeros |
Implementation details
- I use pretrained DenseNet-121, but I train SqueezeNet and small DenseNet from scratch.
- I modify the SqueezeNet architecture by adding batch normalizations and skip connections.
- I quantize all layers except the first CONV layer, the last FC layer, and all BATCH_NORM layers.
How to reproduce results
For example, for small DenseNet:
- Download Tiny ImageNet dataset and extract it to
~/datafolder. - Run
python utils/move_tiny_imagenet_data.pyto prepare the data. - Go to
vanilla_densenet_small/. Runtrain.ipynbto train the model as usual.
Or you can skip this step and usemodel.pytorch_state(the model already trained by me). - Go to
ttq_densenet_small/. - Run
train.ipynbto do TTQ. - Run
test_and_explore.ipynbto explore the quantized model.
To use this on your data you need to edit utils/input_pipeline.py and to change
the model architecture in files like densenet.py and get_densenet.py as you like.
Requirements
- pytorch 0.2, Pilllow, torchvision
- numpy, sklearn, tqdm, matplotlib
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].