Semi-Supervised Learning with Deep Generative Models
Chainer implementation of Variational AutoEncoder(VAE) model M1, M2, M1+M2
この記事で実装したコードです。
Requirements
- Chainer 1.8+
- sklearn
To visualize results, you need
- matplotlib.patches
- PIL
- pandas
Download MNIST
run mnist-tools.py to download and extract MNIST.
How to label my own dataset?
You can provide label information by filename.
format:
{label_id}_{unique_filename}.{extension}
regex:
([0-9]+)_.+\.(bmp|png|jpg)
e.g. MNIST
M1
Parameters
| params | value |
|---|---|
| OS | Windows 7 |
| GPU | GeForce GTX 970M |
| ndim_z | 2 |
| encoder_apply_dropout | False |
| decoder_apply_dropout | False |
| encoder_apply_batchnorm | True |
| decoder_apply_batchnorm | True |
| encoder_apply_batchnorm_to_input | True |
| decoder_apply_batchnorm_to_input | True |
| encoder_units | [600, 600] |
| decoder_units | [600, 600] |
| gradient_clipping | 1.0 |
| learning_rate | 0.0003 |
| gradient_momentum | 0.9 |
| gradient_clipping | 1.0 |
| nonlinear | softplus |
Result
Latent space
M2
Parameters
| params | value |
|---|---|
| OS | Windows 7 |
| GPU | GeForce GTX 970M |
| ndim_z | 50 |
| encoder_xy_z_apply_dropout | False |
| encoder_x_y_apply_dropout | False |
| decoder_apply_dropout | False |
| encoder_xy_z_apply_batchnorm_to_input | True |
| encoder_x_y_apply_batchnorm_to_input | True |
| decoder_apply_batchnorm_to_input | True |
| encoder_xy_z_apply_batchnorm | True |
| encoder_x_y_apply_batchnorm | True |
| decoder_apply_batchnorm | True |
| encoder_xy_z_hidden_units | [500] |
| encoder_x_y_hidden_units | [500] |
| decoder_hidden_units | [500] |
| batchnorm_before_activation | True |
| gradient_clipping | 5.0 |
| learning_rate | 0.0003 |
| gradient_momentum | 0.9 |
| gradient_clipping | 1.0 |
| nonlinear | softplus |
Result
Classification
Training details
| data | # |
|---|---|
| labeled | 100 |
| unlabeled | 49900 |
| validation | 10000 |
| test | 10000 |
| * | # |
|---|---|
| epochs | 490 |
| minutes | 1412 |
| weight updates per epoch | 2000 |
Validation accuracy:
Test accuracy: 0.9018
Analogies
run analogy.py after training
Model was trained with...
| data | # |
|---|---|
| labeled | 100 |
| unlabeled | 49900 |
| data | # |
|---|---|
| labeled | 10000 |
| unlabeled | 40000 |
| data | # |
|---|---|
| labeled | 50000 |
| unlabeled | 0 |
M1+M2
Parameters
M1
| params | value |
|---|---|
| OS | Windows 7 |
| GPU | GeForce GTX 970M |
| ndim_z | 2 |
| encoder_apply_dropout | False |
| decoder_apply_dropout | False |
| encoder_apply_batchnorm | True |
| decoder_apply_batchnorm | True |
| encoder_apply_batchnorm_to_input | True |
| decoder_apply_batchnorm_to_input | True |
| encoder_units | [600, 600] |
| decoder_units | [600, 600] |
| gradient_clipping | 1.0 |
| learning_rate | 0.0003 |
| gradient_momentum | 0.9 |
| gradient_clipping | 1.0 |
| nonlinear | softplus |
We trained M1 for 500 epochs before starting training of M2.
| * | # |
|---|---|
| epochs | 500 |
| minutes | 860 |
| weight updates per epoch | 2000 |
M2
| params | value |
|---|---|
| OS | Windows 7 |
| GPU | GeForce GTX 970M |
| ndim_z | 50 |
| encoder_xy_z_apply_dropout | False |
| encoder_x_y_apply_dropout | False |
| decoder_apply_dropout | False |
| encoder_xy_z_apply_batchnorm_to_input | True |
| encoder_x_y_apply_batchnorm_to_input | True |
| decoder_apply_batchnorm_to_input | True |
| encoder_xy_z_apply_batchnorm | True |
| encoder_x_y_apply_batchnorm | True |
| decoder_apply_batchnorm | True |
| encoder_xy_z_hidden_units | [500] |
| encoder_x_y_hidden_units | [500] |
| decoder_hidden_units | [500] |
| batchnorm_before_activation | True |
| gradient_clipping | 5.0 |
| learning_rate | 0.0003 |
| gradient_momentum | 0.9 |
| gradient_clipping | 1.0 |
| nonlinear | softplus |
Result
Classification
Training details
| data | # |
|---|---|
| labeled | 100 |
| unlabeled | 49900 |
| validation | 10000 |
| test | 10000 |
| * | # |
|---|---|
| epochs | 600 |
| minutes | 4920 |
| weight updates per epoch | 5000 |
Validation accuracy:
Test accuracy
seed1: 0.954
seed2: 0.951