Relation-Network-PyTorch
Implementation of Relation Network. Original paper: https://arxiv.org/abs/1706.01427
Implementation of Recurrent Relational Network. Original paper: https://arxiv.org/abs/1711.08028
This repository uses PyTorch v1.3 (Python3.7).
Implementation details
This implementation tests the Relation Network model (RN) and the Recurrent Relational Network model (RRN) against the babi dataset, available at https://research.fb.com/downloads/babi/
Weights and Biases
This repository uses Weights and Biases (W&B) to monitor experiments. You can create a free account on W&B (https://www.wandb.com/) or comment out the (few) lines starting with wandb
. Without W&B, accuracy and loss plots will still be created and saved locally in the results
folder.
Prerequisites
- Run
pip install -r requirements.txt
- Create a folder
babi
- Download the babi dataset (20 tasks) already divided and preprocessed in train, validation and test at this link: https://drive.google.com/drive/folders/1HkO_w7hbxxWZV4nbGl0Y2w_AuSa4ktub?usp=sharing and put the downloaded folder inside
babi
. - Download the word dictionaries at: https://drive.google.com/drive/folders/1Ktd4kL1FJBiY_R_HoqlicjmRemkq8xzC?usp=sharing and put the downloaded folder inside
babi
. - Create a folder
results
to store experiment results (both plots and saved models)
Train and test RN
- Model implementation is inside
src/models/RN.py
- Train and test functions are inside
task/babi_task/rn/train.py
- The main script is
launch_rn_babi.py
.- Run it with
python launch_rn_babi.py experiment_name [options]
. - Options are listed and explained with
python launch_rn_babi.py --help
.
- Run it with
To reproduce results execute python launch_rn_babi.py test --learning_rate 1e-4 --batch_size 20 --epochs 50
and then check under results/test
to see the results. If you want to do the final test on the test set instead of validation set, use --test_on_test
option. The final accuracy on test set is (task: accuracy):
- 1: 0.973
- 2: 0.828
- 3: 0.786
- 4: 0.884
- 5: 0.957
- 6: 0.985
- 7: 0.962
- 8: 0.996
- 9: 0.979
- 10: 0.967
- 11: 0.958
- 12: 0.968
- 13: 0.973
- 14: 0.972
- 15: 0.97
- 16: 0.463
- 17: 0.545
- 18: 0.957
- 19: 0.483
- 20: 0.999
Observations
- Batchify babi is essential to training performance, both in terms of convergence time and in terms of final accuracy.
- In order to batchify babi it is necessary to pad supporting facts both on #words and #facts dimensions.
- Relu activation dramatically improves accuracy, but only when using
batch size > 1
. Ifbatch size == 1
relu prevents learning, while tanh achieves~74%
accuracy on the joint dataset. - You can find the version with
batch size == 1
in the branchno_batch
.
Train and test RRN
- Model implementation is inside
src/models/RRN.py
- Train and test functions are inside
task/babi_task/rrn/train.py
- The main script is
launch_rrn_babi.py
.- Run it with
python launch_rrn_babi.py experiment_name [options]
. - Options are listed and explained with
python launch_rrn_babi.py --help
.
- Run it with
To reproduce results execute python launch_rrn_babi.py test --cuda --epochs 500 --batch_size 512 --weight_decay 1e-5 --learning_rate 2e-4
and then check under results/rrn/test
to see the results. If you want to do the final test on the test set instead of validation set, use --test_on_test
option. The final accuracy on validation set is (task: accuracy):
- 1: 0.974
- 2: 0.736
- 3: 0.674
- 4: 1.0
- 5: 0.922
- 6: 0.953
- 7: 0.994
- 8: 0.975
- 9: 0.931
- 10: 0.855
- 11: 0.876
- 12: 0.939
- 13: 0.912
- 14: 0.846
- 15: 1.0
- 16: 0.454
- 17: 0.980
- 18: 0.995
- 19: 0.498
- 20: 0.999