Efficient Softmax Approximation
Implementations of Blackout and Adaptive Softmax for efficiently calculating word distribution for language modeling of very large vocabularies.
LSTM language models are derived from rnnlm_chainer.
Available output layers are as follows
- Linear + softmax with cross entropy loss. A usual output layer.
--share-embedding: A variant using the word embedding matrix shared with the input layer for the output layer.--adaptive-softmax: Adaptive softmax--blackout: BlackOut (BlackOut is not faster on GPU.)
Adaptive Softmax
- Efficient softmax approximation for GPUs
- Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, ICML 2017
- paper
- authors' Lua code
BlackOut
- BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
- Shihao Ji, S. V. N. Vishwanathan, Nadathur Satish, Michael J. Anderson, Pradeep Dubey, ICLR 2016
- paper
- authors' C++ code
How to Run
python -u train.py -g 0
Datasets
- PennTreeBank
- Wikitext-2
- Wikitext-103
For wikitext, run prepare_wikitext.sh for downloading the datasets.