MetaLifelongLanguage
This is the official code for the paper Meta-Learning with Sparse Experience Replay for Lifelong Language Learning.
Getting started
- Clone the repository:
git clone [email protected]:Nithin-Holla/MetaLifelongLanguage.git. - Create a virtual environment.
- Install the required packages:
pip install -r MetaLifelongLanguage/requirements.txt.
Downloading the data
- Create a directory for storing the data:
mkdir data. - Navigate to the data directory:
cd data. - Download the five datasets for text classification from here and unzip them in this directory.
- Make a new directory for lifelong relation extraction:
mkdir LifelongFewRel. - Download the files using these commands:
wget https://raw.githubusercontent.com/hongwang600/Lifelong_Relation_Detection/master/data/relation_name.txt wget https://raw.githubusercontent.com/hongwang600/Lifelong_Relation_Detection/master/data/training_data.txt wget https://raw.githubusercontent.com/hongwang600/Lifelong_Relation_Detection/master/data/val_data.txt
- Navigate back:
cd ../... - The directory tree should look like this:
. ├── MetaLifelongLanguage ├── data │ ├── ag_news_csv │ │ ├── classes.txt │ │ ├── readme.txt │ │ ├── test.csv │ │ └── train.csv │ ├── amazon_review_full_csv │ │ ├── readme.txt │ │ ├── test.csv │ │ └── train.csv │ ├── dbpedia_csv │ │ ├── classes.txt │ │ ├── readme.txt │ │ ├── test.csv │ │ └── train.csv │ ├── yahoo_answers_csv │ │ ├── classes.txt │ │ ├── readme.txt │ │ ├── test.csv │ │ └── train.csv │ ├── yelp_review_full_csv │ │ ├── readme.txt │ │ ├── test.csv │ │ └── train.csv │ ├── LifelongFewRel │ │ ├── relation_name.txt │ │ ├── training_data.txt │ │ ├── val_data.txt
Text classification
train_text_cls.py contains the code for training and evaluation on the lifelong text classification benchmark. The usage is:
python train_text_cls.py [-h] --order ORDER [--n_epochs N_EPOCHS] [--lr LR]
[--inner_lr INNER_LR] [--meta_lr META_LR]
[--model MODEL] [--learner LEARNER]
[--mini_batch_size MINI_BATCH_SIZE]
[--updates UPDATES] [--write_prob WRITE_PROB]
[--max_length MAX_LENGTH] [--seed SEED]
[--replay_rate REPLAY_RATE]
[--replay_every REPLAY_EVERY]
optional arguments:
-h, --help show this help message and exit
--order ORDER Order of datasets
--n_epochs N_EPOCHS Number of epochs (only for MTL)
--lr LR Learning rate (only for the baselines)
--inner_lr INNER_LR Inner-loop learning rate
--meta_lr META_LR Meta learning rate
--model MODEL Name of the model
--learner LEARNER Learner method
--n_episodes N_EPISODES
Number of meta-training episodes
--mini_batch_size MINI_BATCH_SIZE
Batch size of data points within an episode
--updates UPDATES Number of inner-loop updates
--write_prob WRITE_PROB
Write probability for buffer memory
--max_length MAX_LENGTH
Maximum sequence length for the input
--seed SEED Random seed
--replay_rate REPLAY_RATE
Replay rate from memory
--replay_every REPLAY_EVERY
Number of data points between replay
Relation extraction
train_rel.py contains the code for training and evaluating on the lifelong relation extraction benchmark. The usage is:
python train_rel.py [-h] [--n_epochs N_EPOCHS] [--lr LR] [--inner_lr INNER_LR]
[--meta_lr META_LR] [--model MODEL] [--learner LEARNER]
[--mini_batch_size MINI_BATCH_SIZE] [--updates UPDATES]
[--write_prob WRITE_PROB] [--max_length MAX_LENGTH]
[--seed SEED] [--replay_rate REPLAY_RATE] [--order ORDER]
[--num_clusters NUM_CLUSTERS]
[--replay_every REPLAY_EVERY]
optional arguments:
-h, --help show this help message and exit
--n_epochs N_EPOCHS Number of epochs (only for MTL)
--lr LR Learning rate (only for the baselines)
--inner_lr INNER_LR Inner-loop learning rate
--meta_lr META_LR Meta learning rate
--model MODEL Name of the model
--learner LEARNER Learner method
--n_episodes N_EPISODES
Number of meta-training episodes
--mini_batch_size MINI_BATCH_SIZE
Batch size of data points within an episode
--updates UPDATES Number of inner-loop updates
--write_prob WRITE_PROB
Write probability for buffer memory
--max_length MAX_LENGTH
Maximum sequence length for the input
--seed SEED Random seed
--replay_rate REPLAY_RATE
Replay rate from memory
--order ORDER Number of task orders to run for
--num_clusters NUM_CLUSTERS
Number of clusters to take
--replay_every REPLAY_EVERY
Number of data points between replay
Citation
If you use this code repository, please consider citing the paper:
@article{holla2020lifelong,
title={Meta-Learning with Sparse Experience Replay for Lifelong Language Learning},
author={Holla, Nithin and Mishra, Pushkar and Yannakoudakis, Helen and Shutova, Ekaterina},
journal={arXiv preprint arXiv:2009.04891},
year={2020}
}