GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → p208p2002 → Transformer-QG-on-SQuAD

p208p2002 / Transformer-QG-on-SQuAD

Licence: other
Implement Question Generator with SOTA pre-trained Language Models (RoBERTa, BERT, GPT, BART, T5, etc.)

Programming Languages

python
139335 projects - #7 most used programming language
Dockerfile
14818 projects

Labels

flasktext-generationapi-serverbartalbertsquadbertquestion-generationrobertagpt2pytorch-lightningt5-model

Projects that are alternatives of or similar to Transformer-QG-on-SQuAD

Text-Summarization
Abstractive and Extractive Text summarization using Transformers.
Stars: ✭ 38 (+35.71%)
Mutual labels:  bart, bert, roberta, gpt2
CLUE pytorch
CLUE baseline pytorch CLUE的pytorch版本基线
Stars: ✭ 72 (+157.14%)
Mutual labels:  albert, bert, roberta
Albert zh
A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS, 海量中文预训练ALBERT模型
Stars: ✭ 3,500 (+12400%)
Mutual labels:  albert, bert, roberta
Tianchi2020ChineseMedicineQuestionGeneration
2020 阿里云天池大数据竞赛-中医药文献问题生成挑战赛
Stars: ✭ 20 (-28.57%)
Mutual labels:  bert, question-generation, roberta
Medi-CoQA
Conversational Question Answering on Clinical Text
Stars: ✭ 22 (-21.43%)
Mutual labels:  albert, squad, bert
Clue
中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
Stars: ✭ 2,425 (+8560.71%)
Mutual labels:  albert, bert, roberta
Roberta zh
RoBERTa中文预训练模型: RoBERTa for Chinese
Stars: ✭ 1,953 (+6875%)
Mutual labels:  bert, roberta, gpt2
Bertviz
Tool for visualizing attention in the Transformer model (BERT, GPT-2, Albert, XLNet, RoBERTa, CTRL, etc.)
Stars: ✭ 3,443 (+12196.43%)
Mutual labels:  bert, roberta, gpt2
keras-bert-ner
Keras solution of Chinese NER task using BiLSTM-CRF/BiGRU-CRF/IDCNN-CRF model with Pretrained Language Model: supporting BERT/RoBERTa/ALBERT
Stars: ✭ 7 (-75%)
Mutual labels:  albert, bert
bert in a flask
A dockerized flask API, serving ALBERT and BERT predictions using TensorFlow 2.0.
Stars: ✭ 32 (+14.29%)
Mutual labels:  albert, bert
tfbert
基于tensorflow1.x的预训练模型调用，支持单机多卡、梯度累积，XLA加速，混合精度。可灵活训练、验证、预测。
Stars: ✭ 54 (+92.86%)
Mutual labels:  albert, bert
Chineseglue
Language Understanding Evaluation benchmark for Chinese: datasets, baselines, pre-trained models,corpus and leaderboard
Stars: ✭ 1,548 (+5428.57%)
Mutual labels:  albert, bert
Spark Nlp
State of the Art Natural Language Processing
Stars: ✭ 2,518 (+8892.86%)
Mutual labels:  albert, bert
MobileQA
离线端阅读理解应用 QA for mobile, Android & iPhone
Stars: ✭ 49 (+75%)
Mutual labels:  albert, bert
ALBERT-Pytorch
Pytorch Implementation of ALBERT(A Lite BERT for Self-supervised Learning of Language Representations)
Stars: ✭ 214 (+664.29%)
Mutual labels:  albert, bert
Texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Stars: ✭ 2,236 (+7885.71%)
Mutual labels:  text-generation, bert
text-generation-transformer
text generation based on transformer
Stars: ✭ 36 (+28.57%)
Mutual labels:  text-generation, bert
KorQuAD-Question-Generation
question generation model with KorQuAD dataset
Stars: ✭ 27 (-3.57%)
Mutual labels:  question-generation, gpt2
les-military-mrc-rank7
莱斯杯：全国第二届“军事智能机器阅读”挑战赛 - Rank7 解决方案
Stars: ✭ 37 (+32.14%)
Mutual labels:  bert, roberta
classifier multi label seq2seq attention
multi-label，classifier，text classification，多标签文本分类，文本分类，BERT，ALBERT，multi-label-classification，seq2seq，attention，beam search
Stars: ✭ 26 (-7.14%)
Mutual labels:  albert, bert
View All Similar Projects ➔

Transformer QG on SQuAD Auto-Build

Implement Question Generator with SOTA pre-trained Language Models (RoBERTa, BERT, GPT, BART, T5, etc.)

The inputs of the model refers to

we integrate C and A into a new C' in the following form.
C' = [c1, c2, ..., [HL], a1, ..., a|A|, [HL], ..., c|C|]

Proposed by Ying-Hong Chan & Yao-Chung Fan. (2019). A Re-current BERT-based Model for Question Generation.

Overview

Features

  • Fully pipline from fine-tune to evaluation
  • Support most of state of the art models
  • Fast deploy as a API server

Data setting

We report two dataset setting as Follow

SQuAD

  • train: 87599
  • validation: 10570

SQuAD: 100,000+ Questions for Machine Comprehension of Text

SQuAD NQG

  • train: 75722
  • dev: 10570
  • test: 11877

Learning to Ask: Neural Question Generation for Reading Comprehension

Available models

  • BART
  • GPT2
  • T5
  • BERT
  • RoBERTa

Expriments

We report score with NQG Scorer which is using in SQuAD NQG.

If not special explanation, the size of the model defaults to "base".

SQuAD

Model Bleu 1 Bleu 2 Bleu 3 Bleu 4 METEOR ROUGE-L
BERT-HLSQG (reimplement, FP16) 51.11 34.73 25.66 19.56 22.77 47.91
BART-HLSQG 54.67 39.26 30.34 24.15 25.43 52.64
GPT2-HLSQG 49.31 33.95 25.41 19.69 22.29 48.82
T5-HLSQG 54.29 39.22 30.43 24.26 25.56 53.11

SQuAD NQG

Model Bleu 1 Bleu 2 Bleu 3 Bleu 4 METEOR ROUGE-L
BERT-HLSQG (Chan et al.) 49.73 34.60 26.13 20.33 23.88 48.23
BERT-HLSQG (reimplement, FP16) 50.48 33.82 24.52 18.36 22.11 47.01
BART-HLSQG 54.12 38.19 28.84 22.35 24.55 51.03
GPT2-HLSQG 49.82 33.69 24.71 18.63 21.90 47.60
T5-HLSQG 53.13 37.60 28.62 22.38 24.48 51.20

Using with Transformers

bart-squad-qg-hl

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
  
tokenizer = AutoTokenizer.from_pretrained("p208p2002/bart-squad-qg-hl")

model = AutoModelForSeq2SeqLM.from_pretrained("p208p2002/bart-squad-qg-hl")

t5-squad-qg-hl

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
  
tokenizer = AutoTokenizer.from_pretrained("p208p2002/t5-squad-qg-hl")

model = AutoModelForSeq2SeqLM.from_pretrained("p208p2002/t5-squad-qg-hl")

and more on HF Model Hub!

Run as API server

Using docker (recommend)

docker run -it -p 5000:5000 p208p2002/transformer-qg-on-squad:lastest train_seq2seq_lm.py --server --base_model p208p2002/bart-squad-qg-hl

From your own checkpoint

python train_xxx_lm.py --server --base_model YOUR_BASE_MODEL --from_checkpoint FROM_CHECKPOINT

Request example

curl --location --request POST 'http://127.0.0.1:5000/' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'context=Harry Potter is a series of seven fantasy novels written by [HL] J. K. Rowling. [HL]'
{"predict": "Who wrote the books?"}

Training from scratch

Environment setup

The hole development is based on Ubuntu system

  1. If you don't have pytorch please install or update first (torch>=1.6,<1.8)

https://pytorch.org/get-started/locally/

  1. Install packages pip install -r requirements.txt

  2. Setup scorer python setup_scorer.py

  3. Download dataset python init_dataset.py

Seq2Seq LM

usage: train_seq2seq_lm.py [-h]
                           [--base_model {facebook/bart-base,facebook/bart-large,t5-small,t5-base,t5-large,p208p2002/bart-squad-qg-hl,p208p2002/bart-squad-nqg-hl,p208p2002/t5-squad-qg-hl,p208p2002/t5-squad-nqg-hl}]
                           [-d {squad,squad-nqg}] [--epoch EPOCH] [--lr LR]
                           [--dev DEV] [--server] [--run_test]
                           [-fc FROM_CHECKPOINT]

optional arguments:
  -h, --help            show this help message and exit
  --base_model {facebook/bart-base,facebook/bart-large,t5-small,t5-base,t5-large,p208p2002/bart-squad-qg-hl,p208p2002/bart-squad-nqg-hl,p208p2002/t5-squad-qg-hl,p208p2002/t5-squad-nqg-hl}
  -d {squad,squad-nqg}, --dataset {squad,squad-nqg}
  --epoch EPOCH
  --lr LR
  --dev DEV
  --server
  --run_test
  -fc FROM_CHECKPOINT, --from_checkpoint FROM_CHECKPOINT

Causal LM

usage: train_causal_lm.py [-h]
                          [--base_model {gpt2,gpt2-large,p208p2002/gpt2-squad-qg-hl,p208p2002/gpt2-squad-nqg-hl}]
                          [-d {squad,squad-nqg}] [--epoch EPOCH] [--lr LR]
                          [--dev DEV] [--server] [--run_test]
                          [-fc FROM_CHECKPOINT]

optional arguments:
  -h, --help            show this help message and exit
  --base_model {gpt2,gpt2-large,p208p2002/gpt2-squad-qg-hl,p208p2002/gpt2-squad-nqg-hl}
  -d {squad,squad-nqg}, --dataset {squad,squad-nqg}
  --epoch EPOCH
  --lr LR
  --dev DEV
  --server
  --run_test
  -fc FROM_CHECKPOINT, --from_checkpoint FROM_CHECKPOINT

Masked LM

usage: train_masked_lm.py [-h] [--base_model {bert-base-uncased,bert-large-uncased,roberta-base,roberta-large,albert-base-v1,albert-large-v1,albert-base-v2,albert-large-v2}]
                          [--batch_size BATCH_SIZE] [-d {squad,squad-nqg}] [--epoch EPOCH] [--lr LR] [--dev DEV] [--server] [--run_test] [-fc FROM_CHECKPOINT] [--precision {16,32}]

optional arguments:
  -h, --help            show this help message and exit
  --base_model {bert-base-uncased,bert-large-uncased,roberta-base,roberta-large,albert-base-v1,albert-large-v1,albert-base-v2,albert-large-v2}
  --batch_size BATCH_SIZE
  -d {squad,squad-nqg}, --dataset {squad,squad-nqg}
  --epoch EPOCH
  --lr LR
  --dev DEV
  --server
  --run_test
  -fc FROM_CHECKPOINT, --from_checkpoint FROM_CHECKPOINT
  --precision {16,32}, -fp {16,32}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].