CyberZHG / Keras Xlnet
Licence: mit
Implementation of XLNet that can load pretrained checkpoints
Stars: ✭ 159
Programming Languages
python
139335 projects - #7 most used programming language
Labels
Projects that are alternatives of or similar to Keras Xlnet
Getlang
Natural language detection package in pure Go
Stars: ✭ 110 (-30.82%)
Mutual labels: language-model
Electra
中文 预训练 ELECTRA 模型: 基于对抗学习 pretrain Chinese Model
Stars: ✭ 132 (-16.98%)
Mutual labels: language-model
Awesome Sentence Embedding
A curated list of pretrained sentence and word embedding models
Stars: ✭ 1,973 (+1140.88%)
Mutual labels: language-model
Lingo
package lingo provides the data structures and algorithms required for natural language processing
Stars: ✭ 113 (-28.93%)
Mutual labels: language-model
Kogpt2 Finetuning
🔥 Korean GPT-2, KoGPT2 FineTuning cased. 한국어 가사 데이터 학습 🔥
Stars: ✭ 124 (-22.01%)
Mutual labels: language-model
Tupe
Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve existing models like BERT.
Stars: ✭ 143 (-10.06%)
Mutual labels: language-model
Easy Bert
A Dead Simple BERT API for Python and Java (https://github.com/google-research/bert)
Stars: ✭ 106 (-33.33%)
Mutual labels: language-model
Transformer Lm
Transformer language model (GPT-2) with sentencepiece tokenizer
Stars: ✭ 154 (-3.14%)
Mutual labels: language-model
Chars2vec
Character-based word embeddings model based on RNN for handling real world texts
Stars: ✭ 130 (-18.24%)
Mutual labels: language-model
Awd Lstm Lm
LSTM and QRNN Language Model Toolkit for PyTorch
Stars: ✭ 1,834 (+1053.46%)
Mutual labels: language-model
Haystack
🔍 Haystack is an open source NLP framework that leverages Transformer models. It enables developers to implement production-ready neural search, question answering, semantic document search and summarization for a wide range of applications.
Stars: ✭ 3,409 (+2044.03%)
Mutual labels: language-model
Dynamic Memory Networks Plus Pytorch
Implementation of Dynamic memory networks plus in Pytorch
Stars: ✭ 123 (-22.64%)
Mutual labels: language-model
Ld Net
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Stars: ✭ 148 (-6.92%)
Mutual labels: language-model
Keras Gpt 2
Load GPT-2 checkpoint and generate texts
Stars: ✭ 113 (-28.93%)
Mutual labels: language-model
Electra pytorch
Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)
Stars: ✭ 149 (-6.29%)
Mutual labels: language-model
Transformers
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Stars: ✭ 55,742 (+34957.86%)
Mutual labels: language-model
Clue
中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
Stars: ✭ 2,425 (+1425.16%)
Mutual labels: language-model
Speecht
An opensource speech-to-text software written in tensorflow
Stars: ✭ 152 (-4.4%)
Mutual labels: language-model
Awesome Speech Recognition Speech Synthesis Papers
Automatic Speech Recognition (ASR), Speaker Verification, Speech Synthesis, Text-to-Speech (TTS), Language Modelling, Singing Voice Synthesis (SVS), Voice Conversion (VC)
Stars: ✭ 2,085 (+1211.32%)
Mutual labels: language-model
Keras XLNet
Unofficial implementation of XLNet. Embedding extraction and embedding extract with memory show how to get the outputs of the last transformer layer using pre-trained checkpoints.
Install
pip install keras-xlnet
Usage
Fine-tuning on GLUE
Click the task name to see the demos with base model:
Task Name | Metrics | Approximate Results on Dev Set |
---|---|---|
CoLA | Matthew Corr. | 52 |
SST-2 | Accuracy | 93 |
MRPC | Accuracy/F1 | 86/89 |
STS-B | Pearson Corr. / Spearman Corr. | 86/87 |
QQP | Accuracy/F1 | 90/86 |
MNLI | Accuracy | 84/84 |
QNLI | Accuracy | 86 |
RTE | Accuracy | 64 |
WNLI | Accuracy | 56 |
(Only 0s are predicted in WNLI dataset)
Load Pretrained Checkpoints
import os
from keras_xlnet import Tokenizer, load_trained_model_from_checkpoint, ATTENTION_TYPE_BI
checkpoint_path = '.../xlnet_cased_L-24_H-1024_A-16'
tokenizer = Tokenizer(os.path.join(checkpoint_path, 'spiece.model'))
model = load_trained_model_from_checkpoint(
config_path=os.path.join(checkpoint_path, 'xlnet_config.json'),
checkpoint_path=os.path.join(checkpoint_path, 'xlnet_model.ckpt'),
batch_size=16,
memory_len=512,
target_len=128,
in_train_phase=False,
attention_type=ATTENTION_TYPE_BI,
)
model.summary()
Arguments batch_size
, memory_len
and target_len
are maximum sizes used for initialization of memories. The model used for training a language model is returned if in_train_phase
is True
, otherwise a model used for fine-tuning will be returned.
About I/O
Note that shuffle
should be False
in either fit
or fit_generator
if memories are used.
in_train_phase
is False
3 inputs:
- IDs of tokens, with shape
(batch_size, target_len)
. - IDs of segments, with shape
(batch_size, target_len)
. - Length of memories, with shape
(batch_size, 1)
.
1 output:
- The feature for each token, with shape
(batch_size, target_len, units)
.
in_train_phase
is True
4 inputs:
- IDs of tokens, with shape
(batch_size, target_len)
. - IDs of segments, with shape
(batch_size, target_len)
. - Length of memories, with shape
(batch_size, 1)
. - Masks of tokens, with shape
(batch_size, target_len)
.
1 output:
- The probability of each token in each position, with shape
(batch_size, target_len, num_token)
.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].