chengchingwen / Transformers.jl
Licence: mit
Julia Implementation of Transformer models
Stars: ✭ 173
Programming Languages
julia
2034 projects
Projects that are alternatives of or similar to Transformers.jl
Nlp Tutorial
Natural Language Processing Tutorial for Deep Learning Researchers
Stars: ✭ 9,895 (+5619.65%)
Mutual labels: natural-language-processing, attention, transformer
Gpt2
PyTorch Implementation of OpenAI GPT-2
Stars: ✭ 64 (-63.01%)
Mutual labels: natural-language-processing, transformer
Multimodal Sentiment Analysis
Attention-based multimodal fusion for sentiment analysis
Stars: ✭ 172 (-0.58%)
Mutual labels: natural-language-processing, attention
Multimodal Toolkit
Multimodal model for text and tabular data with HuggingFace transformers as building block for text data
Stars: ✭ 78 (-54.91%)
Mutual labels: natural-language-processing, transformer
Cell Detr
Official and maintained implementation of the paper Attention-Based Transformers for Instance Segmentation of Cells in Microstructures [BIBM 2020].
Stars: ✭ 26 (-84.97%)
Mutual labels: attention, transformer
Vietnamese Electra
Electra pre-trained model using Vietnamese corpus
Stars: ✭ 55 (-68.21%)
Mutual labels: natural-language-processing, transformer
Absa Pytorch
Aspect Based Sentiment Analysis, PyTorch Implementations. 基于方面的情感分析,使用PyTorch实现。
Stars: ✭ 1,181 (+582.66%)
Mutual labels: natural-language-processing, attention
Speech Transformer
A PyTorch implementation of Speech Transformer, an End-to-End ASR with Transformer network on Mandarin Chinese.
Stars: ✭ 565 (+226.59%)
Mutual labels: attention, transformer
Multiturndialogzoo
Multi-turn dialogue baselines written in PyTorch
Stars: ✭ 106 (-38.73%)
Mutual labels: attention, transformer
Transformers
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Stars: ✭ 55,742 (+32120.81%)
Mutual labels: natural-language-processing, transformer
Bertqa Attention On Steroids
BertQA - Attention on Steroids
Stars: ✭ 112 (-35.26%)
Mutual labels: attention, transformer
Awesome Fast Attention
list of efficient attention modules
Stars: ✭ 627 (+262.43%)
Mutual labels: attention, transformer
Attention Is All You Need Pytorch
A PyTorch implementation of the Transformer model in "Attention is All You Need".
Stars: ✭ 6,070 (+3408.67%)
Mutual labels: natural-language-processing, attention
Deeplearning Nlp Models
A small, interpretable codebase containing the re-implementation of a few "deep" NLP models in PyTorch. Colab notebooks to run with GPUs. Models: word2vec, CNNs, transformer, gpt.
Stars: ✭ 64 (-63.01%)
Mutual labels: attention, transformer
Awesome Bert Nlp
A curated list of NLP resources focused on BERT, attention mechanism, Transformer networks, and transfer learning.
Stars: ✭ 567 (+227.75%)
Mutual labels: natural-language-processing, transformer
Multihead Siamese Nets
Implementation of Siamese Neural Networks built upon multihead attention mechanism for text semantic similarity task.
Stars: ✭ 144 (-16.76%)
Mutual labels: natural-language-processing, attention
Neural sp
End-to-end ASR/LM implementation with PyTorch
Stars: ✭ 408 (+135.84%)
Mutual labels: attention, transformer
Pytorch Original Transformer
My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.
Stars: ✭ 411 (+137.57%)
Mutual labels: attention, transformer
Njunmt Tf
An open-source neural machine translation system developed by Natural Language Processing Group, Nanjing University.
Stars: ✭ 97 (-43.93%)
Mutual labels: attention, transformer
Sightseq
Computer vision tools for fairseq, containing PyTorch implementation of text recognition and object detection
Stars: ✭ 116 (-32.95%)
Mutual labels: attention, transformer
Julia implementation of transformer-based models, with Flux.jl.
Installation
In the Julia REPL:
]add Transformers
For using GPU, install & build:
]add CUDA
]build
julia> using CUDA
julia> using Transformers
#run the model below
.
.
.
Example
Using pretrained Bert with Transformers.jl
.
using Transformers
using Transformers.Basic
using Transformers.Pretrain
ENV["DATADEPS_ALWAYS_ACCEPT"] = true
bert_model, wordpiece, tokenizer = pretrain"bert-uncased_L-12_H-768_A-12"
vocab = Vocabulary(wordpiece)
text1 = "Peter Piper picked a peck of pickled peppers" |> tokenizer |> wordpiece
text2 = "Fuzzy Wuzzy was a bear" |> tokenizer |> wordpiece
text = ["[CLS]"; text1; "[SEP]"; text2; "[SEP]"]
@assert text == [
"[CLS]", "peter", "piper", "picked", "a", "peck", "of", "pick", "##led", "peppers", "[SEP]",
"fuzzy", "wu", "##zzy", "was", "a", "bear", "[SEP]"
]
token_indices = vocab(text)
segment_indices = [fill(1, length(text1)+2); fill(2, length(text2)+1)]
sample = (tok = token_indices, segment = segment_indices)
bert_embedding = sample |> bert_model.embed
feature_tensors = bert_embedding |> bert_model.transformers
See example
folder for the complete example.
Huggingface
We have some support for the models from huggingface/transformers
.
using Transformers.HuggingFace
# loading a model from huggingface model hub
julia> model = hgf"bert-base-cased:forquestionanswering";
┌ Warning: Transformers.HuggingFace.HGFBertForQuestionAnswering doesn't have field cls.
└ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:46
┌ Warning: Some fields of Transformers.HuggingFace.HGFBertForQuestionAnswering aren't initialized with loaded state: qa_outputs
└ @ Transformers.HuggingFace ~/peter/repo/gsoc2020/src/huggingface/models/models.jl:52
Current we only support a few model and the tokenizer part is not finished yet.
For more information
If you want to know more about this package, see the document and the series of blog posts I wrote for JSoC and GSoC. You can also tag me (@chengchingwen) on Julia's slack or discourse if you have any questions, or just create a new Issue on GitHub.
Roadmap
What we have before v0.2
-
Transformer
andTransformerDecoder
support for both 2d & 3d data. -
PositionEmbedding
implementation. -
Positionwise
for handling 2d & 3d input. - docstring for most of the functions.
- runable examples (see
example
folder) -
Transformers.HuggingFace
for handling pretrains fromhuggingface/transformers
What we will have in v0.2.0
- Complete tokenizer APIs
- tutorials
- benchmarks
- more examples
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].