vlarine / Transformers Ru
Licence: apache-2.0
A list of pretrained Transformer models for the Russian language.
Stars: ✭ 150
Projects that are alternatives of or similar to Transformers Ru
Project kojak
Training a Neural Network to Detect Gestures and Control Smart Home Devices with OpenCV in Python
Stars: ✭ 147 (-2%)
Mutual labels: jupyter-notebook
Phonetic Similarity Vectors
Source code to accompany my paper "Poetic sound similarity vectors using phonetic features"
Stars: ✭ 148 (-1.33%)
Mutual labels: jupyter-notebook
Forecasting
Time Series Forecasting Best Practices & Examples
Stars: ✭ 2,123 (+1315.33%)
Mutual labels: jupyter-notebook
Bayesian Analysis With Python
Bayesian Analysis with Python by Packt
Stars: ✭ 148 (-1.33%)
Mutual labels: jupyter-notebook
Www old.julialang.org
Julia Project web site (Old)
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Deeplab v2
基于v2版本的deeplab,使用VGG16模型，在VOC2012，Pascal-context，NYU-v2等多个数据集上进行训练
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Covid 19 Eda Tutorial
This tutorial's purpose is to introduce people to the [2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19) and how to explore it using some foundational packages in the Scientific Python Data Science stack.
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Practicaldl
A Practical Guide to Deep Learning with TensorFlow 2.0 and Keras materials for Frontend Masters course
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Machine Learning
python，机器学习笔记，machine learning，nlp
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Cpndet
Corner Proposal Network for Anchor-free, Two-stage Object Detection
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Carnd Mercedes Sf Utilities
Tools for Sensor Fusion processing.
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Animatedpythonpatterns
Animated GIF patterns made in Python Matplotlib.
Stars: ✭ 148 (-1.33%)
Mutual labels: jupyter-notebook
Machine Learning
🌎 machine learning tutorials (mainly in Python3)
Stars: ✭ 1,924 (+1182.67%)
Mutual labels: jupyter-notebook
Testovoe
Home assignments for data science positions
Stars: ✭ 149 (-0.67%)
Mutual labels: jupyter-notebook
Pytorch Tutorials Kr
🇰🇷PyTorch에서 제공하는 튜토리얼의 한국어 번역을 위한 저장소입니다. (Translate PyTorch tutorials in Korean🇰🇷)
Stars: ✭ 148 (-1.33%)
Mutual labels: jupyter-notebook
Transformers-ru
A list of pretrained Transformer models for the Russian language (including multilingual models).
Code for the model using and visualisation is from the following repos:
Models
There are models form:
- DeepPavlov project
- Hugging Face repository
- Facebook research
- Facebook's fairseq
- Denis Antyukhov Google Colab code
- Russian RuBERTa
| Model description | # params | Config | Vocabulary | Model | BPE codes |
|---|---|---|---|---|---|
| BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters | 170M | [huggingface] 1K | [huggingface] 973K | [huggingface] 682M | |
| BERT-Base, Multilingual Uncased: 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters | 160M | [huggingface] 1K | [huggingface] 852K | [huggingface] 642M | |
| RuBERT, Russian, cased, 12-layer, 768-hidden, 12-heads, 180M parameters | 170M | [deeppavlov] 636M | |||
| SlavicBERT, Slavic (bg, cs, pl, ru), cased, 12-layer, 768-hidden, 12-heads, 180M parameters | 170M | [deeppavlov] 636M | |||
| XLM (MLM) 15 languages | 237M | [huggingface] 1K |
[huggingface] 2,9M [facebook] 1,5M |
[huggingface] 1,3G [facebook] 1,3G |
[huggingface] 1,4M [facebook] 1,4M |
| XLM (MLM+TLM) 15 languages | 237M | [huggingface] 1K |
[huggingface] 2,9M [facebook] 1,5M |
[huggingface] 661M [facebook] 665M |
[huggingface] 1,4M [facebook] 1,4M |
| XLM (MLM) 17 languages | [facebook] 2,9M | [facebook] 1,1G | [facebook] 2,9M | ||
| XLM (MLM) 100 languages | [facebook] 3,0M | [facebook] 1,1G | [facebook] 2,9M | ||
| Denis Antyukhov BERT-Base, Russian, Uncased, 12-layer, 768-hidden, 12-heads | 176M | [bert_resourses] 1,9G | |||
| Facebook-FAIR's WMT'19 en-ru | [fairseq] 12G | ||||
| Facebook-FAIR's WMT'19 ru-en | [fairseq] 12G | ||||
| Facebook-FAIR's WMT'19 ru | [fairseq] 2,1G | ||||
| Russian RuBERTa | [Google Drive] 247M |
Converting TensorFlow models to PyTorch
Downloading and converting the DeepPavlov model:
$ wget 'http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v1.tar.gz'
$ tar -xzf rubert_cased_L-12_H-768_A-12_v1.tar.gz
$ python3 convert_tf_checkpoint_to_pytorch.py \
--tf_checkpoint_path rubert_cased_L-12_H-768_A-12_v1/bert_model.ckpt \
--bert_config_file rubert_cased_L-12_H-768_A-12_v1/bert_config.json \
--pytorch_dump_path rubert_cased_L-12_H-768_A-12_v1/bert_model.bin
Models comparison
There are scripts to train and evaluate models on the Sber SQuAD dataset for the russian language [download dataset].
Comparision of BERT models trained on the Sber SQuAD dataset:
| Model | EM (dev) | F-1 (dev) |
|---|---|---|
| BERT-Base, Multilingual Cased | 64.85 | 83.68 |
| BERT-Base, Multilingual Uncased | 64.73 | 83.25 |
| RuBERT | 66.38 | 84.58 |
| SlavicBERT | 65.23 | 83.68 |
| RuBERTa-base | 59.45 | 78.60 |
Visualization
The attention-head view visualization from BertViz:
The model view visualization from BertViz:
The neuron view visualization from BertViz:
Generative models
GPT-2 models
Mikhail Grankin's model
Code: https://github.com/mgrankin/ru_transformers
Download models:
pip install awscli
aws s3 sync --no-sign-request s3://models.dobro.ai/gpt2/ru/unfreeze_all gpt2
Vladimir Larin's model
- Code: https://github.com/vlarine/ruGPT2
- Model: gpt2_345m.tgz (4,2G)
RNN Models
There are some RNN models for russian language.
ELMo
DeepPavlov
- ELMo on Russian Wikipedia: [config], [model]
- ELMo on Russian WMT News: [config], [model]
- ELMo on Russian Twitter: [config], [model]
RusVectōrēs
- RNC and Wikipedia. December 2018 (tokens): [model]
- RNC and Wikipedia. December 2018 (lemmas): [model]
- Taiga 2048. December 2019 (lemmas): [model]
ULMFit
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].