PoLitBert - Polish RoBERTa model

Polish RoBERTa model trained on Polish Wikipedia, Polish literature and Oscar. Major assumption is that good quality text will give good model.

We believe in open science and knowledge sharing, thus we decided to share complete code, params, experiment details and tensorboards.

Table of Contents

• Experiments setup and goals
• Data
  • Data processing for training
  • Training, testing dataset stats
• Training Polish RoBERTa protocol with Fairseq
• Pretrained models and vocabs
  • KLEJ evaluation
  • Details of models training
• Used libraries
• Acknowledgements
• About Ermlab Software

Experiments setup and goals

During experiments, we want to examine:

• impact of different learning schedulers for training speed and accuracy, tested:
  • linear schedule with warmup
  • cyclic schedule: cosine, triangular
• impact of training time on final accuracy

Data

• Polish Wikipedia dump 03.2020 - archive link https://dumps.wikimedia.org/plwiki/20200301 (not working anymore)
• Polish private book corpus (6GB)
• Cleaned Polish Oscar corpus (remove non-polish sentences, keep only valid sentences etc.)(Cleaned Polish Oscar details)

Data processing for training

Our main assumption is that good quality text should produce good language model. So far the most popular polish dataset was "Polish wikipedia dump" however this text characterize with formal language. Second source of text is polish part of Oscar corpus - crawled text from the polish internet. When we investigate this corpus with more details it appears that it contains a lot of: foreign sentences (in Russian, English, German etc.), too short sentences and not grammatical sentences (as words enumerations).

We prepared a few cleaning heuristics:

• remove sentences shorter than
• remove non polish sentences
• remove ungrammatical sentences (without verbs and with too many nouns)
• perform sentence tokenization and save each sentence in new line, after each document the new line was added

Data was cleaned with use of process_sentences.py script, the whole process is presented in the polish_process_data.ipynb notebook.

• Polish Wikipedia dump (03.2020)
  • corpus_wikipedia_2020-03-01_all_lines.zip (0.58 GB)
• Cleaned Polish Oscar corpus
  • corpus_oscar_2020-04-10_32M_lines.zip (3.35 GB)
  • corpus_oscar_2020-04-10_64M_lines.zip (3.45 GB)
  • corpus_oscar_2020-04-10_96M_lines.zip (3.49 GB)
  • corpus_oscar_2020-04-10_128M_lines.zip (3.53 GB)
  • corpus_oscar_2020-04-10_128M_above_lines.zip (1.93 GB)

Summary of Cleaned Polish Oscar corpus

Training, testing dataset stats

For testing we take ~10% of each corpus

Training Polish RoBERTA protocol with Fairseq

General recipe of the final data preparation and model training process:

1. Prepare huge text file data.txt e.g. Wikipedia text, where each sentence is in a new line and each article is separated by two new lines.
2. Take 10-15M lines and prepare another file for sentencepiece (vocabulary builder) - again, each sentence is in one line.
3. Train sentencepiece vocabulary and save it in fairseq format vocab.fairseq.txt.
4. Encode data.txt with trained sentencepiece model to data.sp.txt.
5. Preprocess data.sp.txt with fairseq-preprocess.
6. Run training.

Detailed data preparation steps for fairseq (vocab gen and binarization) are available in separate notebook polish_roberta_vocab.ipynb.

Commands needed to reproduce fairseq models with various training protocols may be found in polish_roberta_training.ipynb.

Pretrained models and vocabs

• PoLitBert_v32k_linear_50k
• PoLitBert_v32k_tri_50k
• PoLitBert_v32k_cos1_2_50k
• PoLitBert_v32k_tri_125k
• PoLitBert_32k_cos1_5
• PoLitBert_v32k_cos1_5_50k
• PoLitBert_v50k_linear_50k

KLEJ evaluation

All models were evaluated at 26.07.2020 with 9 KLEJ benchmark tasks . Below results were achieved with use of fine-tuning scripts from Polish RoBERTa without any further tweaks. which suggests that the potential of the models may not been fully utilized yet.

A comparison with other developed models is available in the continuously updated leaderboard of evaluation tasks.

Details of models training

We believe in open science and knowledge sharing, thus we decided to share complete code, params, experiment details and tensorboards.

Link to PoLitBert research log (same as below).

Used libraries

• KRNNT - Polish morphological tagger. - we use dockerized version
• langdetect - for detecting sentence language
• polyglot - for detecting sentence language
• sentencepiece
• Fairseq v0.9

Instalation dependecies and problems

• langdetect needs additional package
  • install sudo apt-get install libicu-dev
• sentencepiece was installed from source code

Acknowledgements

This is the joint work of companies Ermlab Software and Literacka

Part of the work was financed from the grant of The Polish National Centre for Research and Development no. POIR.01.01.01-00-1213/19, the beneficiary of which was Literacka. Project title "Asystent wydawniczy - oprogramowanie do analizy treści, wykorzystujące algorytmy sztucznej inteligencji w celu zautomatyzowania procesu wydawniczego i predykcji sukcesów rynkowych publikacji."

We would like to express ours gratitude to NVidia Inception Programme and Amazon AWS for providing the free GPU credits - thank you!

Authors:

• Krzysztof Sopyła
• Łukasz Sawaniewski

Also appreciate the help from

• simonefrancia from Musixmatch for his detailed explanations how they trained RoBERTa Italian model Umberto

About Ermlab Software

Ermlab - Polish machine learning company

🦉 Website | Repository

.