af-ai-center / Swebert
Labels
Projects that are alternatives of or similar to Swebert
Swedish BERT models
Arbetsförmedlingen (The Swedish Public Employment Service) has developed Swedish BERT models which were trained on Swedish Wikipedia with approximately 2 million articles and 300 million words.
Available Model Types
-
bert-base-swedish-uncased
: 12-layer, 768-hidden, 12-heads, 110M parameters -
bert-large-swedish-uncased
: 24-layer, 1024-hidden, 16-heads, 340M parameters
Usage
The models can be used as part of the transformers package like any other built-in or community-uploaded model.
This means that both tokenizer and model can be
instantiated using the from_pretrained()
method
of the BERT-related transformers classes like so:
pretrained_model_name = 'af-ai-center/bert-base-swedish-uncased'
tokenizer = BertTokenizer.from_pretrained(pretrained_model_name)
# PyTorch
model = BertModel.from_pretrained(pretrained_model_name)
# TensorFlow
model = TFBertModel.from_pretrained(pretrained_model_name)
Getting Started
The notebook getting_started_with_swebert.ipynb
shows some more details on how to use the models.
Make sure to run it in a virtual environment with the following packages installed:
pip install torch tensorflow transformers tokenizers notebook
Remarks
-
Note that the corpus that our Swedish BERT models are trained on is significantly smaller than in the case of the original English BERT models.
-
We are part of an ongoing effort to create more sophisticated Swedish language models, see https://www.ri.se/sv/vad-vi-gor/projekt/sprakmodeller-svenska-myndigheter