What is Gensim-data for?

Research datasets regularly disappear, change over time, become obsolete or come without a sane implementation to handle the data format reading and processing.

For this reason, Gensim launched its own dataset storage, committed to long-term support, a sane standardized usage API and focused on datasets for unstructured text processing (no images or audio). This Gensim-data repository serves as that storage.

There's no need for you to use this repository directly. Instead, simply install Gensim and use its download API (see the Quickstart below). It will "talk" to this repository automagically.

💡 When you use the Gensim download API, all data is stored in your ~/gensim-data home folder.

Read more about the project rationale and design decisions in this article: New Download API for Pretrained NLP Models and Datasets.

How does it work?

Technically, the actual (sometimes large) corpora and model files are being stored as release attachments here on Github. Each dataset (and each new version of each dataset) gets its own release, forever immutable.

Each release is accompanied by a usage example and release notes, for example: Corpus of USPTO Patents from 2017; English Wikipedia from 2017 with plaintext section.

🔴 Each dataset comes with its own license, which the users should study carefully before using the dataset!

Quickstart

To load a model or corpus, use either the Python or command line interface of Gensim (you'll need Gensim installed first):

• Python API

  Example: load a pre-trained model (gloVe word vectors):

  import gensim.downloader as api
  
  info = api.info()  # show info about available models/datasets
  model = api.load("glove-twitter-25")  # download the model and return as object ready for use
  model.most_similar("cat")
  
  """
  output:
  
  [(u'dog', 0.9590819478034973),
   (u'monkey', 0.9203578233718872),
   (u'bear', 0.9143137335777283),
   (u'pet', 0.9108031392097473),
   (u'girl', 0.8880630135536194),
   (u'horse', 0.8872727155685425),
   (u'kitty', 0.8870542049407959),
   (u'puppy', 0.886769711971283),
   (u'hot', 0.8865255117416382),
   (u'lady', 0.8845518827438354)]
  
  """
  

  Example: load a corpus and use it to train a Word2Vec model:

  from gensim.models.word2vec import Word2Vec
  import gensim.downloader as api
  
  corpus = api.load('text8')  # download the corpus and return it opened as an iterable
  model = Word2Vec(corpus)  # train a model from the corpus
  model.most_similar("car")
  
  """
  output:
  
  [(u'driver', 0.8273754119873047),
   (u'motorcycle', 0.769528865814209),
   (u'cars', 0.7356342077255249),
   (u'truck', 0.7331641912460327),
   (u'taxi', 0.718338131904602),
   (u'vehicle', 0.7177008390426636),
   (u'racing', 0.6697118878364563),
   (u'automobile', 0.6657308340072632),
   (u'passenger', 0.6377975344657898),
   (u'glider', 0.6374964714050293)]
  
  """
  

  Example: only download a dataset and return the local file path (no opening):

  import gensim.downloader as api
  
  print(api.load("20-newsgroups", return_path=True))  # output: /home/user/gensim-data/20-newsgroups/20-newsgroups.gz
  print(api.load("glove-twitter-25", return_path=True))  # output: /home/user/gensim-data/glove-twitter-25/glove-twitter-25.gz
  

• The same operations, but from CLI, command line interface:

  python -m gensim.downloader --info  # show info about available models/datasets
  python -m gensim.downloader --download text8  # download text8 dataset to ~/gensim-data/text8
  python -m gensim.downloader --download glove-twitter-25  # download model to ~/gensim-data/glove-twitter-50/
  

Available data

Datasets

Models

(generated by generate_table.py based on list.json)

Want to add a new corpus or model?

1. Compress your data set using gzip or bz2.

2. Share the compressed file on any file-sharing service.

3. Create a new issue and give us the dataset link. Add a detailed description on why and how you created the dataset, any related papers or research, plus how do you expect other users should use it. Include a code example where relevant.

Gensim-data is open source software released under the LGPL 2.1 license.

Copyright (c) 2018 RARE Technologies.