language-detector
language-detector detects the language of text
Installation
pip install language-detector
Python Version
Works with both Python 2 and 3
Use
from language_detector import detect_language
text = "I arrived in that city on January 4, 1937"
language = detect_language(text)
# prints English
Features
| Languages Supported |
|---|
| Arabic |
| English |
| Farsi |
| French |
| German |
| Khmer |
| Kurmanci (Kurdish) |
| Mandarin |
| Russian |
| Sorani (Kurdish) |
| Spanish |
| Turkish |
Testing
To test the package run
python -m unittest language_detector.tests.test
Comparison
Test is a comparison of how well language-detector and langid identify languages in the data sources.
| package | language-detector | langid |
|---|---|---|
| test-duration (in seconds) | 0.10 | 3.83 |
| accuracy | 96.77% | 67.74% |
Excluding Languages
If you don't want language-detector to look for certain languages, you can monkey-patch the code. For example, in order to exclude English:
import language_detector
language_detector.char_language = [cl for cl in char_language if cl[1] != "English"]
# proceed as normal
Datasets
The following is a list of datasets used for each language:
| Language | Datasets |
|---|---|
| Arabic | UN Corpora |
| English | UN Corpora |
| Farsi | BBC News Persian |
| French | UN Corpora |
| German | Deutsche Welle |
| Khmer | Cambodia Daily |
| Kurmanci (Kurdish) | Rudaw |
| Mandarin | UN Corpora |
| Russian | UN Corpora |
| Sorani (Kurdish) | Rudaw |
| Spanish | UN Corpora |
| Turkish | BBC News Türkçe |
How Does It Work?
When training the model, we scan all the data sources and compute the frequency of how often a character appears in each specific language. We also compute the frequency of how often a characters appears in all of the data sources for all the languages. For each language, we then calculate a score for each character as frequency_in_language / frequency_in_all_languages. We then save the top ten highest scoring characters for each language.
When detecting a language, we simply iterate through the saved characters (ten for each language), and add their score as a weighted-vote for each language. Whichever, language has the highest score is selected as the winner.
Contributing
If you'd like to contribute a new language, please consult CONTRIBUTING.md
Support
Contact the package author, Daniel J. Dufour, at [email protected]