Language Codes

It is hard to tell the exact number of human languages on this planet because the definition of "language" varies depending on how one defines the distinction between languages and dialects. E.g., some languages can be grouped into a language family, and show small differences from others. For a general definition of language, the number of living languages is over 7,000^[1] but most of them are non-digitized. Here we list 383 languages with their codes, families, regions and etc. This list covers most of the majority languages in the world and a large number of minority languages. Also, we collect links to sites of multi-lingual corpora. They might help when one studies these languages and/or develop multi-lingual natural language processing (NLP) systems.

The Language List

Notes

1. ISO 639 is a standardized nomenclature used to classify languages. Each language is assigned a two-letter (639-1) and three-letter (639-2 and 639-3) lowercase abbreviation, amended in later versions of the nomenclature^[2].
2. For several minority languages without official Chinese name, we consult to other reliable sources^[3] or use its transliteration name.

Corpora

• Multi-lingual Data Projects：

• Multi-lingual Data Shared by MT Conference/Workshop：

  CCMT China Conference on Machine Translation (CCMT) , formerly known as China Workshop on Machine Translation (CWMT), is a flagship conference of machine translation in China.Its evaluations focus mainly on Chinese, English and domestic minority languages (Mongolian, Tibetan, Uyghur, etc.) in domains of news, spoken languages, governmental documents, etc. In addition, CCMT publishes all evaluation-related data on line.

  WMT WMT is hosted by Special Interest Group for Machine Translation (SIGMT) annually since 2006. WMT evaluation campaigns focuses on languages between English and over ten languages such as English, German, Finnish, Czech, Romanian, Polish, Russian, etc. in domains of news, information technology, biomedicine. WMT publishes all evaluation resources specific to each evaluation task, you can find it at Shared Task-Provided Data.

  NIST The NIST machine translation evaluation started in 2001 as part of the DARPA TIDES program. The evaluations are driven and coordinated by NIST as NIST OpenMT. In the early days, NIST evaluations mainly evaluated the translation performance from languages such as Arabic and Chinese to English. Furthermore, NIST has begun to evaluate low-resource language technologies since 2016. Results of past MT evaluations as well as resources specific to each evaluation can be accessed via the year-specific links.

  IWSLT The International Conference on Spoken Language Translation (IWSLT), which has been held annually since 2004, is also a distinctive evaluation campaign on spoken language translation. The test data includes multilingual subtitles of TED talks and QED lectures. The languages involve English, French, German, Czech, Chinese, Arabic and many other languages. Evaluation-related resources can be accessed at Shared Tasks-Training and Development Data.

  WAT The Workshop on Asian Translation (WAT) is a new open evaluation campaign focusing on Asian languages. The successive 8 workshops has been successfully jointly held by the Japan Science and Technology Agency (JST), the National Institute of Information and Communications Technology (NICT) and other institutions. WAT focuses on translation from mainstream Asian languages (Chinese, Korean, Hindi, etc.) and English to Japanese in comprehensive domains such as academic papers, patents, news and recipes. Datasets for evaluation can be accessed at Translation Task-Dataset via the year-specific links.

References

• ↑ "How many languages are there in the world". Ethnologue: Languages of the World. Retrieved 29 April 2021.

• ↑ List of ISO 639-1 codes. Wikipedia.org. Retrieved 29 April 2021.

• ↑ Numeral Systems of the World's Languages Retrieved June 2020.

• World language and its name, language family, writing systems and the number and distribution of its native speakers. Wikipedia.org. Retrieved March 21, 2020.

• Glottolog report for language families. Glottolog.org. Retrieved July 21, 2020.

• World Continents classification Retrieved October 21, 2020.

• SIL report for language and its codes. Retrieved March 20, 2021.

• Ethnologue report for language names, codes, writing systems and etc. Retrieved October 11, 2020.

• Wikidata report for language codes and writing systems. Retrieved May 10, 2021.

• Omniglot report on writing systems by language. Retrieved May 10, 2021.

• Scriptsource report on language features, writing systems, spoken countries and its status. Retrieved May 10, 2021.