All Projects → Kyubyong → Neural_japanese_transliterator

Kyubyong / Neural_japanese_transliterator

Licence: apache-2.0
Can neural networks transliterate Romaji into Japanese correctly?

Programming Languages

python
139335 projects - #7 most used programming language
language
365 projects

Projects that are alternatives of or similar to Neural japanese transliterator

unihandecode
unihandecode is a transliteration library to convert all characters/words in Unicode into ASCII alphabet that aware with Language preference priorities
Stars: ✭ 71 (-58.24%)
Mutual labels:  japanese, transliteration
Inflected
A port of ActiveSupport's inflector to Node.js and the browser.
Stars: ✭ 144 (-15.29%)
Mutual labels:  transliteration
Languagepod101 Scraper
Python scraper for Language Pods such as Japanesepod101.com 👹 🗾 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨
Stars: ✭ 104 (-38.82%)
Mutual labels:  japanese
Fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
Stars: ✭ 125 (-26.47%)
Mutual labels:  japanese
Posuto
🏣📮〠 Japanese postal code data.
Stars: ✭ 109 (-35.88%)
Mutual labels:  japanese
Konoha
🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.
Stars: ✭ 130 (-23.53%)
Mutual labels:  japanese
Nodejs Ja
Node.js 日本語ローカリゼーション
Stars: ✭ 98 (-42.35%)
Mutual labels:  japanese
Avspeechsynthesizer Example
A companion project to the NSHipster article about AVSpeechSynthesizer
Stars: ✭ 157 (-7.65%)
Mutual labels:  transliteration
Pep8 Ja
PEP8 日本語版
Stars: ✭ 138 (-18.82%)
Mutual labels:  japanese
Cutlet
Japanese to romaji converter in Python
Stars: ✭ 124 (-27.06%)
Mutual labels:  japanese
Gse
Go efficient multilingual NLP and text segmentation; support english, chinese, japanese and other. Go 高性能多语言 NLP 和分词
Stars: ✭ 1,695 (+897.06%)
Mutual labels:  japanese
Textlint Rule Preset Jtf Style
JTF日本語標準スタイルガイド for textlint.
Stars: ✭ 112 (-34.12%)
Mutual labels:  japanese
Jprocessing
Japanese Natural Langauge Processing Libraries
Stars: ✭ 135 (-20.59%)
Mutual labels:  japanese
Topokanji
Topologically ordered lists of kanji for effective learning
Stars: ✭ 108 (-36.47%)
Mutual labels:  japanese
Japanese.js
Util collection for Japanese text processing. Hiraganize, Katakanize, and Romanize.
Stars: ✭ 150 (-11.76%)
Mutual labels:  japanese
Source Han Code Jp
Source Han Code JP | 源ノ角ゴシック Code
Stars: ✭ 1,362 (+701.18%)
Mutual labels:  japanese
Ichiran
Linguistic tools for texts in Japanese language
Stars: ✭ 120 (-29.41%)
Mutual labels:  japanese
Nadesiko3
Japanese Programming Language Nadesiko v3 (JavaScript)
Stars: ✭ 125 (-26.47%)
Mutual labels:  japanese
Katakana Terminator
A browser extension to convert gairaigo (Japanese loan words) back to English
Stars: ✭ 168 (-1.18%)
Mutual labels:  japanese
Transliterate
Convert Unicode characters to Latin characters using transliteration
Stars: ✭ 152 (-10.59%)
Mutual labels:  transliteration

Neural Japanese Transliteration—can you do better than SwiftKey™ Keyboard?

Abstract

In this project, I examine how well neural networks can convert Roman letters into the Japanese script, i.e., Hiragana, Katakana, or Kanji. The accuracy evaluation results for 896 Japanese test sentences outperform the SwiftKey™ keyboard, a well-known smartphone multilingual keyboard, by a small margin. It seems that neural networks can learn this task easily and quickly.

Requirements

  • NumPy >= 1.11.1
  • TensorFlow == 1.2
  • regex (Enables us to use convenient regular expression posix)
  • janome (for morph analysis)
  • romkan (for converting kana to romaji)

Background

  • The modern Japanese writing system employs three scripts: Hiragana, Katakana, and Chinese characters (kanji in Japanese).
  • Hiragana and Katakana are phonetic, while Chinese characters are not.
  • In the digital environment, people mostly type Roman alphabet (a.k.a. Romaji) to write Japanese. Basically, they rely on the suggestion the transliteration engine returns. Therefore, how accurately an engine can predict the word(s) the user has in mind is crucial with respect to a Japanese keyboard.
  • Look at the animation on the right. You are to type "nihongo", then the machine shows 日本語 on the suggestion bar.

Problem Formulation

I frame the problem as a seq2seq task.

Inputs: nihongo。
Outputs: 日本語。

Data

  • For training, we used Leipzig Japanese Corpus.
  • For evaluation, 896 Japanese sentences were collected separately. See data/test.csv.

Model Architecture

I adopted the encoder and the first decoder architecture of Tacotron, a speech synthesis model.

Contents

  • hyperparams.py contains hyperparameters. You can change the value if necessary.
  • annotate.py makes Romaji-Japanese parallel sentences.
  • prepro.py defines and makes vocabulary and training data.
  • modules.py has building blocks for networks.
  • networks.py has encoding and decoding networks.
  • data_load.py covers some functions regarding data loading.
  • utils.py has utility functions.
  • train.py is about training.
  • eval.py is about evaluation.

Training

  • STEP 1. Download Leipzig Japanese Corpus and extract jpn_news_2005-2008_1M-sentences.txt to data/ folder.
  • STEP 2. Adjust hyperparameters in hyperparams.py if necessary.
  • STEP 3. Run python annotate.py.
  • STEP 4. Run python prepro.py. Or download the preprocessed files.
  • STEP 5. Run train.py. Or download the pretrained files.

Testing

  • STEP 1. Run eval.py.
  • STEP 2. Install the latest SwiftKey keyboard app and manually test it for the same sentences. (Don't worry. You don't have to because I've done it:))

Results

The training curve looks like this.

The evaluation metric is CER (Character Error Rate). Its formula is

  • edit distance / # characters = CER.

The following is the results after 13 epochs, or 79,898 global steps. Details are available in results/*.csv.

Proposed (Greedy decoding) Proposed (Beam decoding) SwiftKey 6.4.8.57
1595/12057=0.132 1517/12057=0.125 1640/12057=0.136
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].