All Projects → csebuetnlp → banglanmt

csebuetnlp / banglanmt

Licence: other
This repository contains the code and data of the paper titled "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation" published in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), November 16 - November 20, 2020.

Programming Languages

python
139335 projects - #7 most used programming language
perl
6916 projects
shell
77523 projects
emacs lisp
2029 projects
smalltalk
420 projects
ruby
36898 projects - #4 most used programming language

Projects that are alternatives of or similar to banglanmt

Indian ParallelCorpus
Curated list of publicly available parallel corpus for Indian Languages
Stars: ✭ 23 (-74.73%)
Mutual labels:  neural-machine-translation, parallel-corpus, parallel-corpora, low-resource-languages, low-resource-machine-translation
BSD
The Business Scene Dialogue corpus
Stars: ✭ 51 (-43.96%)
Mutual labels:  machine-translation, parallel-corpus, parallel-corpora
Nspm
🤖 Neural SPARQL Machines for Knowledge Graph Question Answering.
Stars: ✭ 156 (+71.43%)
Mutual labels:  machine-translation, neural-machine-translation
Mtbook
《机器翻译:基础与模型》肖桐 朱靖波 著 - Machine Translation: Foundations and Models
Stars: ✭ 2,307 (+2435.16%)
Mutual labels:  machine-translation, neural-machine-translation
Attention-Visualization
Visualization for simple attention and Google's multi-head attention.
Stars: ✭ 54 (-40.66%)
Mutual labels:  machine-translation, neural-machine-translation
Mt Paper Lists
MT paper lists (by conference)
Stars: ✭ 105 (+15.38%)
Mutual labels:  machine-translation, neural-machine-translation
Opus Mt
Open neural machine translation models and web services
Stars: ✭ 111 (+21.98%)
Mutual labels:  machine-translation, neural-machine-translation
Opennmt
Open Source Neural Machine Translation in Torch (deprecated)
Stars: ✭ 2,339 (+2470.33%)
Mutual labels:  machine-translation, neural-machine-translation
Thumt
An open-source neural machine translation toolkit developed by Tsinghua Natural Language Processing Group
Stars: ✭ 550 (+504.4%)
Mutual labels:  machine-translation, neural-machine-translation
bergamot-translator
Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.
Stars: ✭ 181 (+98.9%)
Mutual labels:  machine-translation, neural-machine-translation
parallel-corpora-tools
Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.
Stars: ✭ 35 (-61.54%)
Mutual labels:  machine-translation, neural-machine-translation
Opennmt Tf
Neural machine translation and sequence learning using TensorFlow
Stars: ✭ 1,223 (+1243.96%)
Mutual labels:  machine-translation, neural-machine-translation
Sockeye
Sequence-to-sequence framework with a focus on Neural Machine Translation based on Apache MXNet
Stars: ✭ 990 (+987.91%)
Mutual labels:  machine-translation, neural-machine-translation
Subword Nmt
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
Stars: ✭ 1,819 (+1898.9%)
Mutual labels:  machine-translation, neural-machine-translation
Nematus
Open-Source Neural Machine Translation in Tensorflow
Stars: ✭ 730 (+702.2%)
Mutual labels:  machine-translation, neural-machine-translation
Npmt
Towards Neural Phrase-based Machine Translation
Stars: ✭ 175 (+92.31%)
Mutual labels:  machine-translation, neural-machine-translation
NiuTrans.NMT
A Fast Neural Machine Translation System. It is developed in C++ and resorts to NiuTensor for fast tensor APIs.
Stars: ✭ 112 (+23.08%)
Mutual labels:  machine-translation, neural-machine-translation
Nmt Keras
Neural Machine Translation with Keras
Stars: ✭ 501 (+450.55%)
Mutual labels:  machine-translation, neural-machine-translation
Opennmt Py
Open Source Neural Machine Translation in PyTorch
Stars: ✭ 5,378 (+5809.89%)
Mutual labels:  machine-translation, neural-machine-translation
Modernmt
Neural Adaptive Machine Translation that adapts to context and learns from corrections.
Stars: ✭ 231 (+153.85%)
Mutual labels:  machine-translation, neural-machine-translation

Bangla-NMT

This repository contains the code and data of the paper titled "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation" published in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), November 16 - November 20, 2020.

Table of Contents

Datasets

Download the dataset from here. This includes:

  • Our original 2.75M training corpus (2.75M/)
  • Preprocessed training corpus (data/)
  • RisingNews dev/test sets (data/)
  • Preprocessed sipc dev/test sets (data/)
  • Sentencepiece vocabulary models for Bengali and English (vocab/)

Dependencies

Segmentation

Batch-filtering

Training

License

Contents of this repository are licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

Citation

If you use this dataset or code modules, please cite the following paper:

@inproceedings{hasan-etal-2020-low,
    title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
    author = "Hasan, Tahmid  and
      Bhattacharjee, Abhik  and
      Samin, Kazi  and
      Hasan, Masum  and
      Basak, Madhusudan  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
    doi = "10.18653/v1/2020.emnlp-main.207",
    pages = "2612--2623",
    abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].