185 Open source projects that are alternatives of or similar to Sacremoses

Tooling to play around with multilingual machine translation for Indian Languages.

Thot toolkit for statistical machine translation

Fast and customizable text tokenization library with BPE and SentencePiece support

Masakhane Web is a translation web application for solely African Languages.

Neural machine translation implementation using dynet's python bindings

Library of state-of-the-art models (PyTorch) for NLP tasks

Chinese tokenizer for tantivy, based on jieba-rs

Machine translation is the task of converting one language to other. Unlike the traditional phrase-based translation system which consists of many small sub-components that are tuned separately, neural machine translation attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.

VSCode extension to highlight nested code blocks

A simple tokenizer in Ruby for NLP tasks.

A seq2seq with attention dialogue/MT model implemented by TensorFlow.

英中机器文本翻译

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

A Python implementation of Farasa toolkit

A very simple python tokenizer for Hebrew text.

Lex is an implementation of lex tool in Ruby.

Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.

The Business Scene Dialogue corpus

A Python package that do stemming, tokenization, sentence breaking, segmentation, normalization, POS tagging for Arabic language.

A morphological analysis library.

This is a plugin to allow OmegaT to source machine translations from Tencent Cloud.

Text2Text: Cross-lingual natural language processing and generation toolkit

The official code repository for MetricMT - a reward optimization method for NMT with learned metrics

Curated list of all NLP Resources

A repository to bind mecab for Python 3.5+. Not using swig nor pybind. (Not Maintained Now)

臺灣言語工具

Tokenizers and lemmatizers for Go

A modern CSS value compiler in JavaScript

Some native scoring script plugins for elasticsearch

Visualization for simple attention and Google's multi-head attention.

Tokenize English sentences using neural networks.

Morphological analyzer `mystem` (Russian language) wrapper for JVM languages

Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.

Tracking the progress in end-to-end speech translation

A PSR-2-R code sniffer and code-style auto-correction-tool - including many useful additions

Tokenize CSS according to the CSS Syntax

A PyTorch implementation of Transformer in "Attention is All You Need"

Multilingual tokenizer that automatically tags each token with its type

Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.

High-Performance Stemmer, Tokenizer, and Spell Checker for R

Starter code for the VMT task and challenge

A tool for translating text from source grammar to target grammar (context-free) with corresponding dictionary.

Juman++ (a Morphological Analyzer Toolkit)

Interactive Neural Machine Translation tool

A tool that locates, downloads, and extracts machine translation corpora

A universal lexer library in Swift.

使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 | A DNN Chinese Tokenizer by Using PaddlePaddle

Reader Translator Generator - NMT toolkit based on pytorch

C library for Lexical Analysis

A small godot project with a tokenizer written in GDScript.

This repository contains the code and data of the paper titled "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation" published in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), November 16 - November 20, 2020.

Sanskrit compound segmentation using seq2seq model

A Fast Neural Machine Translation System. It is developed in C++ and resorts to NiuTensor for fast tensor APIs.

Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.

Contains various architectures and novel paper implementations for Natural Language Processing tasks like Sequence Modelling and Neural Machine Translation.

Research code for ACL 2020 paper: "Distilling Knowledge Learned in BERT for Text Generation".

Berserker - BERt chineSE woRd toKenizER

A Pytorch Implementation of "Attention is All You Need" and "Weighted Transformer Network for Machine Translation"

A simple interpreter for a large subset of Pascal language written for educational purposes

Neural Machine Translation on the Nepali-English language pair