1057 Open source projects that are alternatives of or similar to Tokenizer

Thot toolkit for statistical machine translation

THE String Processing Package for R (with ICU)

Open Korean Text Processor - An Open-source Korean Text Processor

[ACL 2020] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

Tracking the progress in non-autoregressive generation (translation, transcription, etc.)

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

Python port of Moses tokenizer, truecaser and normalizer

《机器翻译：基础与模型》肖桐 朱靖波 著 - Machine Translation: Foundations and Models

State of the Art Natural Language Processing

This repo contains the source code in my personal column (https://zhuanlan.zhihu.com/zhaoyeyu), implemented using Python 3.6. Including Natural Language Processing and Computer Vision projects, such as text generation, machine translation, deep convolution GAN and other actual combat code.

Evaluation code for various unsupervised automated metrics for Natural Language Generation.

The new home of the ICU project source code.

The greynir.is natural language processing website for Icelandic

Source code and data for the paper "Towards String-to-Tree Neural Machine Translation"

Tooling to play around with multilingual machine translation for Indian Languages.

International Components for Unicode for .NET

Code for the collection and analysis of the MTNT dataset

Neural machine translation and sequence learning using TensorFlow

Implementations for a family of attention mechanisms, suitable for all kinds of natural language processing tasks and compatible with TensorFlow 2.0 and Keras.

Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

C# wrapper for ICU4C

Solving i18n for client-side and resource-constrained environments.

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

ByteNet for character-level language modelling

Multilingual word vectors in 78 languages

A Neural Framework for MT Evaluation

Drench yourself in Deep Learning, Reinforcement Learning, Machine Learning, Computer Vision, and NLP by learning from these exciting lectures!!

Swift APIs for ICU

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

Open neural machine translation models and web services

Drop-in replacements for base R string functions powered by stringi

Greeb is a simple Unicode-aware regexp-based tokenizer.

A collection of basic python modules for spoken natural language processing

Kadot, the unsupervised natural language processing library.

An overview of the AI-as-a-service landscape

Text tokenization and sentence segmentation (segtok v2)

RASA chatbot use case boilerplate

Calculating ROUGE score between two files (line-by-line)

A collection of Natural language processing pre-trained models.

Preprocessing Library for Natural Language Processing

Lyrics Generator aka Character-level Language Modeling with Multi-layer LSTM Recurrent Neural Network

A two-level morphological analyzer for Turkish.

Code examples in pyTorch and Tensorflow for CS230

PyTorch implementation of latent space reinforcement learning for E2E dialog published at NAACL 2019. It is released by Tiancheng Zhao (Tony) from Dialog Research Center, LTI, CMU

DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

RAT-el is an open source penetration test tool that allows you to take control of a windows machine. It works on the client-server model, the server sends commands and the client executes the commands and sends the result back to the server. The client is completely undetectable by anti-virus software.

Persian (Farsi) Stop Words List

ϲοｎｆｕѕаｂｌе＿һοｍоɡｌｙｐｈｓ

🔮 Neuro.js is machine learning library for building AI assistants and chat-bots (WIP).

aim to use JapaneseTokenizer as easy as possible

NLPCC 2016 微博分词评测项目

搜索所有中文NLP数据集，附常用英文NLP数据集

Beautiful visualizations of how language differs among document types.

Code for paper "Discourse-Aware Neural Extractive Text Summarization" (ACL20)

NLP, before and after spaCy

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

💱 Codepages for JS

Source code tokenizer

Python wraper for MetaMap