65 Open source projects that are alternatives of or similar to codeprep

基于隐式马尔可夫模型和正向最大化匹配的中文分词系统

A Japanese tokenizer based on recurrent neural networks

Objective-C Runtime Analysis (Objective-C运行时分析)

SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

Java port of SymSpell: 1 million times faster through Symmetric Delete spelling correction algorithm

Vietnamese Word Tokenize

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Official code for Group-Transformer (Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model, COLING-2020).

Rakuten MA (Python version)

📍 🏢 🏦 🏣 🏪 🏬 LNEx: Location Name Extractor

百度NLP：分词，词性标注，命名实体识别，词重要性

Code of our paper Applying CodeBERT for Automated Program Repair of Java Simple Bugs which is accepted to MSR 2021.

Spelling correction and string segmentation written in Go

A comparison tool of Japanese tokenizers

Independently Recurrent Neural Networks (IndRNN) implemented in pytorch.

Thai Natural Language Processing in Python.

Thai Word Segmentation and Part-of-Speech Tagging with Deep Learning

BERT for Multitask Learning

⚙️ Code analysis of common Android projects and components.

Hashformers is a framework for hashtag segmentation with transformers.

Differentiable architecture search for convolutional and recurrent networks

A toolkit for Vietnamese word segmentation

Word Segmentation with Dynamic Programming

Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词等序列标注任务。 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label classification tasks of Chinese long text and short text, and supports sequence annotation tasks such as Chinese named entity recognition, part of speech ta…

Recurrent Neural Networks (RNN, GRU, LSTM) and their Bidirectional versions (BiRNN, BiGRU, BiLSTM) for word & character level language modelling in Theano

MONPA 罔拍是一個提供正體中文斷詞、詞性標註以及命名實體辨識的多任務模型

A Python 3 module that provides functions for splitting identifiers found in source code files.

Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. (DEPRECATED)

Cantonese Linguistics and NLP in Python

Detecting Software Plagiarism and Collusion since 1996.

Kiwi(지능형 한국어 형태소 분석기)

Library to analyze and collect metrics on source code

Source code for an ACL2016 paper of Chinese word segmentation

Paper: A Simple and Effective Neural Model for Joint Word Segmentation and POS Tagging

Unsupervised text tokenizer focused on computational efficiency

An implementation of the SZZ algorithm, i.e., an approach to identify bug-introducing commits.

Unsupervised text tokenizer for Neural Network-based text generation.

Sanskrit compound segmentation using seq2seq model

Python port of SymSpell

Fetch source-mapped sources. Peek by file, line, column. Node & browsers. Sync & async.

A Vietnamese natural language processing toolkit (NAACL 2018)

Code accompanying our ECCV-2020 paper on 3D Neural Listeners.

Juman++ (a Morphological Analyzer Toolkit)

An unsupervised Chinese word segmentation tool.

基于Tensorflow的中文分词模型

CKIP CoreNLP Toolkits

High performance unsupervised text tokenization for Ruby

Data Collection System For NLP/Speech Recognition

Fast SymSpell written in c++ and exposes to python via pybind11

An implementation of transformer-based language model for sentence rewriting tasks such as summarization, simplification, and grammatical error correction.

Converts from Chinese characters to pinyin, between simplified and traditional, and does word segmentation.

MozoLM: A language model (LM) serving library

Chinese Word Segmention Base on the Deep Learning and LSTM Neural Network

R package for Byte Pair Encoding / Unigram modelling based on Sentencepiece

Implement Differentiable Architecture Search (DARTS) for RNN with fastai

Neural Networks for Protein Sequence Alignment

Syllable segmentation tool for Myanmar language (Burmese) by Ye.

Java JNI wrapper for SentencePiece: unsupervised text tokenizer for Neural Network-based text generation.

Natural language processing in examples and games

👄 The most accurate natural language detection library for Go, suitable for long and short text alike