94 Open source projects that are alternatives of or similar to tokenizer

Thot toolkit for statistical machine translation

Laravel token management

Fast and customizable text tokenization library with BPE and SentencePiece support

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

A small library for converting tokenized PHP source code into XML (and potentially other formats)

Query Translator is a search query translator with AST representation

A small .NET math library.

A tokenizer for Icelandic text

Solves basic Russian NLP tasks, API for lower level Natasha projects

Text tokenization and sentence segmentation (segtok v2)

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

Bitextor generates translation memories from multilingual websites.

Ruby Object Inspection for IRB

Suika 🍉 is a Japanese morphological analyzer written in pure Ruby

A collection of basic python modules for spoken natural language processing

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

A repository to bind mecab for Python 3.5+. Not using swig nor pybind. (Not Maintained Now)

💠The Lel programming language

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

Essential NLP & ML, short & fast pure Python code

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Source code tokenizer

A tokenizer and sentence splitter for German and English web and social media texts.

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

Greeb is a simple Unicode-aware regexp-based tokenizer.

PHP standards checker.

Tokenizes Chinese texts into words.

Colin's ALM Corner Custom Build Tasks

Tiny JavaScript tokenizer.

PHP calculator library for mathematical terms (expressions) passed as strings

A small godot project with a tokenizer written in GDScript.

The greynir.is natural language processing website for Icelandic

Fast, Consistent Tokenization of Natural Language Text

NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser

The aim of this repository is to show a baseline model for text classification by implementing a LSTM-based model coded in PyTorch. In order to provide a better understanding of the model, it will be used a Tweets dataset provided by Kaggle.

POS Tagger, lemmatizer and stemmer for french language in javascript

Replaced by foonathan/lexy

Fuzzing Parsers with Tokens

A morphological analysis library.

React component for tagging inputs.

Collection of developer toolkits

SNL(Small Nested Language) Compiler. Maven jUnit Tokenizer Lexer Syntax Parser. 编译原理 词法分析 语法分析

C++ Lexer Toolkit Library (LexerTk) https://www.partow.net/programming/lexertk/index.html

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

Parser Building Toolkit for JavaScript

Self-contained Japanese Morphological Analyzer written in pure Go

Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.

Open Korean Text Processor - An Open-source Korean Text Processor

aim to use JapaneseTokenizer as easy as possible

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Vietnamese tokenizer (Maximum Matching and CRF)

Kadot, the unsupervised natural language processing library.

Ruby toolkit for Amazon Alexa service

A universal lexer library in Swift.

Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.

A collection of NLP tools for Sinhalese (සිංහල).

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer