All Categories → Compilers → tokenizer

Top 89 tokenizer open source projects

Bitextor generates translation memories from multilingual websites.

✭ 168

python crawler translation tokenizer dictionaries crawl wget

Tiny JavaScript tokenizer.

✭ 166

javascript ecmascript regex tokenizer

Query Translator

Query Translator is a search query translator with AST representation

✭ 165

elasticsearch parser search generator ast query solr tokenizer

Fast, Consistent Tokenization of Natural Language Text

✭ 161

r nlp rstats r-package text-mining tokenizer

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

✭ 160

r nlp natural-language-processing r-package text-mining tokenizer pos-tagging

Replaced by foonathan/lexy

✭ 137

cplusplus tokenizer lexer

Fast and customizable text tokenization library with BPE and SentencePiece support

✭ 132

python cpp natural-language-processing unicode machine-translation tokenizer icu

Collection of developer toolkits

✭ 131

javascript parser developer-tools devtools development-environment tokenizer lexer

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

✭ 125

nlp japanese tokenizer

Parser Building Toolkit for JavaScript

✭ 1,795

javascript typescript open-source parsing tokenizer lexer parser-library grammars

Text tokenization and sentence segmentation (segtok v2)

✭ 123

python nlp segmentation tokenizer

Japanesetokenizers

aim to use JapaneseTokenizer as easy as possible

✭ 120

python nlp tokenizer japanese-language

Source code tokenizer

✭ 119

regular-expression tokenizer nette nette-framework

Kadot, the unsupervised natural language processing library.

✭ 108

python natural-language-processing text-classification word-embeddings tokenizer text-generation

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer

✭ 100

javascript markdown tokenizer

A tokenizer and sentence splitter for German and English web and social media texts.

✭ 85

python social-media english tokenizer

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

✭ 85

python django web regex routing url tokenizer

PHP standards checker.

✭ 82

ast code-quality tokenizer code-style

Sentence Splitter

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

✭ 82

python nlp tokenizer

Cols Agent Tasks

Colin's ALM Corner Custom Build Tasks

✭ 70

powershell build coverage versioning tokenizer tag

Ruby Object Inspection for IRB

✭ 69

ruby terminal syntax-highlighting tokenizer

PHP calculator library for mathematical terms (expressions) passed as strings

✭ 60

parser math mathematics string calculator tokenizer

Thot toolkit for statistical machine translation

✭ 53

python machine-learning natural-language-processing artificial-intelligence statistics machine-translation tokenizer

The greynir.is natural language processing website for Icelandic

✭ 47

python grammar natural-language-processing parser tokenizer tf-idf

A collection of basic python modules for spoken natural language processing

✭ 46

python natural-language-processing speech-recognition tts tokenizer pulseaudio

NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser

✭ 38

java nlp-machine-learning tokenizer

A small .NET math library.

✭ 36

library dotnet parser canvas math matrix vector geometry calculator draw tokenizer polygon function expression matrices

Nlp Js Tools French

POS Tagger, lemmatizer and stemmer for french language in javascript

✭ 32

javascript nlp postgresql tokenizer stemmer

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

✭ 30

ruby text-classification sentiment-analysis tokenizer

Fuzzing Parsers with Tokens

✭ 28

java parser fuzzing tokenizer

Laravel token management

✭ 10

laravel tokenizer

React Input Tags

React component for tagging inputs.

✭ 10

javascript react component autocomplete input tags token tokenizer tagging tag autosuggest

Lisp Esque Language

💠The Lel programming language

✭ 24

javascript lisp parser tokenizer

SNL(Small Nested Language) Compiler. Maven jUnit Tokenizer Lexer Syntax Parser. 编译原理词法分析语法分析

✭ 19

java tokenizer lexer

Solves basic Russian NLP tasks, API for lower level Natasha projects

✭ 788

python nlp visualization ner embeddings russian syntax tokenizer

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

✭ 689

swift tokenizer

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

✭ 613

python nlp tokenizer

Self-contained Japanese Morphological Analyzer written in pure Go

✭ 554

go hacktoberfest segmentation japanese korean tokenizer nlp-library pos-tagging japanese-language morphological-analysis

A small library for converting tokenized PHP source code into XML (and potentially other formats)

✭ 4,770

PHP xml tokenizer

Open Korean Text

Open Korean Text Processor - An Open-source Korean Text Processor

✭ 438

scala natural-language-processing korean text-processing tokenizer

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

✭ 435

python java nlp tokenizer

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

✭ 433

python nlp text-processing tokenizer nlp-library word-segmentation

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

✭ 434

javascript tokenizer regexp lexer

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

✭ 400

javascript parser development ast static-code-analysis tokenizer lexer

The fast scanner generator for Java™ with full Unicode support

✭ 380

java grammar scanner parsing flex maven-plugin tokenizer regexp lexer bazel-rules

Lex machinary for go.

✭ 335

go regular-expression tokenizer lexer

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

✭ 313

c tokenizer full-text-search chinese-word-segmentation

A multilingual command line sentence tokenizer in Golang

✭ 293

go cli tokenizer

Python port of Moses tokenizer, truecaser and normalizer

✭ 293

python nlp machine-translation tokenizer

Juman++ (a Morphological Analyzer Toolkit)

✭ 254

nlp japanese tokenizer pos-tagging part-of-speech-tagger morphological-analysis word-segmentation

pascal-interpreter

A simple interpreter for a large subset of Pascal language written for educational purposes

✭ 21

javascript parser parse interpreter scanner tokenizer ast visitor lexer lexical-analysis symbol-table syntax-analysis semantic-analysis pascal-language grammar-rules pascal-interpreter ast-nodes ast-node

ArabicProcessingCog

A Python package that do stemming, tokenization, sentence breaking, segmentation, normalization, POS tagging for Arabic language.

✭ 19

python nlp natural-language-processing tokenizer segmentation computational-linguistics text-processing arabic-language

Hebrew-Tokenizer

A very simple python tokenizer for Hebrew text.

✭ 16

python tokenizer hebrew

Chinese tokenizer for tantivy, based on jieba-rs

✭ 48

rust tokenizer full-text-search tantivy

PaddleTokenizer

使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 | A DNN Chinese Tokenizer by Using PaddlePaddle

✭ 14

javascript python nlp machine-learning neural-network tokenizer chinese paddle chinese-tokenizer paddlepaddle tokenizer-framework nature-language-process

Text2Text: Cross-lingual natural language processing and generation toolkit

A modern CSS value compiler in JavaScript

✭ 39

javascript CSS shell css parser generator compiler tokenizer css-in-js ast lexer traverser

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

✭ 32

python nlp tokenizer wordlist lemmatizer morphological-analysis lemmatiser tokenization lemmatization corpus-tools

Morphological analyzer `mystem` (Russian language) wrapper for JVM languages

✭ 21

scala natural-language-processing yandex tokenizer russian-specific computational-linguistics lemmatizer russian-morphology mystem

Tokenize CSS according to the CSS Syntax

✭ 52

typescript javascript css tokenizer

1-60 of 89 tokenizer projects