298 Open source projects that are alternatives of or similar to Tokenizer

A tokenizer and sentence splitter for German and English web and social media texts.

➿ Addons, plugins, components and extensions (@componette ❤️ @nette)

A Cross-Platform String and Regular Expression Library written in Swift.

The greynir.is natural language processing website for Icelandic

A list of cyber-chef recipes and curated links

DebugView++, collects, views, filters your application logs, and highlights information that is important to you!

For composing regular expressions without the need for double-escaping inside strings.

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

Match regular expressions into struct fields

SQL Eval Function | Dynamically Evaluate Expression in SQL Server using C# Syntax

NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser

Self-contained Japanese Morphological Analyzer written in pure Go

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

A collection of basic python modules for spoken natural language processing

A julia code generator for regular expressions

A small .NET math library.

Onigmo is a regular expressions library forked from Oniguruma.

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHP standards checker.

🦖 Composable JavaScript regular expressions

🚧 Regular Expression Excited!

Regular expressions that match the mobile phone number in mainland China. / 一组匹配中国大陆手机号码的正则表达式。

POS Tagger, lemmatizer and stemmer for french language in javascript

Open Korean Text Processor - An Open-source Korean Text Processor

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

One-file semantic DSL-free templates direto da roça for the browser and server.

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

🔍 An RStudio addin slash regex utility belt

A Regular Expression game for Android

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

HaE - BurpSuite Highlighter and Extractor

A fresh approach to string manipulation in R

Analysis of the characteristics of different countries

Blazing fast and accurate glob matcher written JavaScript, with no dependencies and full support for standard and extended Bash glob features, including braces, extglobs, POSIX brackets, and regular expressions.

Kadot, the unsupervised natural language processing library.

A full-featured regex engine in pure Go based on the .NET engine

Tool to generate random strings from Go/RE2 regular expressions (Migrated to https://git.sr.ht/~nilium/regen)

The EasyRichText widget provides an easy way to use RichText.

The fast scanner generator for Java™ with full Unicode support

Regular Expression Analyzer and Composer for Node.js / XPCOM / Browser Javascript, PHP, Python

Utility to convert between various subscription format

✳️  Electron app for generating regular expressions

Fast, minimal glob matcher for node.js. Similar to micromatch, minimatch and multimatch, but without support for extended globs (extglobs), posix brackets or braces, and with complete Bash 4.3 wildcard support: ("*", "**", and "?").

Fuzzing Parsers with Tokens

Find interesting and potentially hazardous commits in git projects

A sane interface for php's built in preg_* functions

A Java library for generating String from a regular expression.

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

Pass two numbers, get a regex-compatible source string for matching ranges. Fast compiler, optimized regex, and validated against more than 2.78 million test assertions. Useful for creating regular expressions to validate numbers, ranges, years, etc.

✅ A simple tool to check source code against a set of Nette coding standards.

👸 API documentation generator (JSON-RPC / REST)

Regular expressions for swift

A multilingual command line sentence tokenizer in Golang

Python port of Moses tokenizer, truecaser and normalizer

Colin's ALM Corner Custom Build Tasks

🔁 Form multiplier & replicator for Nette Framework

‼️ Matches strings against configurable strings, globs, regular expressions, and/or functions