136 Open source projects that are alternatives of or similar to Japanesetokenizers

Self-contained Japanese Morphological Analyzer written in pure Go

Fuzzing Parsers with Tokens

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

A simple interpreter for a large subset of Pascal language written for educational purposes

Japanese pop-up dictionary extension for Chrome and Firefox.

NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

SNL(Small Nested Language) Compiler. Maven jUnit Tokenizer Lexer Syntax Parser. 编译原理 词法分析 语法分析

Rakuten MA (Python version)

Trying to consolidate japanese phonetic, and in particular pitch accent resources into one list

Awesome Japanese learning resource

The greynir.is natural language processing website for Icelandic

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

The fast scanner generator for Java™ with full Unicode support

POS Tagger, lemmatizer and stemmer for french language in javascript

Python port of Moses tokenizer, truecaser and normalizer

Python scraper for Language Pods such as Japanesepod101.com 👹 🗾 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨

A simple app to quiz the user on identifying Japanese characters.

React component for tagging inputs.

Chinese tokenizer for tantivy, based on jieba-rs

Ruby Object Inspection for IRB

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

Perfect for those who forgets kanji pronunciation

使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 | A DNN Chinese Tokenizer by Using PaddlePaddle

Dictionary based Sentiment Analysis for Japanese

Text2Text: Cross-lingual natural language processing and generation toolkit

A small library for converting tokenized PHP source code into XML (and potentially other formats)

A tokenizer and sentence splitter for German and English web and social media texts.

Open Korean Text Processor - An Open-source Korean Text Processor

A collection of basic python modules for spoken natural language processing

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Kadot, the unsupervised natural language processing library.

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

A small .NET math library.

Lex machinary for go.

PHP standards checker.

A multilingual command line sentence tokenizer in Golang

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

Juman++ (a Morphological Analyzer Toolkit)

Source code tokenizer

A Python package that do stemming, tokenization, sentence breaking, segmentation, normalization, POS tagging for Arabic language.

Laravel token management

Encapsulates the official Jisho.org API and also provides kanji, example, and stroke diagram search.

Colin's ALM Corner Custom Build Tasks

A very simple python tokenizer for Hebrew text.

💠The Lel programming language

Evidence-based Explanation Dataset (AACL-IJCNLP 2020)

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer

👨‍💻♐ OOP learning with anime magical girl. (魔法少女で学ぶオブジェクト指向)🧙

Solves basic Russian NLP tasks, API for lower level Natasha projects

pandas 公式チートシートの非公式翻訳版

PHP calculator library for mathematical terms (expressions) passed as strings

Japanese morphological analysis engine written in pure Python

Beautiful visualizations of how language differs among document types.

Topologically ordered lists of kanji for effective learning

A minimal Chrome / Firefox extension to help you learn Japanese words in each new tab.

Thot toolkit for statistical machine translation

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.