All Categories → No Category → tokenization

Top 21 tokenization open source projects

💫 Industrial-strength Natural Language Processing (NLP) in Python

charformer-pytorch

Implementation of the GBST block from the Charformer paper, in Pytorch

✭ 74

python deep-learning artificial-intelligence transformer tokenization

vgs-collect-ios

VGS Collect iOS SDK

✭ 17

swift security ios credit-card pci-dss pci collect tokenization vgs ssn zerodata

A tokenizer based on Unicode text segmentation (UAX 29), for Go

✭ 26

go unicode tokenization uax29

youtokentome-ruby

High performance unsupervised text tokenization for Ruby

✭ 17

ruby C++python unsupervised-learning word-segmentation tokenization npl bpe byte-pair-encoding

auto-data-tokenize

Identify and tokenize sensitive data automatically using Cloud DLP and Dataflow

✭ 21

java shell dataflow deidentification tokenization cloud-migration data-loss-prevention data-governance

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

✭ 32

python nlp tokenizer wordlist lemmatizer morphological-analysis lemmatiser tokenization lemmatization corpus-tools

bert tokenization for java

This is a java version of Chinese tokenization descried in BERT.

✭ 39

java chinese-nlp bert tokenization

The ultimate open source betting protocol. PolyCash is a P2P blockchain platform for wallets, asset issuance, bonds & gaming.

✭ 24

PHP javascript CSS blockchain gambling betting cryptocurrencies tokenization prediction-markets parimutuel

spacy russian tokenizer

Custom Russian tokenizer for spaCy

✭ 35

python natural-language-processing russian tokenization spacy-pipeline

Multilingual tokenizer that automatically tags each token with its type

✭ 51

javascript multilingual german tokenizer tagging latin french hindi wink devanagari marathi tokenization konkani

🦜 Containerized HTTP API for industrial-strength NLP via spaCy and sense2vec

✭ 58

python shell Dockerfile nlp docker spacy named-entity-recognition sense2vec part-of-speech-tagger tokenization sentence-segmentation

Natural Language Processing Toolkit in Golang

✭ 57

go nlp spacy opencc tokenization normalization lemmatization corenlp

nlp-cheat-sheet-python

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

✭ 69

Jupyter Notebook nlp machine-learning spacy named-entity-recognition nltk starter-kit introduction pos-tagging dependency-parsing tokenization lemmatization cheat-sheet sentence-similarity spans spacy-nlp lexnlp

Factom Asset Tokens - Open tokenization standards on Factom

✭ 17

blockchain tokenization factom

xontrib-output-search

Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.

✭ 26

python cli console terminal command-line tokenizer tokenization xonsh xontrib

[LREC 2022] An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset

✭ 84

python Jupyter Notebook javascript shell perl HTML CSS machine-learning natural-language-processing named-entity-recognition dependency-parser ner pos-tagging tokenization text-annotation lemmatization tweet-analysis twitter-nlp nlp-toolkit

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.

Language Modeling and Text Classification in Malayalam Language using ULMFiT

✭ 68

Jupyter Notebook python text-classification word2vec malayalam language-model fastai tokenization ulmfit

Arabic Tokenization Library. It provides many tokenization algorithms.

✭ 45

Jupyter Notebook python nlp tokenization arabic-nlp tkseem

1-21 of 21 tokenization projects