1136 Open source projects that are alternatives of or similar to Tokenizers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Implementations for a family of attention mechanisms, suitable for all kinds of natural language processing tasks and compatible with TensorFlow 2.0 and Keras.

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.

Label data using HuggingFace's transformers and automatically get a prediction service

classy is a simple-to-use library for building high-performance Machine Learning models in NLP.

A Dead Simple BERT API for Python and Java (https://github.com/google-research/bert)

中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard

Multi-class text categorization using state-of-the-art pre-trained contextualized language models, e.g. BERT

🛸 Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy

[NeurIPS 2021] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Character-based word embeddings model based on RNN for handling real world texts

Mapping a variable-length sentence to a fixed-length vector using BERT model

🔍 Haystack is an open source NLP framework that leverages Transformer models. It enables developers to implement production-ready neural search, question answering, semantic document search and summarization for a wide range of applications.

State of the Art Natural Language Processing

Tutorials on getting started with PyTorch and TorchText for sentiment analysis.

Google AI 2018 BERT pytorch implementation

Code & Data for "Tabular Transformers for Modeling Multivariate Time Series" (ICASSP, 2021)

An NLP system for generating reading comprehension questions

Transformer related optimization, including BERT, GPT

This repository contains code for the O'Reilly Live Online Training for BERT

This repo is the home of Romanian Transformers.

Official implementations for various pre-training models of ERNIE-family, covering topics of Language Understanding & Generation, Multimodal Understanding & Generation, and beyond.

Sequence to sequence learning using TensorFlow.

Anonymization of legal cases (Fr) based on Flair embeddings

Abstractive and Extractive Text summarization using Transformers.

BERT Extension in TensorFlow

[ACL-WS] 4th place solution to gendered pronoun resolution challenge on Kaggle

A repository to run gpt-j-6b on low vram machines (4.2 gb minimum vram for 2000 token context, 3.5 gb for 1000 token context). Model loading takes 12gb free ram.

KB국민은행에서 제공하는 경제/금융 도메인에 특화된 한국어 ALBERT 모델

Build a Movie Reviews Sentiment Classifier with Google's BERT Language Model

Discourse Based Evaluation of Language Understanding

Official Repository for 'Practical Natural Language Processing' by O'Reilly Media

Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled corpus and yields massive improvement: "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval" https://arxiv.org/abs/2112.07577

This repository contains demos I made with the Transformers library by HuggingFace.

A minimal TF2 re-implementation of the OpenAI GPT training

🎨 🎨NLP 自然语言处理教程 🎨🎨 https://dataxujing.github.io/NLP-paper/

Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词等序列标注任务。 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label classification tasks of Chinese long text and short text, and supports sequence annotation tasks such as Chinese named entity recognition, part of speech ta…

API for the GPT-J language model 🦜. Including a FastAPI backend and a streamlit frontend

Official Code for "Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents"

Pytorch Implementation of GoEmotions 😍😢😱

Rank 1 / 216

🛠️ Tools for Transformers compression using PyTorch Lightning ⚡

A PyTorch-based toolkit for natural language processing

Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

An open platform for artificial intelligence, chat bots, virtual agents, social media automation, and live chat automation.

[ACM-WSDM] 3rd place solution at WSDM Cup 2019, Fake News Classification on Kaggle.

Utility for analyzing Transformer based representations of language.

Text Classification in Turkish Texts with Bert

An Open-Source Package for Chinese Open-domain Conversational Chatbot (中文闲聊对话系统，一键部署微信闲聊机器人)

Bangla-Bert is a pretrained bert model for Bengali language

HugsVision is a easy to use huggingface wrapper for state-of-the-art computer vision

Contextualised Embeddings and Language Modelling using BERT and Friends using R

Text2Text: Cross-lingual natural language processing and generation toolkit

Emotion recognition in conversation

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

BlueBERT, pre-trained on PubMed abstracts and clinical notes (MIMIC-III).

🤗 AutoNLP: train state-of-the-art natural language processing models and deploy them in a scalable environment automatically

A curated list of Open Information Extraction (OIE) resources: papers, code, data, etc.

Named Entity Recognition