GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → levyfan → sentencepiece-jni

levyfan / sentencepiece-jni

Licence: MIT license
Java JNI wrapper for SentencePiece: unsupervised text tokenizer for Neural Network-based text generation.

Programming Languages

C++
36643 projects - #6 most used programming language
java
68154 projects - #9 most used programming language
CMake
9771 projects

Labels

nlpnatural-language-processingjnineural-machine-translationword-segmentationjava-bindingsgoogle-sentencepiece

Projects that are alternatives of or similar to sentencepiece-jni

Libbulletjme
A JNI interface to Bullet Physics and V-HACD
Stars: ✭ 55 (+111.54%)
Mutual labels:  jni, java-bindings
Sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
Stars: ✭ 5,540 (+21207.69%)
Mutual labels:  neural-machine-translation, word-segmentation
DCGCN
Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning (authors' MXNet implementation for the TACL19 paper)
Stars: ✭ 73 (+180.77%)
Mutual labels:  neural-machine-translation
2018-dlsl
UPC Deep Learning for Speech and Language 2018
Stars: ✭ 18 (-30.77%)
Mutual labels:  neural-machine-translation
Android
Swift library for Android
Stars: ✭ 48 (+84.62%)
Mutual labels:  jni
sentencepiece
R package for Byte Pair Encoding / Unigram modelling based on Sentencepiece
Stars: ✭ 22 (-15.38%)
Mutual labels:  word-segmentation
Neural-Machine-Translation
Several basic neural machine translation models implemented by PyTorch & TensorFlow
Stars: ✭ 29 (+11.54%)
Mutual labels:  neural-machine-translation
bytenet translation
A TensorFlow Implementation of Machine Translation In Neural Machine Translation in Linear Time
Stars: ✭ 60 (+130.77%)
Mutual labels:  neural-machine-translation
ffmpeg4java
FFmpeg4Java provides a JNI wrapper of FFmpeg library
Stars: ✭ 21 (-19.23%)
Mutual labels:  jni
TS3000 TheChatBOT
Its a social networking chat-bot trained on Reddit dataset . It supports open bounded queries developed on the concept of Neural Machine Translation. Beware of its being sarcastic just like its creator 😝 BDW it uses Pytorch framework and Python3.
Stars: ✭ 20 (-23.08%)
Mutual labels:  neural-machine-translation
Word-Level-Eng-Mar-NMT
Translating English sentences to Marathi using Neural Machine Translation
Stars: ✭ 37 (+42.31%)
Mutual labels:  neural-machine-translation
clojure-rust-graalvm
An example of Clojure program calling a Rust library, all combined into one executable using GraalVM.
Stars: ✭ 113 (+334.62%)
Mutual labels:  jni
WordSegmentationDP
Word Segmentation with Dynamic Programming
Stars: ✭ 18 (-30.77%)
Mutual labels:  word-segmentation
wgpu-mc
Rust-based replacement for the default Minecraft renderer
Stars: ✭ 254 (+876.92%)
Mutual labels:  jni
spell
Spelling correction and string segmentation written in Go
Stars: ✭ 24 (-7.69%)
Mutual labels:  word-segmentation
parallel-corpora-tools
Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.
Stars: ✭ 35 (+34.62%)
Mutual labels:  neural-machine-translation
bergamot-translator
Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.
Stars: ✭ 181 (+596.15%)
Mutual labels:  neural-machine-translation
ckipnlp
CKIP CoreNLP Toolkits
Stars: ✭ 92 (+253.85%)
Mutual labels:  word-segmentation
skt
Sanskrit compound segmentation using seq2seq model
Stars: ✭ 21 (-19.23%)
Mutual labels:  word-segmentation
MT-Preparation
Machine Translation (MT) Preparation Scripts
Stars: ✭ 15 (-42.31%)
Mutual labels:  neural-machine-translation
View All Similar Projects ➔

SentencePiece Java Wrapper

Build

Java wrapper for SentencePiece with JNI. This module wraps sentencepiece::SentencePieceProcessor class with the following modifications:

  • Encode and Decode methods are re-defined as EncodeAsIds, EncodeAsPieces, DecodeIds and DecodePieces respectively.
  • SentencePieceText proto is not supported.

SentencePiece Version

v0.1.96

Build and Install SentencePiece

To build and install the Java wrapper from source, please try the following commands:

% mvn clean install

Using sentencepiece-jni as a dependency

Because the resulting JAR is platform-dependent, resolving this dependency is managed by the os-maven-plugin. Follow the instructions there to use this platform-dependent JAR.

Please note you need to have a C++ compiler and cmake installed.

Usage

See SentencePieceProcessorTest for more.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].