lemire / Maskedvbyte
Licence: apache-2.0
Fast decoder for VByte-compressed integers
Stars: ✭ 91
Programming Languages
c
50402 projects - #5 most used programming language
Labels
Projects that are alternatives of or similar to Maskedvbyte
ndzip
A High-Throughput Parallel Lossless Compressor for Scientific Data
Stars: ✭ 19 (-79.12%)
Mutual labels: compression, simd
Streamvbyte
Fast integer compression in C using the StreamVByte codec
Stars: ✭ 195 (+114.29%)
Mutual labels: simd, compression
Turbo-Transpose
Transpose: SIMD Integer+Floating Point Compression Filter
Stars: ✭ 50 (-45.05%)
Mutual labels: compression, simd
Simdcompressionandintersection
A C++ library to compress and intersect sorted lists of integers using SIMD instructions
Stars: ✭ 289 (+217.58%)
Mutual labels: simd, compression
Simdcomp
A simple C library for compressing lists of integers using binary packing
Stars: ✭ 331 (+263.74%)
Mutual labels: simd, compression
Turbopfor Integer Compression
Fastest Integer Compression
Stars: ✭ 520 (+471.43%)
Mutual labels: simd, compression
Compressed Vec
SIMD Floating point and integer compressed vector library
Stars: ✭ 25 (-72.53%)
Mutual labels: simd, compression
Deprecated Lame Mirror
[DEPRECATED] Old, Semi-official mirror of the CVS repository of the LAME MP3 encoder.
Stars: ✭ 73 (-19.78%)
Mutual labels: compression
Gifcompressor
An Android tool to compresses your GIFs into lightweight MP4 video using fast, hardware-accelerated encoders. Supports cropping, rotation, GIF concatenation and much more.
Stars: ✭ 85 (-6.59%)
Mutual labels: compression
Wide
A crate to help you go wide. By which I mean use SIMD stuff.
Stars: ✭ 72 (-20.88%)
Mutual labels: simd
Simd
C++ image processing and machine learning library with using of SIMD: SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2, AVX-512, VMX(Altivec) and VSX(Power7), NEON for ARM.
Stars: ✭ 1,263 (+1287.91%)
Mutual labels: simd
Html Compress Twig
Twig extension for compressing HTML and inline CSS/JS using WyriHaximus/HtmlCompress
Stars: ✭ 72 (-20.88%)
Mutual labels: compression
Zippy
Pure Nim implementation of deflate, zlib, gzip and zip.
Stars: ✭ 88 (-3.3%)
Mutual labels: compression
Stdpack.c
Collection of small public domain de/compressors in plain C.
Stars: ✭ 73 (-19.78%)
Mutual labels: compression
Redoflacs
Parallel BASH commandline FLAC compressor, verifier, organizer, analyzer, and retagger
Stars: ✭ 71 (-21.98%)
Mutual labels: compression
Despacer
C library to remove white space from strings as fast as possible
Stars: ✭ 90 (-1.1%)
Mutual labels: simd
Include Flate
A variant of include_bytes!/include_str! with compile-time deflation and runtime lazy inflation
Stars: ✭ 87 (-4.4%)
Mutual labels: compression
Fossildelta
An efficient delta compression algorithm written in C#
Stars: ✭ 82 (-9.89%)
Mutual labels: compression
MaskedVByte
Fast decoder for VByte-compressed integers in C.
It includes fast differential coding.
It assumes a recent Intel processor (e.g., haswell) but should work with most x64 processors (supporting SSE instruction sets).
The code should build using most standard-compliant modern C compilers (C99). The provided makefile expects a Linux-like system.
Usage:
make
./unit
See example.c for an example.
Short code sample:
size_t compsize = vbyte_encode(datain, N, compressedbuffer); // encoding
// here the result is stored in compressedbuffer using compsize bytes
size_t compsize2 = masked_vbyte_decode(compressedbuffer, recovdata, N); // decoding (fast)
Interesting applications
Greg Bowyer has integrated Masked VByte into Lucene, for higher speeds :
https://github.com/GregBowyer/lucene-solr/tree/intrinsics
Reference
- Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130, February 2018, Pages 1-6 https://arxiv.org/abs/1709.08990
- Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387
See also
- SIMDCompressionAndIntersection: A C++ library to compress and intersect sorted lists of integers using SIMD instructions https://github.com/lemire/SIMDCompressionAndIntersection
- The FastPFOR C++ library : Fast integer compression https://github.com/lemire/FastPFor
- High-performance dictionary coding https://github.com/lemire/dictionary
- LittleIntPacker: C library to pack and unpack short arrays of integers as fast as possible https://github.com/lemire/LittleIntPacker
- The SIMDComp library: A simple C library for compressing lists of integers using binary packing https://github.com/lemire/simdcomp
- StreamVByte: Fast integer compression in C using the StreamVByte codec https://github.com/lemire/streamvbyte
- CSharpFastPFOR: A C# integer compression library https://github.com/Genbox/CSharpFastPFOR
- JavaFastPFOR: A java integer compression library https://github.com/lemire/JavaFastPFOR
- Encoding: Integer Compression Libraries for Go https://github.com/zhenjl/encoding
- FrameOfReference is a C++ library dedicated to frame-of-reference (FOR) compression: https://github.com/lemire/FrameOfReference
- libvbyte: A fast implementation for varbyte 32bit/64bit integer compression https://github.com/cruppstahl/libvbyte
- TurboPFor is a C library that offers lots of interesting optimizations. Well worth checking! (GPL license) https://github.com/powturbo/TurboPFor
- Oroch is a C++ library that offers a usable API (MIT license) https://github.com/ademakov/Oroch
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].