All Projects → Blosc → C Blosc

Blosc / C Blosc

A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`.

Programming Languages

c
50402 projects - #5 most used programming language

Projects that are alternatives of or similar to C Blosc

lz4-napi
Fastest lz4 compression library in Node.js, powered by napi-rs and lz4-flex.
Stars: ✭ 29 (-96.09%)
Mutual labels:  fast, compression
Dandere2x
Dandere2x - Fast Waifu2x Video Upscaling.
Stars: ✭ 723 (-2.56%)
Mutual labels:  compression, fast
Lizard
Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster.
Stars: ✭ 408 (-45.01%)
Mutual labels:  compression, fast
Httpteleport
Transfer 10Gbps http traffic over 1Gbps networks :)
Stars: ✭ 422 (-43.13%)
Mutual labels:  compression, fast
Ahocorasickdoublearraytrie
An extremely fast implementation of Aho Corasick algorithm based on Double Array Trie.
Stars: ✭ 695 (-6.33%)
Mutual labels:  fast
Caesium Image Compressor
Caesium is a cross-platform image compression software aimed at helping photographers, bloggers, webmasters, businesses or casual users at storing, sending and sharing digital pictures. Based on libcaesium.
Stars: ✭ 595 (-19.81%)
Mutual labels:  compression
Zip
A portable, simple zip library written in C
Stars: ✭ 596 (-19.68%)
Mutual labels:  compression
Bottleneck
Fast NumPy array functions written in C
Stars: ✭ 583 (-21.43%)
Mutual labels:  fast
Tolerant Php Parser
An early-stage PHP parser designed for IDE usage scenarios.
Stars: ✭ 717 (-3.37%)
Mutual labels:  fast
Cr
Runs your tasks at maximum concurrency
Stars: ✭ 681 (-8.22%)
Mutual labels:  fast
Agoo
A High Performance HTTP Server for Ruby
Stars: ✭ 679 (-8.49%)
Mutual labels:  fast
Pbf
A low-level, lightweight protocol buffers implementation in JavaScript.
Stars: ✭ 618 (-16.71%)
Mutual labels:  fast
Wavideobox
秒级! 三行代码实现iOS视频压缩、变速、混音、合并、GIF水印、旋转、换音、裁剪 ! 支持不同分辩率,支持你能想到的各种混合操作! 扩展性强...更多功能不断增加中... iOS 8.0 + 有需要的功能或错误欢迎issue,笔者会及时更新
Stars: ✭ 707 (-4.72%)
Mutual labels:  compression
Fast Android Networking
🚀 A Complete Fast Android Networking Library that also supports HTTP/2 🚀
Stars: ✭ 5,346 (+620.49%)
Mutual labels:  fast
Imglab
To speedup and simplify image labeling/ annotation process with multiple supported formats.
Stars: ✭ 723 (-2.56%)
Mutual labels:  fast
Try
Dead simple CLI tool to try Python packages - It's never been easier! 📦
Stars: ✭ 588 (-20.75%)
Mutual labels:  fast
Acl
Animation Compression Library
Stars: ✭ 716 (-3.5%)
Mutual labels:  compression
Atreugo
High performance and extensible micro web framework. Zero memory allocations in hot paths.
Stars: ✭ 661 (-10.92%)
Mutual labels:  fast
Depth clustering
🚕 Fast and robust clustering of point clouds generated with a Velodyne sensor.
Stars: ✭ 657 (-11.46%)
Mutual labels:  fast
Leanify
lightweight lossless file minifier/optimizer
Stars: ✭ 694 (-6.47%)
Mutual labels:  compression

Blosc: A blocking, shuffling and lossless compression library

Author Contact URL
Blosc Development Team [email protected] http://www.blosc.org
Gitter GH Actions NumFOCUS Code of Conduct
Gitter CI CMake Powered by NumFOCUS Contributor Covenant

What is it?

Blosc is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor (that I'm aware of) that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate memory-bound computations.

It uses the blocking technique so as to reduce activity in the memory bus as much as possible. In short, this technique works by dividing datasets in blocks that are small enough to fit in caches of modern processors and perform compression / decompression there. It also leverages, if available, SIMD instructions (SSE2, AVX2) and multi-threading capabilities of CPUs, in order to accelerate the compression / decompression process to a maximum.

See some benchmarks about Blosc performance.

Blosc is distributed using the BSD license, see LICENSES/BLOSC.txt for details.

Meta-compression and other differences over existing compressors

C-Blosc is not like other compressors: it should rather be called a meta-compressor. This is so because it can use different compressors and filters (programs that generally improve compression ratio). At any rate, it can also be called a compressor because it happens that it already comes with several compressor and filters, so it can actually work like a regular codec.

Currently C-Blosc comes with support of BloscLZ, a compressor heavily based on FastLZ (http://fastlz.org/), LZ4 and LZ4HC (https://github.com/Cyan4973/lz4), Snappy (https://github.com/google/snappy), Zlib (http://www.zlib.net/) and Zstd (http://www.zstd.net).

C-Blosc also comes with highly optimized (they can use SSE2 or AVX2 instructions, if available) shuffle and bitshuffle filters (for info on how and why shuffling works see here). However, additional compressors or filters may be added in the future.

Blosc is in charge of coordinating the different compressor and filters so that they can leverage the blocking technique as well as multi-threaded execution (if several cores are available) automatically. That makes that every codec and filter will work at very high speeds, even if it was not initially designed for doing blocking or multi-threading.

Finally, C-Blosc is specially suited to deal with binary data because it can take advantage of the type size meta-information for improved compression ratio by using the integrated shuffle and bitshuffle filters.

When taken together, all these features set Blosc apart from other compression libraries.

Compiling the Blosc library

Blosc can be built, tested and installed using CMake_. The following procedure describes the "out of source" build.


  $ cd c-blosc
  $ mkdir build
  $ cd build

Now run CMake configuration and optionally specify the installation directory (e.g. '/usr' or '/usr/local'):


  $ cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory ..

CMake allows to configure Blosc in many different ways, like preferring internal or external sources for compressors or enabling/disabling them. Please note that configuration can also be performed using UI tools provided by CMake (ccmake or cmake-gui):


  $ ccmake ..      # run a curses-based interface
  $ cmake-gui ..   # run a graphical interface

Build, test and install Blosc:


  $ cmake --build .
  $ ctest
  $ cmake --build . --target install

The static and dynamic version of the Blosc library, together with header files, will be installed into the specified CMAKE_INSTALL_PREFIX.

Codec support with CMake

C-Blosc comes with full sources for LZ4, LZ4HC, Snappy, Zlib and Zstd and in general, you should not worry about not having (or CMake not finding) the libraries in your system because by default the included sources will be automatically compiled and included in the C-Blosc library. This effectively means that you can be confident in having a complete support for all the codecs in all the Blosc deployments (unless you are explicitly excluding support for some of them).

But in case you want to force Blosc to use external codec libraries instead of the included sources, you can do that:


  $ cmake -DPREFER_EXTERNAL_ZSTD=ON ..

You can also disable support for some compression libraries:


  $ cmake -DDEACTIVATE_SNAPPY=ON ..  # in case you don't have a C++ compiler

Examples

In the examples/ directory you can find hints on how to use Blosc inside your app.

Supported platforms

Blosc is meant to support all platforms where a C89 compliant C compiler can be found. The ones that are mostly tested are Intel (Linux, Mac OSX and Windows) and ARM (Linux), but exotic ones as IBM Blue Gene Q embedded "A2" processor are reported to work too.

Mac OSX troubleshooting

If you run into compilation troubles when using Mac OSX, please make sure that you have installed the command line developer tools. You can always install them with:


  $ xcode-select --install

Wrapper for Python

Blosc has an official wrapper for Python. See:

https://github.com/Blosc/python-blosc

Command line interface and serialization format for Blosc

Blosc can be used from command line by using Bloscpack. See:

https://github.com/Blosc/bloscpack

Filter for HDF5

For those who want to use Blosc as a filter in the HDF5 library, there is a sample implementation in the hdf5-blosc project in:

https://github.com/Blosc/hdf5-blosc

Mailing list

There is an official mailing list for Blosc at:

[email protected] http://groups.google.es/group/blosc

Acknowledgments

See THANKS.rst.


Enjoy data!

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].