All Projects → emmanuel-marty → Lzsa

emmanuel-marty / Lzsa

Licence: other
Byte-aligned, efficient lossless packer that is optimized for fast decompression on 8-bit micros

Programming Languages

c
50402 projects - #5 most used programming language
assembler
53 projects

Projects that are alternatives of or similar to Lzsa

apultra
Free open-source compressor for apLib with 5-7% better ratios
Stars: ✭ 84 (-42.47%)
Mutual labels:  compression, 6502
Durexforth
Modern C64 Forth
Stars: ✭ 139 (-4.79%)
Mutual labels:  6502
Gzip
💾 Golang gzip middleware for Gin and net/http | Golang gzip中间件,支持Gin和net/http,开箱即用同时可定制
Stars: ✭ 113 (-22.6%)
Mutual labels:  compression
Bandwidth Hero Proxy
⚡️ Proxy that compresses images to low-resolution
Stars: ✭ 130 (-10.96%)
Mutual labels:  compression
Razzle Material Ui Styled Example
Razzle Material-UI example with Styled Components using Express with compression
Stars: ✭ 117 (-19.86%)
Mutual labels:  compression
Gtz
A high performance and compression ratio compressor for genomic data, powered by GTXLab of Genetalks.
Stars: ✭ 137 (-6.16%)
Mutual labels:  compression
Optimus
Image conversion and optimization desktop app.
Stars: ✭ 111 (-23.97%)
Mutual labels:  compression
Georaptor
Python Geohash Compression Tool
Stars: ✭ 143 (-2.05%)
Mutual labels:  compression
Zipstorer
A Pure C# Class to Store Files in Zip
Stars: ✭ 139 (-4.79%)
Mutual labels:  compression
Minify
CSS & JavaScript minifier, in PHP. Removes whitespace, strips comments, combines files (incl. @import statements and small assets in CSS files), and optimizes/shortens a few common programming patterns.
Stars: ✭ 1,710 (+1071.23%)
Mutual labels:  compression
Lerc
Limited Error Raster Compression
Stars: ✭ 126 (-13.7%)
Mutual labels:  compression
Model Quantization
Collections of model quantization algorithms
Stars: ✭ 118 (-19.18%)
Mutual labels:  compression
Image Optimize Command
Easily optimize images using WP CLI
Stars: ✭ 138 (-5.48%)
Mutual labels:  compression
Deep K Means Pytorch
[ICML 2018] "Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions"
Stars: ✭ 115 (-21.23%)
Mutual labels:  compression
Plasma
Proto Language AsSeMbler for All (formerly Apple)
Stars: ✭ 142 (-2.74%)
Mutual labels:  6502
Shiftresnet Cifar
ResNet with Shift, Depthwise, or Convolutional Operations for CIFAR-100, CIFAR-10 on PyTorch
Stars: ✭ 112 (-23.29%)
Mutual labels:  compression
Cstore fdw
Columnar storage extension for Postgres built as a foreign data wrapper. Check out https://github.com/citusdata/citus for a modernized columnar storage implementation built as a table access method.
Stars: ✭ 1,642 (+1024.66%)
Mutual labels:  compression
Fsarchiver
file system archiver for linux
Stars: ✭ 135 (-7.53%)
Mutual labels:  compression
Digital video introduction
A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding).
Stars: ✭ 12,184 (+8245.21%)
Mutual labels:  compression
Junrar
Plain Java unrar library
Stars: ✭ 142 (-2.74%)
Mutual labels:  compression

LZSA is a collection of byte-aligned compression formats that are specifically engineered for very fast decompression on 8-bit systems. It can compress files of any size by using blocks of a maximum size of 64 Kb with block-interdependent compression and up to 64 Kb of back-references for matches.

Pareto frontier *ZX Spectrum

Check out The Hollow by Darklite and Offense, winner of the Solskogen 2019 wild compo, that uses LZSA on Z80.

Gabba by Stardust ranked 2nd in the ZX Spectrum demo compo at CAFe demoparty 2019 and also used LZSA on Z80.

Myst Demake for the Apple II by Vince Weaver, uses LZSA on 6502.

The 8 bit guy's Commander X16 ROM uses LZSA on 6502 as well.

RomWBW uses LZSA on Z80 for a variety of hobbyist computers.

The popular rasm assembler for Z80 features LZSA-compressed data sections.

The desolate game port to the ZX Spectrum uses LZSA compression on Z80.

The LZSA compression tool uses an aggressive optimal packing strategy to try to find the sequence of commands that gives the smallest packed file that decompresses to the original while maintaining the maximum possible decompression speed.

The compression formats give the user choices that range from decompressing faster than LZ4 on 8-bit systems with better compression, to compressing as well as ZX7 with much better decompression speed. LZSA1 is designed to replace LZ4 and LZSA2 to replace ZX7, in 8-bit scenarios.

Compression ratio comparison between LZSA and other optimal packers, for a workload composed of ZX Spectrum and C64 files:

                     Bytes            Ratio            Decompression speed vs. LZ4
LZSA2                676681           52,49% <------   75%   
MegaLZ 4.89          679041           52,68%           Not measured
ZX7                  687133           53,30%           47,73%
LZ5 1.4.1            727107           56,40%           75%
LZSA1                735785           57,08% <------   90%
Lizard -29           776122           60,21%           Not measured
LZ4_HC -19 -B4 -BD   781049           60,59%           100%
Uncompressed         1289127          100%             N/A

Performance over well-known compression corpus files:

                     Uncompressed     LZ4_HC -19 -B4 -BD    LZSA1                LZSA2
Canterbury           2810784          935827 (33,29%)       850792 (30,27%)      770877 (27,43%)
Silesia              211938580        77299725 (36,47%)     73706340 (34,78%)    68928564 (32,52%)
Calgary              3251493          1248780 (38,40%)      1192123 (36,67%)     1110290 (34,15%)
Large                11159482         3771025 (33,79%)      3648393 (32,69%)     3519480 (31,54%)
enwik9               1000000000       371841591 (37,18%)    355360043 (35,54%)   334900611 (33,49%)

As an example of LZSA1's simplicity, a size-optimized decompressor on Z80 has been implemented in 67 bytes.

The compressor is approximately 2X slower than LZ4_HC but compresses better while maintaining similar decompression speeds and decompressor simplicity.

The main differences between LZSA1 and the LZ4 compression format are:

  • The use of short (8-bit) match offsets where possible. The match-finder and optimizer cooperate to try and use the shortest match offsets possible.
  • Shorter encoding of lengths. As blocks are maximum 64 Kb in size, lengths can only be up to 64 Kb.
  • As a result of the smaller commands due to the possibly shorter match offsets, a minimum match size of 3 bytes instead of 4. The use of small matches is driven by the optimizer, and used where they provide gains.

As for LZSA2:

  • 5-bit, 9-bit, 13-bit and 16-bit match offsets, using nibble encoding
  • Rep-matches
  • Shorter encoding of lengths, also using nibbles
  • A minmatch of 2 bytes
  • No (slow) bit-packing. LZSA2 uses byte alignment in the hot path, and nibbles.

Inspirations:

  • LZ4 by Yann Collet.
  • LZ5/Lizard by Przemyslaw Skibinski and Yann Collet.
  • The suffix array intervals in Wimlib by Eric Biggers.
  • ZX7 by Einar Saukas
  • apc by Sven-Åke Dahl
  • Charles Bloom's compression blog

License:

  • The LZSA code is available under the Zlib license.
  • The match finder (matchfinder.c) is available under the CC0 license due to using portions of code from Eric Bigger's Wimlib in the suffix array-based matchfinder.

8-bit assembly code:

  • Z80 decompressors (size- and speed-optimized) written by introspec with optimizations by uniabis
  • 6502 and 8088 size-optimized improvements by Peter Ferrie
  • 6502 speed-optimized decompressor by John Brandwood
  • 8088 speed-optimized decompressor by Jim Leonard
  • 6809 decompressors (Tandy CoCo, Thomson MO/TO, Dragon 32/64..) optimized by Doug Masten
  • Hitachi 6309 decompressors (Tandy CoCo 3) also contributed by Doug Masten

External links:

Compressed format

Decompression code is provided for common 8-bit CPUs such as Z80 and 6502. However, if you would like to write your own, or understand the encoding, LZSA compresses data to a format that is fast and simple to decompress on 8-bit CPUs. It is encoded in either a stream of blocks, or as a single raw block, depending on command-line settings. The encoding is deliberately designed to avoid complicated operations on 8-bits (such as 16-bit math).

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].