Multi-threaded 7-Zip with Zstandard, Brotli, Lz4, Lz5 and Lizard Compression support
7-Zip-zstd
Great compression performance now available in Windows, Linux and MacOS
https://github.com/mcmilk/7-Zip-zstd
You can also get there the multithreading c library for Brotli, Lizard, LZ4, LZ5 and ZStandard used in it
https://github.com/mcmilk/zstdmt
Python LZ4 lib
I’ve also found a good python multi-threaded lib for LZ4
https://github.com/Iotic-Labs/py-lz4framed
Easy to use
import lz4framed
compressed = lz4framed.compress(b'my normal text')
uncompressed = lz4framed.decompress(compressed)
file compression
with open('somefile.csv', 'wb') as f:
# Context automatically finalises frame on completion, unless an exception occurs
with Compressor(f) as c:
try:
while (...):
c.update(moreData)
except Lz4FramedNoDataError:
pass
file decompression
with open('somefile.csv.lz4', 'rb') as f:
try:
for chunk in Decompressor(f):
decoded.append(chunk)
except Lz4FramedNoDataError:
# Compress frame data incomplete - error case
command line utility
python3 -mlz4framed
USAGE: lz4framed (compress|decompress) (INFILE|-) [OUTFILE]
(De)compresses an lz4 frame. Input is read from INFILE unless set to '-', in
which case stdin is used. If OUTFILE is not specified, output goes to stdout.
A Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO
Tested were:
- Compression ratio
- Compression time
- Decompression time
- Memory requirements on compression
- Memory requirements on decompression
- Time to compress
- Time to decompress
- Memory used
- Compression ratio
can be found here