1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
|
.. default-role:: obj
lz4.stream sub-package
======================
.. warning::
This module is unmaintained.
This sub-package is considered experimental. It was submitted by a community
member who is not able to continue to maintain the module.
This module is not built as part of the distributed wheels. If you wish to
build and use this module you will need to download and build from source
with the environment variable PYLZ4_EXPERIMENTAL set to TRUE.
The module needs some re-write, and the tests need extensive work, for this
to become production ready. If you are interested in working on this, please
reach out to the package maintainers.
This sub-package provides the capability to compress and decompress data using
the `stream specification
<https://github.com/lz4/lz4/blob/master/examples/streaming_api_basics.md>`_,
especially the `stream specification based on a double buffer
<https://github.com/lz4/lz4/blob/master/examples/blockStreaming_doubleBuffer.md>`_.
Because the LZ4 stream format does not define a container format, the
Python bindings will by default insert the compressed data size as an
integer at the start of the compressed payload. However, it is
possible to set the bit depth of this compressed data size.
So far, only the double-buffer based approach is implemented.
Example usage
-------------
To use the lz4 stream format bindings is straightforward:
.. doctest::
>>> from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor
>>> import os
>>> block_size_length = 2 # LZ4 compressed block size stored on 2 bytes
>>> page_size = 8192 # LZ4 context double buffer page size
>>> origin_stream = 10 * 1024 * os.urandom(1024) # 10MiB
>>> # LZ4 stream compression of origin_stream into compressed_stream:
>>> compressed_stream = bytearray()
>>> with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(origin_stream):
... chunk = origin_stream[offset:offset + page_size]
... block = proc.compress(chunk)
... compressed_stream.extend(block)
... offset += page_size
>>> # LZ4 stream decompression of compressed_stream into decompressed_stream:
>>> decompressed_stream = bytearray()
>>> with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(compressed_stream):
... block = proc.get_block(compressed_stream[offset:])
... chunk = proc.decompress(block)
... decompressed_stream.extend(chunk)
... offset += block_size_length + len(block)
>>> decompressed_stream == origin_stream
True
Out-of-band block size record example
-------------------------------------
.. doctest::
>>> from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor
>>> import os
>>> page_size = 8192 # LZ4 context double buffer page size
>>> out_of_band_block_sizes = [] # Store the block sizes
>>> origin_stream = 10 * 1024 * os.urandom(1024) # 10MiB
>>> # LZ4 stream compression of origin_stream into compressed_stream:
>>> compressed_stream = bytearray()
>>> with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=0) as proc:
... offset = 0
... while offset < len(origin_stream):
... chunk = origin_stream[offset:offset + page_size]
... block = proc.compress(chunk)
... out_of_band_block_sizes.append(len(block))
... compressed_stream.extend(block)
... offset += page_size
>>> # LZ4 stream decompression of compressed_stream into decompressed_stream:
>>> decompressed_stream = bytearray()
>>> with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=0) as proc:
... offset = 0
... for block_len in out_of_band_block_sizes:
... # Sanity check:
... if offset >= len(compressed_stream):
... raise LZ4StreamError("Truncated stream")
... block = compressed_stream[offset:offset + block_len]
... chunk = proc.decompress(block)
... decompressed_stream.extend(chunk)
... offset += block_len
>>> decompressed_stream == origin_stream
True
Contents
----------------
.. automodule:: lz4.stream
:members: LZ4StreamCompressor, LZ4StreamDecompressor
|
