.. py:currentmodule:: lz4.frame
.. default-role:: obj

Quickstart
==========

Simple usage
------------

The recommended binding to use is the LZ4 frame format binding, since this
provides interoperability with other implementations and language bindings.

The simplest way to use the frame bindings is via the :py:func:`compress` and
:py:func:`decompress` functions:

.. doctest::

  >>> import os
  >>> import lz4.frame
  >>> input_data = 20 * 128 * os.urandom(1024)  # Read 20 * 128kb
  >>> compressed = lz4.frame.compress(input_data)
  >>> decompressed = lz4.frame.decompress(compressed)
  >>> decompressed == input_data
  True

The :py:func:`compress` function reads the input data and compresses it and
returns a LZ4 frame. A frame consists of a header, and a sequence of blocks of
compressed data, and a frame end marker (and optionally a checksum of the
uncompressed data). The :py:func:`decompress` function takes a full LZ4 frame,
decompresses it (and optionally verifies the uncompressed data against the
stored checksum), and returns the uncompressed data.

Working with data in chunks
---------------------------

It's often inconvenient to hold the full data in memory, and so functions are
also provided to compress and decompress data in chunks:

.. doctest::

  >>> import lz4.frame
  >>> import os
  >>> input_data = 20 * 128 * os.urandom(1024)
  >>> c_context = lz4.frame.create_compression_context()
  >>> compressed = lz4.frame.compress_begin(c_context)
  >>> compressed += lz4.frame.compress_chunk(c_context, input_data[:10 * 128 * 1024])
  >>> compressed += lz4.frame.compress_chunk(c_context, input_data[10 * 128 * 1024:])
  >>> compressed += lz4.frame.compress_flush(c_context)

Here a compression context is first created which is used to maintain state
across calls to the LZ4 library. This is an opaque PyCapsule object.
:py:func:`compress_begin` starts a new frame and returns the frame header.
:py:func:`compress_chunk` compresses input data and returns the compressed data.
:py:func:`compress_flush` ends the frame and returns the frame end marker. The
data returned from these functions is catenated to form the compressed frame.

:py:func:`compress_flush` also flushes any buffered data; by default,
:py:func:`compress_chunk` may buffer data until a block is full. This buffering
can be disabled by specifying ``auto_flush=True`` when calling
:py:func:`compress_begin`. Alternatively, the LZ4 buffers can be flushed at any
time without ending the frame by calling :py:func:`compress_flush` with
``end_frame=False``.

Decompressing data can also be done in a chunked fashion:

.. doctest::

  >>> d_context = lz4.frame.create_decompression_context()
  >>> d1, b, e = lz4.frame.decompress_chunk(d_context, compressed[:len(compressed)//2])
  >>> d2, b, e = lz4.frame.decompress_chunk(d_context, compressed[len(compressed)//2:])
  >>> d1 + d2 == input_data
  True

Note that :py:func:`decompress_chunk` returns a tuple ``(decompressed_data,
bytes_read, end_of_frame_indicator)``. ``decompressed_data`` is the decompressed
data, ``bytes_read`` reports the number of bytes read from the compressed input.
``end_of_frame_indicator`` is ``True`` if the end-of-frame marker is encountered
during the decompression, and ``False`` otherwise. If the end-of-frame marker is
encountered in the input, no attempt is made to decompress the data after the
marker.

Rather than managing compression and decompression context objects manually, it
is more convenient to use the :py:class:`LZ4FrameCompressor` and
:py:class:`LZ4FrameDecompressor` classes which provide context manager
functionality:

.. doctest::

  >>> import lz4.frame
  >>> import os
  >>> input_data = 20 * 128 * os.urandom(1024)
  >>> with lz4.frame.LZ4FrameCompressor() as compressor:
  ...     compressed = compressor.begin()
  ...     compressed += compressor.compress(input_data[:10 * 128 * 1024])
  ...     compressed += compressor.compress(input_data[10 * 128 * 1024:])
  ...     compressed += compressor.flush()
  >>> with lz4.frame.LZ4FrameDecompressor() as decompressor:
  ...     decompressed = decompressor.decompress(compressed[:len(compressed)//2])
  ...     decompressed += decompressor.decompress(compressed[len(compressed)//2:])
  >>> decompressed == input_data
  True


Working with compressed files
-----------------------------

The frame bindings provide capability for working with files containing LZ4
frame compressed data. This functionality is intended to be a drop in
replacement for that offered in the Python standard library for bz2, gzip and
LZMA compressed files. The :py:func:`lz4.frame.open()` function is the most
convenient way to work with compressed data files:

.. doctest::

  >>> import lz4.frame
  >>> import os
  >>> input_data = 20 * os.urandom(1024)
  >>> with lz4.frame.open('testfile', mode='wb') as fp:
  ...     bytes_written = fp.write(input_data)
  ...     bytes_written == len(input_data)
  True
  >>> with lz4.frame.open('testfile', mode='r') as fp:
  ...     output_data = fp.read()
  >>> output_data == input_data
  True

The library also provides the class :py:class:`lz4.frame.LZ4FrameFile` for
working with compressed files.


Controlling the compression
---------------------------

Beyond the basic usage described above, there are a number of keyword arguments
to tune and control the compression. A few of the key ones are listed below,
please see the documentation for full details of options.


Controlling the compression level
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``compression_level`` argument specifies the level of compression used with
0 (default) being the lowest compression (0-2 are the same value), and 16 the
highest compression. Values below 0 will enable "fast acceleration",
proportional to the value. Values above 16 will be treated as 16. The following
module constants are provided as a convenience:

- `lz4.frame.COMPRESSIONLEVEL_MIN`: Minimum compression (0, default)
- `lz4.frame.COMPRESSIONLEVEL_MINHC`: Minimum high-compression mode (3)
- `lz4.frame.COMPRESSIONLEVEL_MAX`: Maximum compression (16)

Availability: :py:func:`lz4.frame.compress()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameCompressor`, :py:class:`lz4.frame.LZ4FrameFile`.


Controlling the block size
~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``block_size`` argument specifies the maximum block size to use for the
blocks in a frame. Options:

- `lz4.frame.BLOCKSIZE_DEFAULT` or 0: the lz4 library default
- `lz4.frame.BLOCKSIZE_MAX64KB` or 4: 64 kB
- `lz4.frame.BLOCKSIZE_MAX256KB` or 5: 256 kB
- `lz4.frame.BLOCKSIZE_MAX1MB` or 6: 1 MB
- `lz4.frame.BLOCKSIZE_MAX4MB` or 7: 4 MB

If unspecified, will default to `lz4.frame.BLOCKSIZE_DEFAULT` which is
currently equal to `lz4.frame.BLOCKSIZE_MAX64KB`

Availability: :py:func:`lz4.frame.compress()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameCompressor`, :py:class:`lz4.frame.LZ4FrameFile`.


Controlling block linking
~~~~~~~~~~~~~~~~~~~~~~~~~

The ``block_linked`` argument specifies whether to use block-linked compression.
If ``True``, the compression process will use data between sequential blocks to
improve the compression ratio, particularly for small blocks. The default is
``True``.

Availability: :py:func:`lz4.frame.compress()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameCompressor`, :py:class:`lz4.frame.LZ4FrameFile`.


Data checksum validation
~~~~~~~~~~~~~~~~~~~~~~~~

The ``content_checksum`` argument specifies whether to enable checksumming of
the uncompressed content. If ``True``, a checksum of the uncompressed data is
stored at the end of the frame, and checked during decompression. Default is
``False``.

The ``block_checksum`` argument specifies whether to enable checksumming of the
uncompressed content of each individual block in the frame. If ``True``, a
checksum is stored at the end of each block in the frame, and checked during
decompression. Default is ``False``.

Availability: :py:func:`lz4.frame.compress()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameCompressor`, :py:class:`lz4.frame.LZ4FrameFile`.


Data buffering
~~~~~~~~~~~~~~

The LZ4 library can be set to buffer data internally until a block is filed in
order to optimize compression. The ``auto_flush`` argument specifies whether the
library should buffer input data or not.

When ``auto_flush`` is ``False`` the LZ4 library may buffer data internally. In
this case, the compression functions may return no compressed data when called.
This is the default.

When ``auto_flush`` is ``True``, the compression functions will return
compressed data immediately.

Availability: :py:func:`lz4.frame.compress()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameCompressor`, :py:class:`lz4.frame.LZ4FrameFile`.


Storing the uncompressed source data size in the frame
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``store_size`` and ``source_size`` arguments allow for storing the size of
the uncompressed data in the frame header. Storing the source size in the frame
header adds an extra 8 bytes to the size of the compressed frame, but allows the
decompression functions to better size memory buffers during decompression.

If ``store_size`` is ``True`` the size of the uncompressed data will be stored in
the frame header. Default is ``True``.

Availability of ``store_size``: :py:func:`lz4.frame.compress()`

The ``source_size`` argument optionally specifies the uncompressed size of the
source data to be compressed. If specified, the size will be stored in the frame
header.

Availability of ``source_size``: :py:meth:`lz4.frame.LZ4FrameCompressor.begin()`,
:py:func:`lz4.frame.compress_begin()`, :py:func:`lz4.frame.open()`,
:py:class:`lz4.frame.LZ4FrameFile`.


Working with streamed compressed data
-------------------------------------

The stream bindings provide capability for working with stream compressed LZ4
data. This functionality is based on the usage of a ring-buffer (not implemented
yet) or a double-buffer, with the length of each block preceding the compressed
payload in the stream.

The stream compression reuses a context between each processed block for
performance gain.

Most of the arguments used to initialize the LZ4 stream context are shared with
the block API. Hereafter, those specific to the LZ4 stream API are detailed.


Controlling the buffer size
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``buffer_size`` argument represents the base buffer size used internally for
memory allocation:

* In the case of the double-buffer strategy, this is the size of each buffer of
  the double-buffer.

When compressing, this size is the maximal length of the input uncompressed
chunks.

When decompressing, this size is the maximal length of the decompressed data.


Storing the compressed data size in the block
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``store_comp_size`` argument allows tuning of the size (in bytes) of the
compressed block, which is prepended to the actual LZ4 compressed payload.
This size can be either on ``1``, ``2`` or ``4`` bytes, or ``0`` for out-of-band
block size record.