# lcsync - Librecast Multicast Sync Tool

<a href="https://librecast.net/lcsync.html"><img height="150" align="left" src="https://librecast.net/media/lcsync.svg" alt="lcsync logo"></a>

Librecast file and data syncing tool.

https://librecast.net/lcsync.html

Compare data with merkle trees, sync via multicast.

Work in progress.

## File Syncing

[![Coverity Scan Build Status](https://scan.coverity.com/projects/28355/badge.svg)](https://scan.coverity.com/projects/lcsync)

[![Packaging status](https://repology.org/badge/vertical-allrepos/lcsync.svg)](https://repology.org/project/lcsync/versions)

Data is compared by generating a merkle tree using BLAKE3 hashes.

For local file syncing we walk the trees and compare the hashes to find which
data blocks are different.

To sync remote files, each file is split into blocks and a merkle tree is built
by hashing the blocks using BLAKE3. On the sending/server side, this
tree is sent on Librecast Channel (IPv6 multicast group) that is formed from the
hash of the filename.  The receiver/client joins this channel, and receives the
tree.  If the client already has some data to compare, it builds a merkle tree
of the destination file and uses this to quickly compare which blocks differ. It
builds a bitmap with this information, and then joins the Channel(s) for the
block(s) required which are sent by the server.

Forward Error Correction (FEC) is enabled by default using RaptorQ (RFC 6330)
from the Librecast LCRQ library.

Symmetric encryption is provided using the XSalsa20 stream cipher from libsodium
with Poly1305 MAC authentication tags. A keyfile can be provided, or a key can
be derived from a user-supplied password.

There is no unicast communication with the server. There are no requests sent,
and the server can sit behind a firewall which is completely closed to inbound
TCP and UDP traffic.  Instead, the server listens on a raw socket for Multicast
Listener Discovery (MLD) reports. It compares any MLD multicast group JOINs
against the index it built on startup and finds matches for file (tree) and
blocks. In this way, the server only sends data when at least one client is
subscribed.  If more clients want to download the data, the server need take
no further action.  Thus, the load on the server does not change at all,
regardless of whether there is one client or a billion.

### Bloom Filters and Timers

lcsync uses an experimental form of MLD triggering. Instead of using linked-lists for tracking multicast groups, as the Linux kernel does, we use something more scalable. There can potentially be 2112 multicast groups in IPv6, so beyond a certain point the O(n) search on a linked-list does not scale.

Early versions of lcsync used SIMD (CPU vector operations) to implement counted bloom filters, as well as we are calling a "bloom timer", which lets us track active multicast groups in O(1) constant time. This works, but has the drawback that even for 0 active groups, CPU usage is constant. The size of the bloom filters can be tuned depending on the expected number of simultaneous groups.

Since lcsync v0.1.0 the bloom timer has been replaced with a simple bloom filter which eliminates the need for heavy vector operations, and still provides constant time O(1) searches and updates.

Most of lcsync's functionality was merged into Librecast 0.7.0 with the new Librecast Sync API.

## Building

### Prerequisites:

- [liblibrecast](https://codeberg.org/librecast/librecast) - the Librecast IPv6 multicast library (>= 0.8)
- [libcrq](https://codeberg.org/librecast/lcrq) - the Librecast RaptorQ library
  (>=0.1)
- [libsodium](https://doc.libsodium.org/) - a modern, easy-to-use software library for encryption

### Installation:

NB: GNU Make is required. On \*BSD install and use gmake. The bash shell is also required for `make test`.

```
./configure
make
make test (optional)
make install # requires root privileges
```

## Usage

*lcshare* is the server-side component which indexes files and listens for MLD multicast group joins and sends the file and directory block data.

*lcsync* is the client-side component which joins multicast groups to receive files and directories.

Serve a a single file:
:`lcshare [OPTION...] FILENAME`

Serve all files below a directory:
:`lcshare [OPTION...] DIRECTORY`

Sync remote file(s) with local:
:`lcsync [OPTION...] REMOTEFILENAME LOCALFILENAME`

Sync two local files (Path required. Can be ./):
:`lcsync ./LOCALFILE1 ./LOCALFILE2`

<p>lcsync assumes source and destination are network addresses unless told otherwise.  To refer to a local destination, you must specify the path.  For files in the local directory, prefix them with `./` (a remote source can be forced with `--remote`)</p>

<h3>Options</h3>
<h4>lcshare (server) Options:</h4>

`--loopback`
:enable multicast loopback (sending host will receive sent data)

<h4>lcsync (client) Options:</h4>

`-a, --archive`
:set archive options [-g -o -p -r -t]

`-n, --dry-run`
:don’t copy any data

`-g, --group`
:set group on destination

`-o, --owner`
:set owner on destination

`-p, --perms`
:set permissions on destination

`--remote`
:source path is remote

`-t, --times`
:set modification times on destination


<h4>General Options:</h4>


`-b, --batch`
:Batch mode. No prompting. NB: if you do not specify a keyfile for encryption, encryption will be disabled.

`--bwlimit INTEGER`
:set send rate limit (bps). An SI prefix of T, G, M or K may be added (eg. --bwlimit 10M)

`--hex`
:print file hashes in hex

`-i, --interface INTERFACE`
:set network interface to use (default: all)

`--keyfile KEYFILE`
:Read  symmetric key from keyfile, which must be the path to a file containing a 128 byte
              random key. This can be created with a command like: `dd if=/dev/random of=keyfile count=1 bs=128`

`--loglevel INTEGER`
:set loglevel

`-r, --recursive`
:recurse into directories

`-q, --quiet`
:shhh - we’re hunting wabbits

`-v, --verbose`
:increase verbosity

`-V, --version`
:display version and exit

## Options

-a / --archive
: set archive options [presently only -p]

--hex
: print file hashes in hex

-i / --interface interface
: send/recv on specified interface

--loglevel integer
: set loglevel

-n / --dry-run
: don't copy any data

-p / --perms
: set file permissions on destination

-q / --quiet
: shhh - we're hunting wabbits

-v / --verbose
: increase verbosity

## Testing

`sudo make net-setup` (`sudo make net-teardown` when finished)

```sudo ip netns exec vnet0 sudo -u `id -un` /bin/bash```

Now we can run `make test` and `sudo make cap` in our test namespace.

## License

GPLv2 or (at your option) GPLv3

<hr />

<p class="bigbreak">
This project was funded through the <a href="https://nlnet.nl/discovery">NGI0 Discovery</a> and <a href="https://nlnet.nl/assure">NGI Assure</a> Funds, established by NLnet with financial support from the European
Commission's <a href="https://ngi.eu">Next Generation Internet</a> programme, under the aegis of DG Communications Networks, Content and Technology under grant agreements 825322 and 957073. *Applications are still open, you can <a href="https://nlnet.nl/propose">apply today</a>*
</p>

<p>
  <a href="https://nlnet.nl/project/LibrecastLive/">
      <img width="250" src="https://nlnet.nl/logo/banner.png" alt="Logo NLnet: abstract logo of four people seen from above" class="logocenter" />
  </a>
  &nbsp;
  <a href="https://ngi.eu/">
      <img width="250" src="https://nlnet.nl/image/logos/NGI0_tag.png" alt="Logo NGI Zero: letterlogo shaped like a tag" class="logocenter" />
  </a>
  &nbsp;
  <a href="https://ngi.eu/">
      <img width="250" src="https://nlnet.nl/image/logos/NGIAssure_tag.svg" alt="Logo NGI Assure: letterlogo shaped like a tag" class="logocenter" />
  </a>
</p>