syrep 0.9

   Copyright 2003-2006 Lennart Poettering <mzflerc (at) 0pointer (dot)
   de>
     * [1]License
     * [2]News
     * [3]Overview
     * [4]Current Status
     * [5]Documentation
     * [6]Requirements
     * [7]Installation
     * [8]Acknowledgements
     * [9]Download

License

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 2 of the License, or (at
   your option) any later version.

   This program is distributed in the hope that it will be useful, but
   WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
   General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

News

   Fri May 19 2006:

   [10]Version 0.9 released; Changes include: ported to Win32/Cygwin
   (patches from Bohdan Futerko); compatibility with Berkeley DB 4.4

   Sat Apr 22 2006:

   [11]Version 0.8 released; Changes include: when resuming a canceled
   merge operation, try harder not to loose any files; when merging and a
   file already exists, and the user chooses not to replace it, don't
   treat this as an error; when updating an existing snapshot locally
   which originates from a different host, update the origin field

   Mon Jul 1 2005:

   [12]Version 0.7 released; Changes include: fix trash cleanup; fix long
   standing file copying bug; don't save device info data in the MD cache
   any longer, use the new option --check-dev to reenable this feature;
   update to Berkeley DB 4.3

   Wed Sep 22 2004:

   [13]Version 0.6 released; Changes include: fix an ugly bug which made
   snapshots where --forget was used unusable; add debian/ directory for
   easily making Debian packages

   Mon Jul 19 2004:

   [14]Version 0.5 released; Changes include: optionally show sizes of
   file on --diff, implement new command --forget, check for extended
   attribute user.syrep on --update on file systems that support it.

   Mon Mar 22 2004:

   [15]Version 0.4 released; Changes include: fix annonoying SIGBUS
   failure when working on files >= 100 MB, update to Berkeley DB 4.2,
   use madvise() to improve file copying throughput on newer kernels,
   minor other fixes

   Sun Nov 30 2003:

   [16]Version 0.3 released; Changes include: new options --sort,
   --check-md, --always-copy; implemented direct bi-directory merges,
   documentation updates, build system updates, assorted fixes.

   Tue Sep 9 2003:

   [17]Version 0.2 released; Fixes include: documentation update, --diff
   output improved, --merge output fixed.

   Mon Sep 8 2003:

   [18]Version 0.1 released.

Overview

   syrep is a generic file repository synchronization tool. It may be
   used to synchronize large file hierarchies bidirectionally by
   exchanging patch files. Syrep is truely peer-to-peer, no central
   servers are involved. Synchronizations between more than two
   repositories are supported. The patch files may be transferred via
   offline media, e.g. removable hard disks or compact discs.

   Files are tracked by their message digests, currently MD5. The
   following file operations are tracked in the snapshot files: creation,
   deletion, modification, creation of new hard or symbolic links,
   renaming. (The latter is nothing more than a new hard link and removal
   of the old file). syrep doesn't distuinguish between soft and hard
   links. In fact even copies of files are treated as the same.
   Currently, syrep doesn't synchronize file attributes like access modes
   or modification times.

   syrep was written to facilitate the synchronization of two large
   digital music repositories without direct network connection. Patch
   files of several gigabytes are common in this situation.

   syrep is able to cope with 64 bit file sizes. (LFS)

   syrep is optimized for speed. It may make use of a message digest
   cache to accelerate the calculation of digests of a whole directory
   hierarchy.

  How does syrep compare with rsync, cvs, Subversion, arch/tla, BitKeeper,
  xdelta2, diff/patch?

   syrep is kind of a bidirectional rsync, but stores and makes use of a
   file hierarchy history. Synchronization with syrep is based on patch
   files, and doesn't require a direct connection between the
   synchronizing peers.

   syrep has many things in common with version control systems like CVS
   or SVN: it stores a history and has operations similar to update and
   commit. However: the history doesn't contain file contents and is not
   line based, it stores the MD5 digest and some meta data only. There is
   no central server, instead all peers have the same role. There is no
   distinction between repositories and checkouts. In fact checkout and
   repository are identical.

   syrep has even more things in common with arch/tla and BitKeeper. All
   three are patch based and are "peer-to-peer". However, there are
   certain differences: syrep doesn't differentiate between repositories
   and checkouts. syrep doesn't keep a file contents history of any kind.

   syrep resembles diff/patch or xdelta2 in some way. While the latter
   work on file contents, syrep works on file hierarchies.

   In contrast two most of the software mentioned above, syrep is capable
   of synchronizing repositories of several 100GB of size, with only a
   very small overhead. (i.e. 4 MB of control data for half a year
   history for 100 GB of user data)

Current Status

   Version 0.9 is more or less stable and fulfills its purpose.

Documentation

   Have a look on the man page [19]syrep(1). (A XSLT capable browser is
   required)

  Method of operation

   Syrep's operation relies on "snapshots" of a file repository. A
   snapshot contains information about all files existent in the
   hierarchy combined with a limited history log of file operations.
   Snapshots may be compared, files missing or deleted on one of both
   sides may be detected this way. Based on this knowledge patch files
   containing all missing files may be created and merged.

   To keep the file operation log in a sensible state it is crucial to
   update the snapshot frequently, probably by adding a new cron job.

  Example usage

   Fred and Karl want to synchronize their digital music libraries by
   exchanging an USB hard disk with patch files. As first step, both
   initialize their repositories for usage with syrep:
fred$ syrep -zp --update ~/mp3/
...
karl$ syrep -zp --update ~/mp3/

   Depending on the size of the repositories this takes a lot of time,
   since a message digest is calculated for every file. Since exact
   tracking of all file operations on the repository is crucial for
   effetive synchronization, they both use the time passing to add a new
   entry to their crontab:
1 3 * * * syrep -z --update ~/mp3/

   When the snapshot creation finished, they send the newly created patch
   files ~/mp3/.syrep/curent.syrep to each other. As these snapshots are
   only about 400K of size for a 80GB repository they do that via email:
fred$ mutt -a ~/mp3/.syrep/current.syrep -s "The current snapshot of fred" karl
...
karl$ mutt -a ~/mp3/.syrep/current.syrep -s "The current snapshot of karl" fred

   When the mails arrive they both detach the snapshot and create a patch
   on their USB harddisk containing all local files not existing on the
   other siede:
fred$ mount /mnt/usb
fred$ syrep -p -o /mnt/usb/patch-for-karl --makepatch ~/mp3/ ~/karls-current.sy
rep
fred$ umount /mnt/usb
...
karl$ mount /mnt/usb
karl$ syrep -p -o /mnt/usb/patch-for-fred --makepatch ~/mp3/ ~/freds-current.sy
rep
karl$ umount /mnt/usb

   As next step they exchange their harddisks. Back at home they merge
   the newly acquired patch into their own repository:
fred$ mount /mnt/usb
fred$ syrep -pT --merge /mnt/usb/patch-for-fred ~/mp3/
fred$ umount /mnt/usb
...
karl$ mount /mnt/usb
karl$ syrep -pT --merge /mnt/usb/patch-for-karl ~/mp3/
karl$ umount /mnt/usb

   At this moment both have the same file hierarchy. To update the local
   snapshot log with the newly merged files they both should run --update
   now. This update run should be much quicker since the message digests
   of all unchanged files are read from a message cache created and
   update each time --update runs:
fred$ syrep -zp --update ~/mp3/
...
karl$ syrep -zp --update ~/mp3/

   Some time later Fred got plenty of new music files, while Karl didn't
   change anything on his repository. Thus, Fred is able to use the old
   snapshot he recieved from Karl to generate a new patch for him. He
   does it exactly the same way he did the last time, see above.

   And now, several iterations of the story described above follow.

   That's the end of the story.

   OK, not quite. Sometimes a conflict happens, e.g. at the same time
   both created a file foo.mp3 with different contents. When this happens
   the local copy is always copied into the patch and the user may decide
   during merge which file version he wants to have locally. Because of
   that merging is an interactive task and cannot be automated
   completely.

   There is no need that the synchronization operations happen in such a
   "symmetric" way as described above.

Requirements

   syrep requires installed development versions of [20]zlib and
   [21]Berkeley DB 4.3. If you want build syrep with support for extended
   attributes (currently supported on Linux only) you have to install
   libattr and a kernel that supports it.

   syrep was developed and tested on Debian GNU/Linux "testing" from
   September 2003, it should work on most other Linux distributions and
   may be POSIX implementations since it uses GNU autoconf for source
   code configuration.

   Some support for for big endian architectures is included, however, it
   is incomplete. You're welcome to send me patches.

   If the syrep build system detects Oliver Kurth's [22]xmltoman the man
   page is rebuilt. Otherwise the pre-compiled versions shipped with
   syrep are used.

Installation

   As this package is made with the GNU autotools you should run
   ./configure inside the distribution directory for configuring the
   source tree. After that you should run make for compilation and make
   install (as root) for installation of syrep.

Acknowledgements

   This software includes an implementation of the MD5 algorithm by L.
   Peter Deutsch. Thanks to him for this.

   Bohdan Futerko for porting syrep to Win32/Cygwin.

Download

   The newest release is always available from
   [23]http://0pointer.de/lennart/projects/syrep/

   The current release is [24]0.9.

   [25]Win32/x86 binaries courtesy of Bohdan Futerko

   Get syrep's development sources from the [26]Subversion [27]repository
   ([28]viewcvs):
svn checkout svn://svn.0pointer.net/syrep/trunk syrep

   You may find an up-to-date Debian package (i386) of syrep in my
   [29]local Debian package repository.

   If you want to be notified whenever I release a new version of this
   software use the subscription feature of [30]Freshmeat.
     _________________________________________________________________


    Lennart Poettering <mzflerc (at) 0pointer (dot) de>, May 2006

   $Id: README.html.in 113 2006-05-19 10:09:00Z lennart $

References

   1. README#license
   2. README#news
   3. README#overview
   4. README#status
   5. README#documentation
   6. README#requirements
   7. README#installation
   8. README#acks
   9. README#download
  10. http://0pointer.de/lennart/projects/syrep/syrep-0.9.tar.gz
  11. http://0pointer.de/lennart/projects/syrep/syrep-0.8.tar.gz
  12. http://0pointer.de/lennart/projects/syrep/syrep-0.7.tar.gz
  13. http://0pointer.de/lennart/projects/syrep/syrep-0.6.tar.gz
  14. http://0pointer.de/lennart/projects/syrep/syrep-0.5.tar.gz
  15. http://0pointer.de/lennart/projects/syrep/syrep-0.4.tar.gz
  16. http://0pointer.de/lennart/projects/syrep/syrep-0.3.tar.gz
  17. http://0pointer.de/lennart/projects/syrep/syrep-0.2.tar.gz
  18. http://0pointer.de/lennart/projects/syrep/syrep-0.1.tar.gz
  19. http://0pointer.de/lennart/projects/syrep/syrep.1.xml
  20. http://www.gzip.org/zlib/
  21. http://www.sleepycat.com/
  22. http://masqmail.cx/xml2man/
  23. http://0pointer.de/lennart/projects/syrep/
  24. http://0pointer.de/lennart/projects/syrep/syrep-0.9.tar.gz
  25. http://0pointer.de/lennart/projects/syrep/syrep-0.9.zip
  26. http://subversion.tigris.org/
  27. svn://svn.0pointer.net/syrep
  28. http://0pointer.net/cgi-bin/viewcvs.cgi/?root=syrep
  29. http://0pointer.de/debian/
  30. http://freshmeat.net/projects/syrep/