# Development Guide for rdiff-backup

> Suggest [improvements to this documentation](https://github.com/rdiff-backup/rdiff-backup/issues/new?title=Docs%20feedback:%20/docs/DEVELOP.md)!


## GETTING THE SOURCE

Simply clone the source with:

	git clone https://github.com/rdiff-backup/rdiff-backup.git

> **NOTE:** If you plan to provide your own code, you should first fork
our repo and clone your own forked repo (probably using ssh not https).
How is described at
https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/working-with-forks


## GENERAL GUIDELINES

- Before committing to a lot of writing or coding, please file an issue on Github and discuss your plans and gather feedback. Eventually it will be much easier to merge your change request if the idea and design has been agreed upon, and there will be less work for you as a contributor if you implement your idea along the correct lines to begin with.
- Please check out [existing issues](https://github.com/rdiff-backup/rdiff-backup/issues) and [existing merge requests](https://github.com/rdiff-backup/rdiff-backup/pulls) and browse the [git history](https://github.com/rdiff-backup/rdiff-backup/commits/master) to see if somebody already tried to address the thing you have are interested in. It might provide useful insight why the current state is as it is.
- Changes can be submitted using the typical Github workflow: clone this repository, make your changes, test and verify, and submit a Pull Request (PR).
- For all code changes, please remember also to include inline comments and
  update tests where needed.

### License

Rdiff-backup is licensed with GNU General Public License v2.0 or later. By
contributing to this repository you agree that your work is licensed using the
chosen project license.

### Branching model and pull requests

The *master* branch is always kept in a clean state. Anybody can at any time
clone this repository and branch off from *master* and expect test suite to pass
and the code and other contents to be of good quality and a reasonable
foundation for them to continue development on.

Each PR focuses on some topic and resist changing anything else. Keeping the
scope clear also makes it easier to review the pull request. A good pull
request has only one or a few commits, with each commit having a good commit
subject and if needed also a body that explains the change.

Each pull request has only one author, but anybody can give feedback. The
original author should be given time to address the feedback – reviewers should
not do the fixes for the author, but instead let the author keep the authorship.
Things can always be iterated and extended in future commits once the PR has
been merged, or even in parallel if the changes are in different files or at
least on different lines and do not cause merge conflicts if worked on.

It is the responsibility of the PR author to keep it without conflict with
master (e.g. if not quickly merged) and overall to support the review process.

Ideally each pull request gets some feedback within 24 hours from it having been
filed, and is merged within days or a couple of weeks. Each author should
facilitate quick reviews and merges by making clean and neat commits and pull
requests that are quick to review and do not spiral out in long discussions.

If something is of interest for the changelog, prefix the statement in the
commit body with a three uppercase letters and a column; which acronym is
not that important but here is a list of recommended ones (see the release
section to understand why it's important):

* FIX: for a bug fix
* NEW: for a new feature
* CHG: for a change requesting consideration when upgrading
* DOC: for documentation aspects
* WEB: anything regarding the website

#### Merging changes to master

Currently the rdiff-backup Github repository is configured so that merging a
pull request is possible only if it:
- passes the CI testing
- has at least one approving review

While anybody can make forks, pull requests and comment them, only a developer
with write access to the main repository can merge and land commits in the
master branch. To get write access, the person mush exhibit commitment to high
standards and have a track record of meaningful contributions over several
months.

It is the responsibility of the merging developer to make sure that the PR
is _squashed_ and that the squash commit message helps the release process
with the right description and 3-capital-letters prefix (it is still the
obligation of the PR author to provide enough information in their commit
messages).

### Coding style

This project is written in Python, and must follow the official [PEP 8 coding
standard](https://www.python.org/dev/peps/pep-0008/) as enforced via the CI
system.

### Versioning

In versioning we utilize git tags as understood by
[setuptools_scm](https://github.com/pypa/setuptools_scm/#default-versioning-scheme).
Version strings follow the [PEP-440
standard](https://www.python.org/dev/peps/pep-0440/).

The rules are currently as follows (check the `.travis.yml` file for details):

- all commits tagged with an underscore at the end or with a tag looking like a
  version number (i.e. as in next two bullets) are released to
  [GitHub](https://github.com/rdiff-backup/rdiff-backup/releases/).
- all commits tagged with alpha, beta, rc or final format are released to
  [PyPI](https://pypi.org/project/rdiff-backup/#history), i.e. the ones looking
  like: vX.Y.ZaN (alpha), vX.Y.ZbN (beta), vX.Y.ZrcN (release candidate) or
  vX.Y.Z (final).
- all commits where the "version tag" is a development one, i.e. like previously
  with an additional `.devM` at the end, are released to 
  [Test PyPI](https://test.pypi.org/project/rdiff-backup/#history).
  They are meant mostly to test the deployment itself (use alpha versions to
  release development code).

> **NOTE:** the GitHub releases are created as draft, meaning that a maintainer
>	must review them and publish them before they become visible.

> **CAUTION:** due to a bug in Travis CI, the Windows wheel can't currently be
>	published to PyPI and needs to be downloaded from GitHub and manually
>	uploaded to PyPI.

## Releases

There is no prior release schedule – they are made when deemed fit.


## BUILD AND INSTALL

### Pre-requisites

The same pre-requisites as for the installation of rdiff-backup also apply for building:

* Python 3.5 or higher
* librsync 1.0.0 or higher
* pylibacl (optional, to support ACLs)
* pyxattr (optional, to support extended attributes) - even if the xattr library (without py) isn't part of our CI/CD pipeline, feel free to use it for your development
* python3-setuptools (for a proper version instead of DEV)

Additionally are following pre-requisites needed:

* python3-dev (or -devel)
* librsync-dev (or -devel)
* a C compiler (gcc)
* python3-setuptools (for setup.py)
* setuptools-scm (also for setup.py, to gather all source files in sdist)
* libacl-devel (for sys/acl.h)
* tox (for testing)
* rdiff (for testing)
* flake8 (optional, but helpful to validate code correctness locally)
* coverage (optional, but helpful to validate test coverage locally)

All of those should come packaged with your system or available from
https://pypi.org/ but if you need them otherwise, here are some sources:

* Python - https://www.python.org/
* Librsync - http://librsync.sourceforge.net/
* Pywin32 - https://github.com/mhammond/pywin32
* Pylibacl - http://pylibacl.sourceforge.net/
* Pyxattr - http://pyxattr.sourceforge.net/

### Build and install using Makefile

The project has a [Makefile](../Makefile) that defines steps like `all`,
`build`, `test` and others. You can view the contents to see what it exactly
does. Using the `Makefile` is the easiest way to quickly build and test the
source code.

By default the `Makefile` runs all of it's command in a clean Docker container,
thus making sure all the build dependencies are correctly defined and also
protecting the host system from having to install them.

The [Travis-CI](https://travis-ci.org/rdiff-backup/rdiff-backup) integration
also uses the `Makefile`, so if all commands in the `Makefile` succeed locally,
the CI is most likely to pass as well.

### Build and install with setup.py

To install, simply run:

	python3 setup.py install

The build process can be also be run separately:

	python3 setup.py build

The setup script expects to find librsync headers and libraries in the
default location, usually /usr/include and /usr/lib.  If you want the
setup script to check different locations, use the --librsync-dir
switch or the LIBRSYNC_DIR environment variable.  For instance to instruct
the setup program to look in `/usr/local/include` and `/usr/local/lib`
for the librsync files run:

	python3 setup.py --librsync-dir=/usr/local build

Finally, the `--lflags` and `--libs` options, and the `LFLAGS` and `LIBS`
environment variables are also recognized.  Running setup.py with no
arguments will display some help. Additional help is displayed by the
command:

	python3 setup.py install --help

More information about using setup.py and how rdiff-backup is installed
is available from the Python guide, Installing Python Modules for System
Administrators, located at https://docs.python.org/3/install/index.html

> **NOTE:** There is no uninstall command provided by the Python
distutils/setuptools system. One strategy is to use the `python3 setup.py
install --record <file>` option to save a list of the files installed to `<file>`,
another is to created a wheel package with `python3 setup.py bdist_wheel`,
as it can be installed and deinstalled.

> **NOTE:** if you plan to use `./setup.py bdist_rpm` to create an RPM, you
> would need rpm-build but be aware that it will currently fail due to a [known
> bug in setuptools with compressed man
> pages](https://github.com/pypa/setuptools/issues/1277).

To build from source on Windows, check the [Windows tools](../tools/windows)
to build a single executable file which contains Python, librsync, and
all required modules.


## TESTING

Clone, unpack and prepare the testfiles by calling the script `tools/setup-testfiles.sh`
from the cloned source Git repo. You will most probably be asked for your password
so that sudo can extract and prepare the testfiles (else the tests will fail).

That's it, you can now run the tests:

* run `tox` to use the default `tox.ini`
* or `tox -c tox_slow.ini` for long tests
* or `sudo tox -c tox_root.ini` for the few tests needing root rights

For more details on testing, see the `test` sections in the [Makefile](../Makefile)
and the [.travis-ci.yml definitions](../.travis-ci.yml).

## DEBUGGING

### Trace back a coredump

At the time of writing these notes, there was an issue where calling the program
generates a `Segmentation fault (core dumped)`. This chapter is based on this
experience debugging under Fedora 29.

References:

* https://ask.fedoraproject.org/en/question/98776/where-is-core-dump-located/
* Adventures in Python core dumping: https://gist.github.com/toolness/d56c1aab317377d5d17a
* Debugging dynamically loaded extensions: https://www.oreilly.com/library/view/python-cookbook/0596001673/ch16s08.html
* Debugging Memory Problems: https://www.oreilly.com/library/view/python-cookbook/0596001673/ch16s09.html

> **NOTE:** This assumes gdb was already installed.

1. First install:

		sudo dnf install python3-debug
		sudo dnf debuginfo-install python3-debug-3.7.3-1.fc29.x86_64
		sudo dnf debuginfo-install bzip2-libs-1.0.6-28.fc29.x86_64 glibc-2.28-27.fc29.x86_64 \
			librsync-1.0.0-8.fc29.x86_64 libxcrypt-4.4.4-2.fc29.x86_64 \
			openssl-libs-1.1.1b-3.fc29.x86_64 popt-1.16-15.fc29.x86_64 \
			sssd-client-2.1.0-2.fc29.x86_64 xz-libs-5.2.4-3.fc29.x86_64 zlib-1.2.11-14.fc29.x86_64

2. Then run:

		python3 ./setup.py clean --all
		python3-debug ./setup.py clean --all
		CFLAGS='-Wall -O0 -g' python3-debug ./setup.py build
		PATH=$PWD/build/scripts-3.7:$PATH PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7-pydebug/ rdiff-backup -v 10 \
			/some/dir1 /some/dir2
		[...]
		Segmentation fault (core dumped)

> **NOTE:** The CFLAGS avoids optimizations making debugging too complicated

At this stage `coredumpctl list` shows that coredump is the last one, so that
one can call `coredumpctl gdb`, which itself tells (in multiple steps) that we
missing some more debug information, hence the above `debuginfo-install`
statements (assuming guess you could install the packages without version
information if you're sure they fit the installed package versions).

So now back into `coredumpctl gdb`, with some commands:

	help
	help stack
	backtrace
	bt full
	py-bt
	frame <FrameNumber>
	p <SomeVar>

1. get a backtrace of all function calls leading to the coredump (also `bt`)
2. backtrace with local vars
3. py-bt is the Python version of backtrace
4. jump between frames as listed by bt using their `#FrameNumber`
5. print some variable/expression in the context of the selected frame

Jumping between frames and printing the different variables, we can recognize that:

1. the core dump is due to a seek on a null file pointer
2. that the file pointer comes from the job pointer handed over to the function rs_job_iter
3. the job pointer itself comes from the self variable handed over to `_librsync_patchmaker_cycle`
4. reading through the https://librsync.github.io/rdiff.html[librsync documentation], it appears that the job type is opaque, i.e. I can't directly influence and it has been created via the `rs_patch_begin` function within the function `_librsync_new_patchmaker` in `rdiff_backup/_librsyncmodule.c`.

At this stage, it seems that the core file has given most of its secrets and we need to debug the live program:

	$ PYTHONTRACEMALLOC=1 PATH=$PWD/build/scripts-3.7:$PATH PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7-pydebug/ gdb python3-debug
	(gdb) break rdiff_backup/_librsyncmodule.c:_librsync_new_patchmaker
	(gdb) run build/scripts-3.7/rdiff-backup /some/source/dir /some/target/dir

The debugger runs until the breakpoint is reached, after which a succession of `next` and `print <SomeVar>` allows me to analyze the code step by step, and to come to the conclusion that `cfile = fdopen(python_fd, ...` is somehow wrong as it creates a null file pointer whereas `python_fd` looks like a valid file descriptor (an integer equal to 5).

### ResourceWarning unclosed file

If you get something looking like a `ResourceWarning: Enable tracemalloc to get the object allocation traceback`

	PYTHONTRACEMALLOC=1 PATH=$PWD/build/scripts-3.7:$PATH \
	PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7-pydebug/ \
		rdiff-backup -v 10 /tmp/äłtèr /var/tmp/rdiff

This tells you indeed where the file was opened: `Object allocated at (most recent call last)` but it still requires deeper analysis to understand the reason.

> **Reference:** https://docs.python.org/3/library/tracemalloc.html

### Debug client / server mode

In order to make sure the debug messages are properly sorted, you need to have the verbosity
level 9 set-up, mix stdout and stderr, and then use the date/time output to properly sort
the lines coming both from server and client, while making sure that lines belonging together
stay together. The result command line might look as follows:

	rdiff-backup -v9 localhost::/sourcedir /backupdir 2>&1 | awk \
		'/^2019-09-16/ { if (line) print line; line = $0 } ! /^2019-09-16/ { line = line " ## " $0 }' \
		| sort | sed 's/ ## /\n/g'

### Debug iterators

When debugging, the fact that rdiff-backup uses a lot of iterators makes it
rather complex to understand what's happening. It would sometimes make it
easier to have a list to study at once of iterating painfully through each
_but_ if you simply use `p list(some_iter_var)`, you basically run through
the iterator and it's lost for the program, which can only fail.

The solution is to use `itertools.tee`, create a copy of the iterator and
print the copy, e.g.:

```
(Pdb) import itertools
(Pdb) inc_pair_iter,mycopy = itertools.tee(inc_pair_iter)
(Pdb) p list(map(lambda x: [str(x[0]),list(map(str,x[1]))], mycopy))
[... whatever output ...]
```

Assuming the iteration has no side effects, the initial variable `inc_pair_iter`
is still valid for the rest of the program, whereas the `mycopy` is "dried out"
(but you can repeat the `tee` operation as often as you want).

### Profile rdiff-backup

#### Profiling without code changes

After having called `./setup.py build`, you may call something like the following
to profile the current code (adapt to your Python version):

```
PATH=$PWD/build/scripts-3.8:$PATH PYTHONPATH=$PWD/build/lib.linux-x86_64-3.8 \
	python -m cProfile -s tottime \
	build/scripts-3.8/rdiff-backup [... rdiff-backup parameters ...]
```

The `-s tottime` option _sorts_ by total time spent in the function.
More information can be found in the
[profile documentation](https://docs.python.org/3/library/profile.html).

> **TIP:** if you're into graphical tools and overviews, have a look e.g.
	at  https://pythonhosted.org//ProfileEye/ ?

You may also do memory profiling using the
[memory-profiler](https://pypi.org/project/memory-profiler/),
though more detailed information requires changes to the code by adding
the `@profile` decorator to functions:

```
pip install --user memory-profiler
PATH=$PWD/build/scripts-3.8:$PATH PYTHONPATH=$PWD/build/lib.linux-x86_64-3.8 \
	mprof run \
	build/scripts-3.8/rdiff-backup [... rdiff-backup parameters ...]
mprof plot
mprof clean
```

> **NOTE:** sometimes calling rdiff-backup this way fails, it's due to the
	script having a wrong interpreter (because of wheel building).
	Call `./setup.sh clean --all && ./setup.py build` to fix it.

> **TIP:** there is also a
	[line-profiler](https://pypi.org/project/line-profiler/),
	but I didn't try it because it requires changes to the code
	(again the `@profile` decorator).

#### More profiling with code changes

Once you have found by profiling an object that uses a lot of memory,
one can use `print(sys.getsizeof(x))` to print it's memory footprint then iterating for a code solution to bring it down.

Memory can be freed manually with:

```
import gc
collected_objects = gc.collect()
```

This can also be run in Python:

```
import cProfile, pstats, StringIO 
pr = cProfile.Profile() 
pr.enable() 
# ... do something ... pr.disable() 
s = StringIO.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats(‘cumulative’) 
ps.print_stats() 
print s.getvalue()
```

## RELEASING

We use [Travis CI](https://travis-ci.org) to release automatically, as setup in the [Travis configuration file](../.travis.yml).

The following rules apply:

* each modification to master happens through a Pull Request (PR) which triggers
  a pipeline job, which must be succesful for the merge to have a chance to
  happen. Such PR jobs will _not_ trigger a release.
* GitHub releases are generated as draft only on Git tags looking like a release.
  The release manager reviews then the draft release, names and describes it
  before they makes it visible. An automated Pypi release is foreseen but not
  yet implemented.
* If you need to trigger a job for test purposes (e.g. because you changed
  something to the pipeline), create a branch or a tag with an underscore at
  the end of their name. Just make sure that you remove such tags, and
  potential draft releases, after usage.
* If you want, again for test purposes, to trigger a PyPI deployment towards
  test.pypi.org, tag the commit before you push it with a development release
  tag, like `vA.B.CbD.devN`, then explicitly push the tag and the branch at
  the same time e.g. with `git push origin vA.B.CbD.devN myname-mybranch`.

> **TIP:** Travis will not trigger again on a commit which has already gone
           through the pipeline, even if you add a tag. This applies especially
           to PR commits merged to master without squashing.

Given the above rules, a release cycle looks roughly as follows:

1. Call `./tools/get_changelog_since.sh PREVIOUSTAG` to get a list of changes
   (see above) since the last release and a sorted and unique list of authors,
   on which basis you can extend the [CHANGELOG](../CHANGELOG.md) for the
   new release.
   **IMPORTANT:** make sure that the PR is squashed or you won't be able to
   trigger the release pipeline via a tag on master.
2. Make sure you have the latest master commits with
   `git checkout master && git pull --prune`.
3. Tag the last commit with `git tag vX.Y.ZbN` (beta) or `git tag vX.y.Z" (stable).
4. Push the tag to GitHub with `git push --tags`.
5. You won't see anything in GitHub at first and need to go directly to
   [Travis builds](https://travis-ci.org/rdiff-backup/rdiff-backup/builds) to
   verify that the pipeline has started.
6. If everything goes well, you should see the
   [new draft release](https://github.com/rdiff-backup/rdiff-backup/releases)
   with all assets (aka packages) attached to it after all jobs have finished
   in Travis.
7. Give the release a title and description and save it to make it visible to
   everybody.
8. You'll get a notification e-mail telling you that rdiff-backup-admin has
   released a new version.
9. Use this e-mail to inform the [rdiff-backup users](rdiff-backup-users@nongnu.org).

> **IMPORTANT:** if not everything goes well, remove the tag both locally with
                 `git tag -d TAG` and remotely with `git push -d origin TAG`.
                 Then fix the issue with a new PR and start from the beginning.

> **TIP:** the PyPI deploy pipeline is for now broken under Windows on Travis-CI.
           You may download the Windows wheel(s) from GitHub and upload them to
           PyPI from the command line using twine:
           `twine upload [--repository-url https://test.pypi.org/legacy/] dist/rdiff\_backup-*-win32.whl`

The following sub-chapters list some learnings and specifities in case you need to modify the pipeline.

### Install the Travis client locally

See https://github.com/travis-ci/travis.rb for details, here only the gist of it:

```
ruby -v               # version >= 2
dnf install rubygems  # or zipper, apt, yum...
gem install travis    # as non-root keeps everybody more happy
travis version        # 1.8.10 -> all OK
```

> **NOTE:** installing travis gem also pulls the dependencies multipart-post, faraday, faraday_middleware, highline, backports, net-http-pipeline, net-http-persistent, addressable, multi_json, gh, launchy, ethon, typhoeus, websocket, pusher-client. You might want to install some of them via your preferred package manager instead.

### Create an OAuth key

Use the travis client to generate a secure API key (you can throw away other changes to the `.travis.yml` file). You will need the password of the rdiff-backup-admin, hence only project admins can generate it:

```
$ travis setup releases
Detected repository as rdiff-backup/rdiff-backup, is this correct? |yes| 
Username: rdiff-backup-admin
Password for rdiff-backup-admin: ********************
File to Upload: dist/*
Deploy only from rdiff-backup/rdiff-backup? |yes| 
Encrypt API key? |yes| 
```

The key to add looks then as follows for GitHub deployment (the concrete key shown here isn't valid though):

```
deploy:
  provider: releases
  api_key:
    secure: lqg+HZoy68WudiogbEnOmhxfw9zEJhPOyM4bLJdU2lRBlUZbf0uFvpVJdJqPB7rovKpDknapg4xdXdpbLbD0r/PwsSI9UyFLmyhGn24pnSlrFFjFm2AIQQJUMiCcqsPqNc7fXNMC1BwuM1/RjO3hIxfPxI+A9MSVqW3qhzmerOKXeKFiOLXJ0FkTomRdWGhCEafWO1Ibz5O2d5psK1N/r1ni8kv+E6GPjHk54vmKNcFg8uB7+cPs7ONtW2F+M/h12UVZkC+hy8Bss+esQIMYdVLW5JkKSFfNwKs57qDYYd0lWLzMRti+S+0k/1O6l51BzLY61C4FlRwrMWAy4HIYn5ui39GXIYtGXq9zW+EpYvqTsar+KDU+DGzsr+hAt+eCQpbmZ2SpA7B8Mb3x+BwAcEkvCql789FhWCOd3arUm3H6Ng6yNt50crafJeboHhmitgFQ9uTM7AnXwMnIYVkl6IAZlPkIj20TF1JSdmzpPG2jEJATsMybCuaAuS+ngq4DnJ1axGcclIr4AY9RkSI8EVrL1HTcVLaIH0JnWdO/YC7DSZloC0oswbch1qaW3WsWkJspeaLRvochyFYsatAbvZ46Mzt5uuJUPtSNUVizeb7kBhVGzLVYIepd5XYPgc3Qxp23hu2k9lwg4vjq8WFegC5a34SW/zEZeuFP3HTnD+4=
```

### Delete draft releases

Because there is one draft release created for each pipeline job, it can be quite a lot when one tests the release pipeline. The GitHub WebUI requires quite a lot of clicks to delete them. A way to simplify (a bit) the deletion is to install the command line tool `hub` and call the following command:

```
hub release --include-drafts -f '%U %S %cr%n' | \
	awk '$2 == "draft" && $4 == "days" && $3 > 2 {print $1}' | xargs firefox
```

the `2` compared to `$3` is the number of days, so that you get one tab opened in firefox for each draft release, so that you only need 2 clicks and one Ctrl+W (close the tab) to delete those releases.

> **NOTE:** deletion directly using hub isn't possible as it only supports tags and not release IDs. Drafts do NOT have tags...