File: manual.tex

package info (click to toggle)
s3ql 5.2.3%2Bdfsg-2
links: PTS, VCS
area: main
in suites: forky, sid, trixie
size: 4,416 kB
sloc: python: 19,027; cpp: 408; makefile: 147; sh: 133
file content (5526 lines) | stat: -rw-r--r-- 240,884 bytes
%% Generated by Sphinx.
\def\sphinxdocclass{report}
\documentclass[letterpaper,10pt,english]{sphinxmanual}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsible pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}


\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}



\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}



\usepackage[Bjarne]{fncychap}
\usepackage{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}


% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}


\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{S3QL Documentation}
\date{Nov 02, 2024}
\release{5.2.3}
\author{Nikolaus Rath}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{Release}
\makeindex
\begin{document}

\ifdefined\shorthandoff
  \ifnum\catcode`\=\string=\active\shorthandoff{=}\fi
  \ifnum\catcode`\"=\active\shorthandoff{"}\fi
\fi

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}


\sphinxstepscope


\chapter{S3QL}
\label{\detokenize{about:s3ql}}\label{\detokenize{about::doc}}
\sphinxAtStartPar
S3QL is a file system that stores all its data online using storage
services like \sphinxhref{https://cloud.google.com/storage/docs}{Google Storage}, \sphinxhref{https://aws.amazon.com/s3/}{Amazon S3}, or \sphinxhref{https://www.openstack.org/software/}{OpenStack}. S3QL
effectively provides a virtual drive of dynamic, infinite capacity that
can be accessed from any computer with internet access.

\sphinxAtStartPar
S3QL is a full featured UNIX file system that is conceptually indistinguishable from a
local file system like ext4.  Furthermore, S3QL has additional features like compression
encryption, data de\sphinxhyphen{}duplication, immutable trees and snapshotting which make it especially
suitable for online backup and archival.

\sphinxAtStartPar
S3QL is designed to favor simplicity and elegance over performance and
feature\sphinxhyphen{}creep. Care has been taken to make the source code as
readable and serviceable as possible. Solid error detection and error
handling have been included from the very first line, and S3QL comes
with extensive automated test cases for all its components.


\section{Features}
\label{\detokenize{about:features}}\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Transparency.} Conceptually, S3QL is indistinguishable from a
local file system. For example, it supports hardlinks, symlinks,
standard unix permissions, extended attributes and file
sizes up to 2 TB.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Dynamic Size.} The size of an S3QL file system grows and shrinks
dynamically as required.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Compression.} Before storage, all data may be compressed with the
LZMA, bzip2 or deflate (gzip) algorithm.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Encryption.} After compression (but before upload), all data can be
AES encrypted with a 256 bit key. An additional SHA256 HMAC checksum
is used to protect the data against manipulation.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Data De\sphinxhyphen{}duplication.} If several files have identical contents,
the redundant data will be stored only once. This works across all
files stored in the file system, and also if only some parts of the
files are identical while other parts differ.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Immutable Trees.} Directory trees can be made immutable, so that
their contents can no longer be changed in any way whatsoever. This
can be used to ensure that backups can not be modified after they
have been made.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Copy\sphinxhyphen{}on\sphinxhyphen{}write snapshots.} S3QL can replicate entire directory
trees without using any additional storage space. Only if one of the
copies is modified, the part of the data that has been modified will
take up additional storage space. This can be used to create
intelligent snapshots that preserve the state of a directory at
different points in time using a minimum amount of space.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Performance independent of network latency.} All operations
that do not write or read file contents (like creating directories
or moving, renaming, and changing permissions of files and
directories) are very fast because they are carried out without any
network transactions.

\sphinxAtStartPar
S3QL achieves this by saving the entire file and directory structure
in a database. This database is locally cached and the remote
copy updated asynchronously.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Support for low bandwidth connections.} S3QL splits file contents
into smaller blocks and caches blocks locally. This minimizes both
the number of network transactions required for reading and writing
data, and the amount of data that has to be transferred when only
parts of a file are read or written.

\end{itemize}


\section{Development Status}
\label{\detokenize{about:development-status}}
\sphinxAtStartPar
S3QL is considered stable and suitable for production use.  Starting
with version 2.17.1, S3QL uses semantic versioning. This means that
backwards\sphinxhyphen{}incompatible versions (e.g., versions that require an
upgrade of the file system revision) will be reflected in an increase
of the major version number.


\section{Supported Platforms}
\label{\detokenize{about:supported-platforms}}
\sphinxAtStartPar
S3QL is developed and tested under Linux. Users have also reported
running S3QL successfully on OS\sphinxhyphen{}X, FreeBSD and NetBSD. We try to
maintain compatibility with these systems, but (due to lack of
pre\sphinxhyphen{}release testers) we cannot guarantee that every release will run
on all non\sphinxhyphen{}Linux systems. Please report any bugs you find, and we will
try to fix them.


\section{Contributing}
\label{\detokenize{about:contributing}}
\sphinxAtStartPar
The S3QL source code is available on \sphinxhref{https://github.com/s3ql/main}{GitHub}.

\sphinxstepscope


\chapter{Installation}
\label{\detokenize{installation:installation}}\label{\detokenize{installation::doc}}
\sphinxAtStartPar
S3QL depends on several other programs and libraries that have to be
installed first. The best method to satisfy these dependencies depends
on your distribution.


\section{Dependencies}
\label{\detokenize{installation:dependencies}}
\sphinxAtStartPar
The following is a list of the programs and libraries required for
running S3QL. Generally, you should first check if your distribution
already provides a suitable packages and only install from source if
that is not the case.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Kernel: Linux 3.9 or newer.

\item {} 
\sphinxAtStartPar
The \sphinxhref{http://psmisc.sf.net/}{psmisc} utilities.

\item {} 
\sphinxAtStartPar
\sphinxhref{http://www.sqlite.org/}{SQLite} version 3.7.0 or newer. SQLite
has to be installed as a \sphinxstyleemphasis{shared library} with development headers.

\item {} 
\sphinxAtStartPar
\sphinxhref{http://www.python.org/}{Python} 3.8 or newer. Make sure to also
install the development headers.

\item {} 
\sphinxAtStartPar
The following Python modules:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://pypi.python.org/pypi/setuptools}{setuptools}, version 1.0 or newer.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://cryptography.io/en/latest/installation/}{cryptography}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://pypi.python.org/pypi/defusedxml/}{defusedxml}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/rogerbinns/apsw}{apsw}, version 3.42.0 or
newer.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/python-trio/trio}{trio}, version 0.15 or newer.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/libfuse/pyfuse3/}{pyfuse3}, any
version between 3.2.0 (inclusive) and 4.0 (exclusive)

\item {} 
\sphinxAtStartPar
\sphinxhref{http://pytest.org/}{pytest}, version 3.7 or newer (optional, to run unit tests)

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/systemd/python-systemd}{systemd} (optional,
for enabling systemd support). Do \sphinxstyleemphasis{not} install the module from
PyPi, this is from a third\sphinxhyphen{}party developer and incompatible with
the official module from the systemd developers.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://pypi.python.org/pypi/requests/}{requests} (optional,
required for OAuth2 authentication with Google Storage)

\item {} 
\sphinxAtStartPar
\sphinxhref{https://pypi.python.org/project/google-auth/}{google\sphinxhyphen{}auth}
(optional, required for ADC authentication with Google Storage)

\item {} 
\sphinxAtStartPar
\sphinxhref{https://pypi.python.org/project/google-auth-oauthlib/}{google\sphinxhyphen{}auth\sphinxhyphen{}oauthlib}
(optional, required for browser\sphinxhyphen{}based authentication with Google Storage)

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/python-trio/pytest-trio}{pytest\_trio} (optional, to run unit tests)

\end{itemize}

\sphinxAtStartPar
To check if a specific module \sphinxcode{\sphinxupquote{\textless{}module\textgreater{}}} is installed, execute
\sphinxcode{\sphinxupquote{python3 \sphinxhyphen{}c \textquotesingle{}import \sphinxstyleemphasis{\textless{}module\textgreater{}};
print(\sphinxstyleemphasis{\textless{}module\textgreater{}}.\_\_version\_\_)\textquotesingle{}}}. This will result in an
\sphinxcode{\sphinxupquote{ModuleNotFoundError}} if the module is not installed, and will print the
installed version if the module is installed.

\end{itemize}


\section{Installing S3QL}
\label{\detokenize{installation:installing-s3ql}}\label{\detokenize{installation:inst-s3ql}}
\sphinxAtStartPar
To build and install S3QL, first download the release tarball from
\sphinxhref{https://github.com/s3ql/s3ql/releases}{GitHub}. Then validate the tarball using
\sphinxhref{https://github.com/aperezdc/signify}{signify}:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{signify \PYGZhy{}V \PYGZhy{}m s3ql\PYGZhy{}XX.tar.gz \PYGZhy{}p s3ql\PYGZhy{}XX.pub}
\end{sphinxVerbatim}

\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{s3ql\sphinxhyphen{}XX.pub}} file needs to be obtained from a trustworthy source (it contains the
signing key). Each S3QL release contains the signing key for the release after it in the
\sphinxcode{\sphinxupquote{signify}} directory, so you only need to manually acquire this file once when you install
S3QL for the first time).

\sphinxAtStartPar
After validating the tarball, unpack it and change into the newly created \sphinxcode{\sphinxupquote{s3ql\sphinxhyphen{}X.Y.Z}}
directory. Then you have three options:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Run the S3QL commands directly from the \sphinxcode{\sphinxupquote{bin/}} directory .

\item {} 
\sphinxAtStartPar
Install S3QL into \sphinxcode{\sphinxupquote{\textasciitilde{}/.local}}

\item {} 
\sphinxAtStartPar
You can install S3QL system\sphinxhyphen{}wide for all users.

\end{itemize}


\subsection{Running S3QL commands directly}
\label{\detokenize{installation:running-s3ql-commands-directly}}
\sphinxAtStartPar
Create a virtual environment in the S3QL source directory:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 \PYGZhy{}m venv venv/}
\end{sphinxVerbatim}

\sphinxAtStartPar
and install S3QL dependencies with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{venv/bin/pip install \PYGZbs{}}
\PYG{l}{  apsw cryptography defusedxml trio pyfuse3 pytest \PYGZbs{}\PYGZbs{}}
\PYG{l}{  requests google\PYGZhy{}auth google\PYGZhy{}auth\PYGZhy{}oauthlib pytest\PYGZus{}trio}
\end{sphinxVerbatim}

\sphinxAtStartPar
Then build and test S3QL with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{venv/bin/python3 setup.py build\PYGZus{}ext \PYGZhy{}\PYGZhy{}inplace}
\PYG{l}{venv/bin/python3 \PYGZhy{}m pytest tests/}
\end{sphinxVerbatim}

\sphinxAtStartPar
If this fails, ask for help on the \sphinxhref{http://groups.google.com/group/s3ql}{mailing list} or report a bug in the \sphinxhref{https://github.com/s3ql/s3ql/issues}{issue tracker}.

\sphinxAtStartPar
You can now run the S3QL commands in \sphinxcode{\sphinxupquote{bin}} by calling e.g. \sphinxcode{\sphinxupquote{venv/bin/python3
bin/mkfs.s3ql}} or \sphinxcode{\sphinxupquote{venv/bin/python3 bin/mount.s3ql}}. When using bash, you can
also \sphinxcode{\sphinxupquote{source venv/bin/activate}} (which activates the virtual environment for
this shell) and then call the commands directly (with e.g. \sphinxcode{\sphinxupquote{bin/mkfs.s3ql}}).


\subsection{Installing S3QL for the current user}
\label{\detokenize{installation:installing-s3ql-for-the-current-user}}
\sphinxAtStartPar
Create a virtual environment with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 \PYGZhy{}m venv \PYGZti{}/.local/lib/s3ql\PYGZhy{}venv/}
\end{sphinxVerbatim}

\sphinxAtStartPar
and install S3QL dependencies with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{\PYGZti{}/.local/lib/s3ql\PYGZhy{}venv/bin/pip install \PYGZbs{}}
\PYG{l}{  apsw cryptography defusedxml trio pyfuse3 pytest \PYGZbs{}\PYGZbs{}}
\PYG{l}{  requests google\PYGZhy{}auth google\PYGZhy{}auth\PYGZhy{}oauthlib pytest\PYGZus{}trio}
\end{sphinxVerbatim}

\sphinxAtStartPar
Then build and test S3QL with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{\PYGZti{}/.local/lib/s3ql\PYGZhy{}venv/bin/python3 setup.py build\PYGZus{}ext \PYGZhy{}\PYGZhy{}inplace}
\PYG{l}{\PYGZti{}/.local/lib/s3ql\PYGZhy{}venv/bin/python3 \PYGZhy{}m pytest tests/}
\end{sphinxVerbatim}

\sphinxAtStartPar
If this fails, ask for help on the \sphinxhref{http://groups.google.com/group/s3ql}{mailing list} or report a bug in the \sphinxhref{https://github.com/s3ql/s3ql/issues}{issue tracker}. Otherwise, install S3QL with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{\PYGZti{}/.local/lib/s3ql\PYGZhy{}venv/bin/python3 setup.py install \PYGZhy{}\PYGZhy{}user}
\end{sphinxVerbatim}

\sphinxAtStartPar
Make sure that \sphinxcode{\sphinxupquote{\textasciitilde{}/.local/bin}} is in your \sphinxcode{\sphinxupquote{\$PATH}}.


\subsection{Installing S3QL for all users}
\label{\detokenize{installation:installing-s3ql-for-all-users}}
\sphinxAtStartPar
As much as possible, use your system’s package manager to install S3QL
dependencies. If something is not available, install it with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{sudo python3 \PYGZhy{}m pip install }\PYG{n+nv}{\PYGZlt{}package\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Then build and test S3QL with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{python3 setup.py build\PYGZus{}ext \PYGZhy{}\PYGZhy{}inplace}
\PYG{l}{python3 \PYGZhy{}m pytest tests/}
\end{sphinxVerbatim}

\sphinxAtStartPar
If this fails, ask for help on the \sphinxhref{http://groups.google.com/group/s3ql}{mailing list} or report a bug in the \sphinxhref{https://github.com/s3ql/s3ql/issues}{issue tracker}. Otherwise, install S3QL with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{sudo python3 setup.py install}
\end{sphinxVerbatim}


\section{Development Version}
\label{\detokenize{installation:development-version}}
\sphinxAtStartPar
If you have checked out the unstable development version from the
Git repository, a bit more effort is required. You’ll also need:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Version 0.28.1 or newer of the \sphinxhref{http://www.cython.org/}{Cython} compiler.

\item {} 
\sphinxAtStartPar
Version 1.2b1 or newer of the \sphinxhref{http://sphinx.pocoo.org/}{Sphinx} document processor.

\end{itemize}

\sphinxAtStartPar
With these additional dependencies installed, S3QL can be build and
tested as explained above under “Running S3QL commands directly”.
After installing the dependencies into the virtual environment,
you need to execute:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{venv/bin/python3 setup.py build\PYGZus{}cython}
\end{sphinxVerbatim}

\sphinxAtStartPar
This step will generate the file \sphinxcode{\sphinxupquote{src/s3ql/sqlite3ext.cpp}}.
It is necessary for the \sphinxcode{\sphinxupquote{build\_ext}} step.

\sphinxAtStartPar
Note that when building from the Git repository, building and testing is done with several
additional checks. This may cause compilation and/or tests to fail even though there are
no problems with functionality. For example, any use of functions that are scheduled for
deprecation in future Python version will cause tests to fail. If you would rather just
check for functionality, you can delete the \sphinxcode{\sphinxupquote{MANIFEST.in}} file. In that case, the
build system will behave as it does for a regular release.

\sphinxAtStartPar
The HTML and PDF documentation can be generated with

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{./build\PYGZus{}docs.sh}
\PYG{l}{(cd doc/pdf \PYGZam{}\PYGZam{} make)}
\end{sphinxVerbatim}


\section{Running tests requiring remote servers}
\label{\detokenize{installation:running-tests-requiring-remote-servers}}
\sphinxAtStartPar
By default, tests requiring a connection to a remote storage backend
are skipped. If you would like to run these tests too (which is always
a good idea), you have to create additional entries in your
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2}} file that tell S3QL what server and credentials to
use for these tests. These entries have the following form:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[\PYGZlt{}BACKEND\PYGZgt{}\PYGZhy{}test]}
\PYG{l}{backend\PYGZhy{}login: }\PYG{n+nv}{\PYGZlt{}user\PYGZgt{}}
\PYG{l}{backend\PYGZhy{}password: }\PYG{n+nv}{\PYGZlt{}password\PYGZgt{}}
\PYG{l}{test\PYGZhy{}fs: }\PYG{n+nv}{\PYGZlt{}storage\PYGZhy{}url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Here \sphinxstyleemphasis{\textless{}BACKEND\textgreater{}} specifies the backend that you want to test
(e.g. \sphinxstyleemphasis{s3}, \sphinxstyleemphasis{s3c}, \sphinxstyleemphasis{gs}, or \sphinxstyleemphasis{swift}), \sphinxstyleemphasis{\textless{}user\textgreater{}} and \sphinxstyleemphasis{\textless{}password\textgreater{}} are
the backend authentication credentials, and \sphinxstyleemphasis{\textless{}storage\sphinxhyphen{}url\textgreater{}} specifies
the full storage URL that will be used for testing. \sphinxstylestrong{Any existing
S3QL file system in this storage URL will be destroyed during
testing}.

\sphinxAtStartPar
For example, to run tests that need connection to a Google Storage
server, you would add something like

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[gs\PYGZhy{}test]}
\PYG{l}{backend\PYGZhy{}login: GOOGIGWLONT238MD7HZ4}
\PYG{l}{backend\PYGZhy{}password: rmEbstjscoeunt1249oes1298gauidbs3hl}
\PYG{l}{test\PYGZhy{}fs: gs://joes\PYGZhy{}gs\PYGZhy{}bucket/s3ql\PYGZus{}tests/}
\end{sphinxVerbatim}

\sphinxstepscope


\chapter{Storage Backends}
\label{\detokenize{backends:storage-backends}}\label{\detokenize{backends:id1}}\label{\detokenize{backends::doc}}
\sphinxAtStartPar
S3QL supports different \sphinxstyleemphasis{backends} to store data at different service
providers and using different protocols. A \sphinxstyleemphasis{storage url} specifies a
backend together with some backend\sphinxhyphen{}specific information and uniquely
identifies an S3QL file system. The syntax of the storage url depends on
the backend and is described for every backend below.

\sphinxAtStartPar
Furthermore, every S3QL commands that accepts a storage url also
accepts a \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options}} parameter than can be used to
pass backend\sphinxhyphen{}specific options to the backend module. The available
options are documented with the respective backends below.

\sphinxAtStartPar
All storage backends respect the \sphinxcode{\sphinxupquote{http\_proxy}} (for plain HTTP
connections) and \sphinxcode{\sphinxupquote{https\_proxy}} (for SSL connections)
environment variables.

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
Storage backends are not necessarily compatible. Don’t expect that
you can e.g. copy the data stored by the local backend into Amazon
S3 using some non\sphinxhyphen{}S3QL tool and then access it with S3QL’s S3
backend).
\end{sphinxadmonition}


\section{Google Storage}
\label{\detokenize{backends:google-storage}}
\sphinxAtStartPar
\sphinxhref{https://cloud.google.com/storage/}{Google Storage} is an online
storage service offered by Google. In order to use it with S3QL, make
sure that you enable the JSON API in the \sphinxhref{https://console.cloud.google.com/apis/library/}{GCP Console API Library}

\sphinxAtStartPar
The Google Storage backend uses OAuth2 authentication or \sphinxhref{https://cloud.google.com/docs/authentication/production}{ADC}
(Application Default Credentials).

\sphinxAtStartPar
To use OAuth2 authentication, specify \sphinxcode{\sphinxupquote{oauth2}} as the backend login
and a valid OAuth2 refresh token as the backend password. To obtain a
refresh token, you can use the {\hyperref[\detokenize{man/oauth_client:oauth-client}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_oauth\_client}}}}
program. It will instruct you to open a specific URL in your browser,
enter a code and authenticate with your Google account. Once this
procedure is complete, {\hyperref[\detokenize{man/oauth_client:oauth-client}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_oauth\_client}}}} will
print out the refresh token. Note that you need to do this procedure
only once, the refresh token will remain valid until you explicitly
revoke it.

\sphinxAtStartPar
To use ADC, specify \sphinxcode{\sphinxupquote{adc}} as the backend login and use an arbitrary
value for the backend password.

\sphinxAtStartPar
To create a Google Storage bucket, you can use e.g. the \sphinxhref{https://console.cloud.google.com/storage/browser}{Google
Storage Manager}. The storage URL for accessing the bucket in S3QL is
then

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{gs://}\PYG{n+nv}{\PYGZlt{}bucketname\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Here \sphinxstyleemphasis{bucketname} is the name of the bucket, and \sphinxstyleemphasis{prefix} can be an
arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
Google Storage bucket.

\sphinxAtStartPar
The Google Storage backend accepts the following backend options:
\index{gs\_backend command line option@\spxentry{gs\_backend command line option}!ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\index{ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}!gs\_backend command line option@\spxentry{gs\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-gs_backend-arg-ssl-ca-path}}\phantomsection\label{\detokenize{backends:cmdoption-gs-backend-arg-ssl-ca-path}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{gs\_backend command line option@\spxentry{gs\_backend command line option}!tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}}\index{tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}!gs\_backend command line option@\spxentry{gs\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-gs_backend-arg-tcp-timeout}}\phantomsection\label{\detokenize{backends:cmdoption-gs-backend-arg-tcp-timeout}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tcp\sphinxhyphen{}timeout}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re\sphinxhyphen{}established (default: 20 seconds).

\end{fulllineitems}



\section{Amazon S3}
\label{\detokenize{backends:amazon-s3}}
\sphinxAtStartPar
\sphinxhref{http://aws.amazon.com/s3}{Amazon S3} is the online storage service offered by \sphinxhref{http://aws.amazon.com/}{Amazon
Web Services (AWS)}.  Buckets need to be created with the \sphinxhref{https://console.aws.amazon.com/s3/home}{AWS
Management Console}. For best performance, it is
recommend to create the bucket in the geographically closest storage region.

\sphinxAtStartPar
The storage URL for accessing S3 buckets in S3QL has the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3://}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}bucket\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
\sphinxstyleemphasis{prefix} can be an arbitrary prefix that will be prepended to all
object names used by S3QL. This allows you to store several S3QL file
systems in the same S3 bucket. For example, the storage URL

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3://ap\PYGZhy{}south\PYGZhy{}1/foomart.net/data/s3ql\PYGZus{}backup/}
\end{sphinxVerbatim}

\sphinxAtStartPar
refers to the \sphinxstyleemphasis{foomart.net} bucket in the \sphinxstyleemphasis{ap\sphinxhyphen{}south\sphinxhyphen{}1} region. All
storage objects that S3QL stores in this bucket will be prefixed with
\sphinxstyleemphasis{data/s3ql\_backup/}.

\sphinxAtStartPar
The backend login and password for accessing S3 are not the user id and password that you
use to log into the Amazon Webpage, but the \sphinxstyleemphasis{AWS access key id} and \sphinxstyleemphasis{AWS secret access
key} shown under \sphinxhref{https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8\&action=access-key}{My Account/Access Identifiers}.

\sphinxAtStartPar
The Amazon S3 backend accepts the following backend options:
\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}}\index{no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-no-ssl}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-no-ssl}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{no\sphinxhyphen{}ssl}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Disable encrypted (https) connections and use plain HTTP instead.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\index{ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-ssl-ca-path}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-ssl-ca-path}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}}\index{tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-tcp-timeout}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-tcp-timeout}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tcp\sphinxhyphen{}timeout}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re\sphinxhyphen{}established (default: 20 seconds).

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!sse@\spxentry{sse}}\index{sse@\spxentry{sse}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-sse}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-sse}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{sse}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Enable server side encryption. Both costs \& benefits of S3 server
side encryption are probably rather small, and this option does
\sphinxstyleemphasis{not} affect any client side encryption performed by S3QL itself.

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!it@\spxentry{it}}\index{it@\spxentry{it}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-it}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-it}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{it}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Use INTELLIGENT\_TIERING storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!ia@\spxentry{ia}}\index{ia@\spxentry{ia}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-ia}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-ia}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{ia}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Use STANDARD\_IA (infrequent access) storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!oia@\spxentry{oia}}\index{oia@\spxentry{oia}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-oia}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-oia}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{oia}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Use ONEZONE\_IA (infrequent access) storage class for new objects.
See \sphinxhref{https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html}{AWS S3 Storage classes}

\end{fulllineitems}

\index{s3\_backend command line option@\spxentry{s3\_backend command line option}!rrs@\spxentry{rrs}}\index{rrs@\spxentry{rrs}!s3\_backend command line option@\spxentry{s3\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3_backend-arg-rrs}}\phantomsection\label{\detokenize{backends:cmdoption-s3-backend-arg-rrs}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{rrs}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Enable reduced redundancy storage for newly created objects
(overwrites the \sphinxstyleemphasis{ia} option).

\sphinxAtStartPar
When enabling this option, it is strongly recommended to
periodically run {\hyperref[\detokenize{fsck:s3ql-verify}]{\sphinxcrossref{\DUrole{std,std-ref}{s3ql\_verify}}}}, because objects
that are lost by the storage backend may cause subsequent data loss
even later in time due to the data de\sphinxhyphen{}duplication feature of S3QL (see
{\hyperref[\detokenize{durability:backend-reliability}]{\sphinxcrossref{\DUrole{std,std-ref}{Data Durability}}}} for details).

\end{fulllineitems}



\section{OpenStack/Swift}
\label{\detokenize{backends:openstack-swift}}\label{\detokenize{backends:openstack-backend}}
\sphinxAtStartPar
\sphinxhref{http://www.openstack.org/}{OpenStack} is an open\sphinxhyphen{}source cloud server application suite. \sphinxhref{http://openstack.org/projects/storage/}{Swift} is
the cloud storage module of OpenStack. Swift/OpenStack storage is
offered by many different companies.

\sphinxAtStartPar
There are two different storage URL for the OpenStack backend that
make use of different authentication APIs. For legacy (v1)
authentication, the storage URL is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{swift://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{g+ge}{[:\PYGZlt{}port\PYGZgt{}]}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

\sphinxAtStartPar
for Keystone (v2 and v3) authentication, the storage URL is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{swiftks://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{g+ge}{[:\PYGZlt{}port\PYGZgt{}]}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{:}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

\sphinxAtStartPar
When using Keystone v3 authentication, the \sphinxcode{\sphinxupquote{domain}} backend option (see below) must
be specified too.

\sphinxAtStartPar
In both cases, \sphinxstyleemphasis{hostname} name should be the name of the
authentication server.  The storage container must already exist (most
OpenStack providers offer either a web frontend or a command line tool
for creating containers). \sphinxstyleemphasis{prefix} can be an arbitrary prefix that
will be prepended to all object names used by S3QL, which can be used
to store multiple S3QL file systems in the same container.

\sphinxAtStartPar
When using legacy authentication, the backend login and password
correspond to the OpenStack username and API Access Key. When using
Keystone authentication, the backend password is your regular
OpenStack password and the backend login combines you OpenStack
username and tenant/project in the form \sphinxcode{\sphinxupquote{\textless{}tenant\textgreater{}:\textless{}user\textgreater{}}}.
If no tenant is required, the OpenStack username alone may be used as
backend login. For Keystone v2 \sphinxcode{\sphinxupquote{\textless{}tenant\textgreater{}}} needs to be the
tenant name (\sphinxcode{\sphinxupquote{OS\_TENANT\_NAME}} in the OpenStack RC File).
For Keystone v3 \sphinxcode{\sphinxupquote{\textless{}tenant\textgreater{}}} needs to be the project ID
(\sphinxcode{\sphinxupquote{OS\_TENANT\_ID}} in the OpenStack RC File).

\sphinxAtStartPar
The OpenStack backend accepts the following backend options:
\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}}\index{no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-no-ssl}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-no-ssl}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{no\sphinxhyphen{}ssl}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Use plain HTTP to connect to the authentication server. This option
does not directly affect the connection to the storage
server. Whether HTTPS or plain HTTP is used to connect to the
storage server is determined by the authentication server.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\index{ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-ssl-ca-path}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-ssl-ca-path}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}}\index{tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-tcp-timeout}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-tcp-timeout}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tcp\sphinxhyphen{}timeout}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re\sphinxhyphen{}established (default: 20 seconds).

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!disable\sphinxhyphen{}expect100@\spxentry{disable\sphinxhyphen{}expect100}}\index{disable\sphinxhyphen{}expect100@\spxentry{disable\sphinxhyphen{}expect100}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-disable-expect100}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-disable-expect100}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{disable\sphinxhyphen{}expect100}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If this option is specified, S3QL does not use the \sphinxcode{\sphinxupquote{Expect:
continue}} header (cf. \sphinxhref{http://tools.ietf.org/html/rfc2616\#section-8.2.3}{RFC2616, section 8.2.3}) when uploading
data to the server. This can be used to work around broken storage
servers that don’t fully support HTTP 1.1, but may decrease
performance as object data will be transmitted to the server more
than once in some circumstances.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!no\sphinxhyphen{}feature\sphinxhyphen{}detection@\spxentry{no\sphinxhyphen{}feature\sphinxhyphen{}detection}}\index{no\sphinxhyphen{}feature\sphinxhyphen{}detection@\spxentry{no\sphinxhyphen{}feature\sphinxhyphen{}detection}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-no-feature-detection}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-no-feature-detection}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{no\sphinxhyphen{}feature\sphinxhyphen{}detection}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If this option is specified, S3QL does not try to dynamically detect
advanced features of the Swift backend. In this case S3QL can only
use the least common denominator of supported Swift versions and
configurations.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!domain@\spxentry{domain}}\index{domain@\spxentry{domain}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-domain}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-domain}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{domain}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If this option is specified, S3QL will use the Keystone v3 API. The
default domain ID for OpenStack installations is \sphinxcode{\sphinxupquote{default}}. If this
option is specified without setting the \sphinxcode{\sphinxupquote{project\sphinxhyphen{}domain}} option, this
will be used for both the project and the user domain.
You need to provide the domain ID not the domain name to this option.
If your provider did not give you a domain ID, then it is most likely
\sphinxcode{\sphinxupquote{default}}.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!domain\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{domain\sphinxhyphen{}is\sphinxhyphen{}name}}\index{domain\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{domain\sphinxhyphen{}is\sphinxhyphen{}name}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-domain-is-name}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-domain-is-name}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{domain\sphinxhyphen{}is\sphinxhyphen{}name}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If your provider only supplies you with the name of your domain and not the uuid,
you need to set this \sphinxcode{\sphinxupquote{domain\sphinxhyphen{}is\sphinxhyphen{}name}} option, whereby the \sphinxcode{\sphinxupquote{domain}} is used as the domain name,
not the domain id.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!project\sphinxhyphen{}domain@\spxentry{project\sphinxhyphen{}domain}}\index{project\sphinxhyphen{}domain@\spxentry{project\sphinxhyphen{}domain}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-project-domain}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-project-domain}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{project\sphinxhyphen{}domain}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
In simple cases, the project domain will be the same as the auth
domain. If the \sphinxcode{\sphinxupquote{project\sphinxhyphen{}domain}} option is not specified, it will be
assumed to be the same as the user domain.
You need to provide the domain ID not the domain name to this option.
If your provider did not give you a domain ID, then it is most likely
\sphinxcode{\sphinxupquote{default}}.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name}}\index{project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-project-domain-is-name}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-project-domain-is-name}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If your provider only supplies you with the name of your project domain and not the uuid,
you need to set this \sphinxcode{\sphinxupquote{project\sphinxhyphen{}domain\sphinxhyphen{}name}} option, whereby the \sphinxcode{\sphinxupquote{project\sphinxhyphen{}domain}} is used
as the name of the project domain, not the id of the project domain.
If project\sphinxhyphen{}domain\sphinxhyphen{}is\sphinxhyphen{}name is not set, it is assumed the same as domain\sphinxhyphen{}is\sphinxhyphen{}name.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!tenant\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{tenant\sphinxhyphen{}is\sphinxhyphen{}name}}\index{tenant\sphinxhyphen{}is\sphinxhyphen{}name@\spxentry{tenant\sphinxhyphen{}is\sphinxhyphen{}name}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-tenant-is-name}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-tenant-is-name}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tenant\sphinxhyphen{}is\sphinxhyphen{}name}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Some providers use the tenant name to specify the storage location, and others use the tenant id.
If your provider uses the tenant name and not the id, you need to set this \sphinxcode{\sphinxupquote{tenant\sphinxhyphen{}is\sphinxhyphen{}name}} option.
If \sphinxcode{\sphinxupquote{tenant\sphinxhyphen{}is\sphinxhyphen{}name}} is provided, the \sphinxcode{\sphinxupquote{\textless{}tenant\textgreater{}}} component of the login is used as the tenant
name, not the tenant id.

\end{fulllineitems}

\index{swift\_backend command line option@\spxentry{swift\_backend command line option}!identity\sphinxhyphen{}url@\spxentry{identity\sphinxhyphen{}url}}\index{identity\sphinxhyphen{}url@\spxentry{identity\sphinxhyphen{}url}!swift\_backend command line option@\spxentry{swift\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-swift_backend-arg-identity-url}}\phantomsection\label{\detokenize{backends:cmdoption-swift-backend-arg-identity-url}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{identity\sphinxhyphen{}url}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If your provider does not use hostname:port/v3/auth/tokens but instead has another identity URL, you can use this option.
It allows to replace /v3/auth/tokens with another path like for example /identity/v3/auth/tokens

\end{fulllineitems}



\section{Rackspace CloudFiles}
\label{\detokenize{backends:rackspace-cloudfiles}}
\sphinxAtStartPar
\sphinxhref{http://www.rackspace.com/}{Rackspace} CloudFiles uses \sphinxhref{http://www.openstack.org/}{OpenStack} internally, so it is possible to
just use the OpenStack/Swift backend (see above) with
\sphinxcode{\sphinxupquote{auth.api.rackspacecloud.com}} as the host name. For convenience,
there is also a special \sphinxcode{\sphinxupquote{rackspace}} backend that uses a storage URL
of the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{rackspace://}\PYG{n+nv}{\PYGZlt{}region\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}container\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

\sphinxAtStartPar
The storage container must already exist in the selected
region. \sphinxstyleemphasis{prefix} can be an arbitrary prefix that will be prepended to
all object names used by S3QL and can be used to store several S3QL
file systems in the same container.

\sphinxAtStartPar
You can create a storage container for S3QL using the \sphinxhref{https://mycloud.rackspace.com/}{Cloud Control
Panel} (click on \sphinxstyleemphasis{Files} in the
topmost menu bar).

\sphinxAtStartPar
The Rackspace backend accepts the same backend options as the
{\hyperref[\detokenize{backends:openstack-backend}]{\sphinxcrossref{\DUrole{std,std-ref}{OpenStack backend}}}}.


\section{S3 compatible}
\label{\detokenize{backends:s3-compatible}}
\sphinxAtStartPar
The S3 compatible backend allows S3QL to access any storage service
that uses the same protocol as Amazon S3. The storage URL has the form

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3c://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{l}{:}\PYG{n+nv}{\PYGZlt{}port\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}bucketname\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Here \sphinxstyleemphasis{bucketname} is the name of an (existing) bucket, and \sphinxstyleemphasis{prefix}
can be an arbitrary prefix that will be prepended to all object names
used by S3QL. This allows you to store several S3QL file systems in
the same bucket.

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{s3c://}} authenticates API requests using AWS V2 signatures, which are
deprecated by AWS but still accepted by many S3 compatible services.

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{s3c4://}} denotes a variant of this backend that works the same
but uses AWS V4 signatures for request authentication instead:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3c4://}\PYG{n+nv}{\PYGZlt{}hostname\PYGZgt{}}\PYG{l}{:}\PYG{n+nv}{\PYGZlt{}port\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}bucketname\PYGZgt{}}\PYG{l}{/}\PYG{n+nv}{\PYGZlt{}prefix\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
The S3 compatible backend accepts the following backend options:
\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}}\index{no\sphinxhyphen{}ssl@\spxentry{no\sphinxhyphen{}ssl}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-no-ssl}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-no-ssl}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{no\sphinxhyphen{}ssl}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Disable encrypted (https) connections and use plain HTTP instead.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\index{ssl\sphinxhyphen{}ca\sphinxhyphen{}path@\spxentry{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-ssl-ca-path}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-ssl-ca-path}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{ssl\sphinxhyphen{}ca\sphinxhyphen{}path}}\sphinxcode{\sphinxupquote{=\textless{}path\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
Instead of using the system’s default certificate store, validate
the server certificate against the specified CA
certificates. \sphinxcode{\sphinxupquote{\textless{}path\textgreater{}}} may be either a file containing
multiple certificates, or a directory containing one certificate
per file.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}}\index{tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-tcp-timeout}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-tcp-timeout}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tcp\sphinxhyphen{}timeout}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re\sphinxhyphen{}established (default: 20 seconds).

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!disable\sphinxhyphen{}expect100@\spxentry{disable\sphinxhyphen{}expect100}}\index{disable\sphinxhyphen{}expect100@\spxentry{disable\sphinxhyphen{}expect100}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-disable-expect100}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-disable-expect100}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{disable\sphinxhyphen{}expect100}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If this option is specified, S3QL does not use the \sphinxcode{\sphinxupquote{Expect:
continue}} header (cf. \sphinxhref{http://tools.ietf.org/html/rfc2616\#section-8.2.3}{RFC2616, section 8.2.3}) when uploading
data to the server. This can be used to work around broken storage
servers that don’t fully support HTTP 1.1, but may decrease
performance as object data will be transmitted to the server more
than once in some circumstances.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!dumb\sphinxhyphen{}copy@\spxentry{dumb\sphinxhyphen{}copy}}\index{dumb\sphinxhyphen{}copy@\spxentry{dumb\sphinxhyphen{}copy}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-dumb-copy}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-dumb-copy}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{dumb\sphinxhyphen{}copy}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If this option is specified, S3QL assumes that a COPY request to
the storage server has succeeded as soon as the server returns a
\sphinxcode{\sphinxupquote{200 OK}} status. The \sphinxhref{http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html}{S3 COPY API} specifies that the
storage server may still return an error in the request body (see
the \sphinxhref{https://doc.s3.amazonaws.com/proposals/copy.html}{copy proposal} for the rationale), so this
option should only be used if you are certain that your storage
server only returns \sphinxcode{\sphinxupquote{200 OK}} when the copy operation has been
completely and successfully carried out. Using this option may be
necessary if your storage server does not return a valid response
body for a successful copy operation.

\end{fulllineitems}

\index{s3c\_backend command line option@\spxentry{s3c\_backend command line option}!sig\sphinxhyphen{}region@\spxentry{sig\sphinxhyphen{}region}}\index{sig\sphinxhyphen{}region@\spxentry{sig\sphinxhyphen{}region}!s3c\_backend command line option@\spxentry{s3c\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-s3c_backend-arg-sig-region}}\phantomsection\label{\detokenize{backends:cmdoption-s3c-backend-arg-sig-region}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{sig\sphinxhyphen{}region}}\sphinxcode{\sphinxupquote{=\textless{}region\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
For \sphinxcode{\sphinxupquote{s3c4://}} variant only: Region to use for calculating V4
request signatures. Contrary to S3, the region is not a defined
part of the storage URL and must be specified separately.
Defaults to \sphinxcode{\sphinxupquote{us\sphinxhyphen{}east\sphinxhyphen{}1}}.

\end{fulllineitems}



\section{Backblaze B2}
\label{\detokenize{backends:backblaze-b2}}
\sphinxAtStartPar
Backblaze B2 is a cloud storage with its own API.

\begin{sphinxadmonition}{warning}{Warning:}
\sphinxAtStartPar
S3QL developers do not have access to a Backblaze instance, so this backend is not
being tested before release and may break randomly. This backend depends on Backblaze
users to test and maintain it (aka submit pull requests when it doesn’t work).
\end{sphinxadmonition}

\sphinxAtStartPar
The storage URL for backblaze b2 storage is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{b2://}\PYG{n+nv}{\PYGZlt{}bucket\PYGZhy{}name\PYGZgt{}}\PYG{g+ge}{[/\PYGZlt{}prefix\PYGZgt{}]}
\end{sphinxVerbatim}

\sphinxAtStartPar
\sphinxstyleemphasis{bucket\sphinxhyphen{}name} is an (existing) bucket which has to be accessible with
the provided account key. The \sphinxstyleemphasis{prefix} will be appended to all names
used by S3QL and can be used to hold separate S3QL repositories in the
same bucket.

\sphinxAtStartPar
It is also possible to use an application key. The required key capabilities
are the following:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{listBuckets}}

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{listFiles}}

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{readFiles}}

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{writeFiles}}

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{deleteFiles}}

\end{itemize}
\index{b2\_backend command line option@\spxentry{b2\_backend command line option}!disable\sphinxhyphen{}versions@\spxentry{disable\sphinxhyphen{}versions}}\index{disable\sphinxhyphen{}versions@\spxentry{disable\sphinxhyphen{}versions}!b2\_backend command line option@\spxentry{b2\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-b2_backend-arg-disable-versions}}\phantomsection\label{\detokenize{backends:cmdoption-b2-backend-arg-disable-versions}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{disable\sphinxhyphen{}versions}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If versioning of the bucket is not enabled, this option can be set.
When deleting objects, the bucket will not be scanned for all file versions
because it will be implied that only the one (the most recent) version of a
file exists. This will use only one class B transaction instead of
(possibly) multiple class C transactions.

\end{fulllineitems}

\index{b2\_backend command line option@\spxentry{b2\_backend command line option}!retry\sphinxhyphen{}on\sphinxhyphen{}cap\sphinxhyphen{}exceeded@\spxentry{retry\sphinxhyphen{}on\sphinxhyphen{}cap\sphinxhyphen{}exceeded}}\index{retry\sphinxhyphen{}on\sphinxhyphen{}cap\sphinxhyphen{}exceeded@\spxentry{retry\sphinxhyphen{}on\sphinxhyphen{}cap\sphinxhyphen{}exceeded}!b2\_backend command line option@\spxentry{b2\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-b2_backend-arg-retry-on-cap-exceeded}}\phantomsection\label{\detokenize{backends:cmdoption-b2-backend-arg-retry-on-cap-exceeded}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{retry\sphinxhyphen{}on\sphinxhyphen{}cap\sphinxhyphen{}exceeded}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
If there are data/transaction caps set for the backblaze account, this option
controls if operations should be retried as cap counters are reset every day.
Otherwise the exception would abort the program.

\end{fulllineitems}

\index{b2\_backend command line option@\spxentry{b2\_backend command line option}!test\_mode@\spxentry{test\_mode}}\index{test\_mode@\spxentry{test\_mode}!b2\_backend command line option@\spxentry{b2\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-b2_backend-arg-test_mode}}\phantomsection\label{\detokenize{backends:cmdoption-b2-backend-arg-test-mode}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{test\_mode}}\sphinxcode{\sphinxupquote{=\textless{}value\textgreater{}}}}
\pysigstopsignatures
\sphinxAtStartPar
This option puts the backblaze B2 server into test mode by adding a special header to the
requests. Use this option only to test the failure resiliency of the backend implementation as it
causes unnecessary traffic, delays and transactions.

\sphinxAtStartPar
Valid values are documented in
\sphinxurl{https://www.backblaze.com/docs/en/cloud-storage-integration-checklist} and include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{fail\_some\_uploads}} to randomly fail some uploads.

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{expire\_some\_account\_authorization\_tokens}} to let the server fail some authorization tokens.

\item {} 
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{force\_cap\_exceeded}} to let the server to behave as if the data/transaction caps were exceeded.

\end{itemize}

\end{fulllineitems}

\index{b2\_backend command line option@\spxentry{b2\_backend command line option}!tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}}\index{tcp\sphinxhyphen{}timeout@\spxentry{tcp\sphinxhyphen{}timeout}!b2\_backend command line option@\spxentry{b2\_backend command line option}}

\begin{fulllineitems}
\phantomsection\label{\detokenize{backends:cmdoption-b2_backend-arg-tcp-timeout}}\phantomsection\label{\detokenize{backends:cmdoption-b2-backend-arg-tcp-timeout}}
\pysigstartsignatures
\pysigline{\sphinxbfcode{\sphinxupquote{tcp\sphinxhyphen{}timeout}}\sphinxcode{\sphinxupquote{}}}
\pysigstopsignatures
\sphinxAtStartPar
Specifies the timeout used for TCP connections. If no data can be
exchanged with the remote server for longer than this period, the
TCP connection is closed and re\sphinxhyphen{}established (default: 20 seconds).

\end{fulllineitems}



\section{Local}
\label{\detokenize{backends:local}}
\sphinxAtStartPar
S3QL is also able to store its data on the local file system. This can
be used to backup data on external media, or to access external
services that S3QL can not talk to directly (e.g., it is possible to
store data over SSH by first mounting the remote system using \sphinxhref{http://fuse.sourceforge.net/sshfs.html}{sshfs}
and then using the local backend to store the data in the sshfs
mountpoint).

\sphinxAtStartPar
The storage URL for local storage is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{local://}\PYG{n+nv}{\PYGZlt{}path\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Note that you have to write three consecutive slashes to specify an
absolute path, e.g. \sphinxcode{\sphinxupquote{local:///var/archive}}. Also, relative paths will
automatically be converted to absolute paths before the authentication
file (see {\hyperref[\detokenize{authinfo:authinfo}]{\sphinxcrossref{\DUrole{std,std-ref}{Storing Backend Information and Credentials}}}}) is read, i.e. if you are in the
\sphinxcode{\sphinxupquote{/home/john}} directory and try to mount \sphinxcode{\sphinxupquote{local://s3ql}}, the
corresponding section in the authentication file must match the
storage url \sphinxcode{\sphinxupquote{local:///home/john/s3ql}}.

\sphinxAtStartPar
The local backend does not accept any backend options.

\sphinxstepscope


\chapter{Important Rules to Avoid Losing Data}
\label{\detokenize{durability:important-rules-to-avoid-losing-data}}\label{\detokenize{durability:durability}}\label{\detokenize{durability::doc}}
\sphinxAtStartPar
Most S3QL backends store data in distributed storage systems. These
systems differ from a traditional, local hard disk in several
important ways. In order to avoid losing data, this section should be
read very carefully.


\section{Rules in a Nutshell}
\label{\detokenize{durability:rules-in-a-nutshell}}
\sphinxAtStartPar
To avoid losing your data, obey the following rules:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Know what durability you can expect from your chosen storage
provider. The durability describes how likely it is that a stored
object becomes damaged over time. Such data corruption can never be
prevented completely, techniques like geographic replication and
RAID storage just reduce the likelihood of it to happen (i.e.,
increase the durability).

\item {} 
\sphinxAtStartPar
When choosing a backend and storage provider, keep in mind that
when using S3QL, the effective durability of the file system data
will be reduced because of S3QL’s data de\sphinxhyphen{}duplication feature.

\item {} 
\sphinxAtStartPar
Make sure that your storage provider provides “immediate consistency” when reading, writing, and
listing storage objects. Some providers (including, in the past, Amazon S3) only support
\sphinxstyleemphasis{eventual consistency} which is \sphinxstyleemphasis{insufficient for S3QL} and can lead to complete
data\sphinxhyphen{}loss. (Eventual consistency enables higher availability, but means that changes
to an object may not be immediately visible to readers).

\item {} 
\sphinxAtStartPar
Do not attempt to mount the same file system on different computers (or on
the same computer but with different \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}cachedir}} directories)
at the same time.

\item {} 
\sphinxAtStartPar
Do not attempt to mount (or fsck) a file system on a different computer (or on
the same computer but with a different \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}cachedir}} directory)
when it was not cleanly unmounted.

\end{enumerate}


\section{Data Durability}
\label{\detokenize{durability:data-durability}}\label{\detokenize{durability:backend-reliability}}
\sphinxAtStartPar
The durability of a storage service is measure of the
probability of a storage object to become corrupted over time. The
lower the chance of data loss, the higher the durability. Storage
services like Amazon S3 claim to achieve a durability of up to
99.999999999\% over a year, i.e. if you store 100000000 objects for 100
years, you can expect that at the end of that time at most one object will be
corrupted or lost.

\sphinxAtStartPar
S3QL is designed to reduce redundancy and store data in the smallest
possible form. Therefore, S3QL is generally not able to compensate for
any such losses, and when choosing a storage service you should
carefully review if the offered durability matches your requirements.
When doing this, there are two factors that should be kept in mind.

\sphinxAtStartPar
Firstly, even though S3QL is not able to compensate for storage
service failures, it is able to detect them: when trying to access
data that has been lost or corrupted by the storage service, an IO
error will be returned.

\sphinxAtStartPar
Secondly, the consequences of a data loss by the storage service can
be significantly more severe than you may expect because of S3QL’s
data de\sphinxhyphen{}duplication feature: a data loss in the storage service at
time \sphinxstyleemphasis{x} may cause data that is written \sphinxstyleemphasis{after} time \sphinxstyleemphasis{x} to be lost as
well. Consider the following scenario:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
You store an important file in the S3QL file system.

\item {} 
\sphinxAtStartPar
The storage service loses the data blocks of this file. As long as you
do not access the file or run \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}}, S3QL is not
aware that the data has been lost by the storage service.

\item {} 
\sphinxAtStartPar
You save an additional copy of the important file in a different
location on the same S3QL file system.

\item {} 
\sphinxAtStartPar
S3QL detects that the contents of the new file are identical to the
data blocks that have been stored earlier. Since at this point S3QL
is not aware that these blocks have been lost by the storage service, it
does not save another copy of the file contents in the storage service but
relies on the (presumably) existing blocks instead.

\item {} 
\sphinxAtStartPar
Therefore, even though you saved another copy, you still do not
have a backup of the important file (since both copies refer to the
same data blocks that have been lost by the storage service).

\end{enumerate}

\sphinxAtStartPar
For some storage services, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} can mitigate this
effect. When \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} runs, it asks the storage service
for a list of all stored objects. If objects are missing, it can then
mark the damaged files and prevent the problem from spreading forwards
in time. Figuratively speaking, this establishes a “checkpoint”: data
loss that occurred before running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} can not affect
any file system operations that are performed after the check.
Unfortunately, many storage services only “discover” that objects are
missing or broken when the object actually needs to be retrieved. In
this case, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} will not learn anything by just
querying the list of objects.

\sphinxAtStartPar
The problem be further mitigated by using the \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command in addition
to \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}. \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} asks the storage service to look up every
stored object and may therefore take much longer than running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, but
can also offer a much stronger assurance that no data has been lost by the storage
service. To “recover” from damaged storage objects in the backend, the damaged objects
found by \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} have to be explicitly deleted (so that a successive
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} is able detect them as missing, correct the file system metadata, and
move any affected files to \sphinxcode{\sphinxupquote{lost+found}}). This procedure is currently not automated,
so it is generally a good idea to choose a storage service where the expected data
durability is high enough so that the possibility of a lost object (and thus the need to
run any full checks) can be neglected over long periods of time.

\sphinxstepscope


\chapter{File System Creation}
\label{\detokenize{mkfs:file-system-creation}}\label{\detokenize{mkfs::doc}}
\sphinxAtStartPar
A S3QL file system is created with the \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command. It has the
following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mkfs.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}L \textless{}name\textgreater{}]  
\sphinxAtStartPar
Filesystem label
\item [\sphinxhyphen{}\sphinxhyphen{}data\sphinxhyphen{}block\sphinxhyphen{}size \textless{}size\textgreater{}]  
\sphinxAtStartPar
Block size (in KiB) to use for storing file contents.
Files larger than this size will be stored in multiple
backend objects. Backend object size may be smaller
than this due to compression. Default: 1024 KiB.
\item [\sphinxhyphen{}\sphinxhyphen{}metadata\sphinxhyphen{}block\sphinxhyphen{}size \textless{}size\textgreater{}]  
\sphinxAtStartPar
Block size (in KiB) to use for storing filesystem
metadata. Backend object size may be smaller than this
due to compression. Default: 64 KiB.
\item [\sphinxhyphen{}\sphinxhyphen{}plain]  
\sphinxAtStartPar
Create unencrypted file system.
\end{optionlist}
\end{quote}

\sphinxAtStartPar
Unless you have specified the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}plain}} option,
\sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} will ask you to enter an encryption
password. This password will \sphinxstyleemphasis{not} be read from an authentication file
specified with the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}authfile}} option to prevent accidental
creation of an encrypted file system.

\sphinxAtStartPar
Note that:
\begin{itemize}
\item {} 
\sphinxAtStartPar
All data that is stored under the given storage url is assumed to
managed exclusively by S3QL. Trying to manually save additional
objects (or remove or manipulate existing objects) will lead to file
system corruption, and \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} may delete objects that
do not belong to the file system.

\item {} 
\sphinxAtStartPar
With most storage backends, slashes in the storage url prefix do not
have special meaning. For example, the storage urls
\sphinxcode{\sphinxupquote{s3://mybucket/myprefix/}} and \sphinxcode{\sphinxupquote{s3://mybucket/myprefix}} are
distinct. In the first case, the prefix is \sphinxcode{\sphinxupquote{myprefix/}}, while in
the second it is \sphinxcode{\sphinxupquote{myprefix}}.

\item {} 
\sphinxAtStartPar
S3QL file systems can not be “stacked”, i.e. you cannot have one
file system stored at \sphinxcode{\sphinxupquote{s3://bucketname/outerprefix}} and a second
one at \sphinxcode{\sphinxupquote{s3://bucketname/outerprefix/innerprefix}}.

\end{itemize}

\sphinxstepscope


\chapter{Managing File Systems}
\label{\detokenize{adm:managing-file-systems}}\label{\detokenize{adm::doc}}
\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{s3qladm}} command performs various operations on \sphinxstyleemphasis{unmounted} S3QL
file systems. The file system \sphinxstyleemphasis{must not be mounted} when using
\sphinxcode{\sphinxupquote{s3qladm}} or things will go wrong badly.

\sphinxAtStartPar
The syntax is

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage\PYGZhy{}url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{passphrase}}, \sphinxstyleliteralstrong{\sphinxupquote{restore\sphinxhyphen{}metadata}},
\sphinxstyleliteralstrong{\sphinxupquote{clear}}, \sphinxstyleliteralstrong{\sphinxupquote{recover\sphinxhyphen{}key}} or \sphinxstyleliteralstrong{\sphinxupquote{upgrade}}.

\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} accepts the following general options, no
matter what specific action is being invoked:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}

\sphinxAtStartPar
Hint: run \sphinxcode{\sphinxupquote{s3qladm \textless{}action\textgreater{} \sphinxhyphen{}\sphinxhyphen{}help}} to get help on the additional arguments
that the different actions take.


\section{Changing the Passphrase}
\label{\detokenize{adm:changing-the-passphrase}}
\sphinxAtStartPar
To change the passphrase of a file system, use the \sphinxcode{\sphinxupquote{passphrase}}
subcommand:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm passphrase  }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This will not actually re\sphinxhyphen{}encrypt all the stored data with the new passphrase. Rather, S3QL
generates an internal “master key” when the file system is created and uses this key to
encrypt all stored data. The user\sphinxhyphen{}supplied passphrase is used to encrypt/decrypt only the
master key, and can therefore be changed. This means, however, that if a passphrase has
been disclosed to someone untrusted, changing it afterwards will \sphinxstylestrong{not revoke access to
people who had access to the old passphrase} (because they could have decrypted the
master key and retained a copy).


\section{Recovering the master key}
\label{\detokenize{adm:recovering-the-master-key}}
\sphinxAtStartPar
If you’ve been extremely unlucky, then it is possible that a bug in S3QL or data\sphinxhyphen{}loss by
your storage provider has resulted in object that holds the filesystem master key becoming
inaccessible. In this case, knowing the passphrase does not help, as there is no master
key to decrypt with it.

\sphinxAtStartPar
If you have also been diligent, then you can restore the master key from the external copy
that \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} has asked you to make on filesystem creation time:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm recover\PYGZhy{}key }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\section{Restoring Metadata Backups}
\label{\detokenize{adm:restoring-metadata-backups}}
\sphinxAtStartPar
If the most\sphinxhyphen{}recent copy of the file system metadata has been damaged
irreparably, it is possible to restore one of the automatically
created backup copies.

\sphinxAtStartPar
The command

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm restore\PYGZhy{}metadata }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
will give you a list of the available metadata backups and allow you
to select one to restore.

\begin{sphinxadmonition}{warning}{Warning:}
\sphinxAtStartPar
You should probably not use this functionality without having asked
for help on the mailing list first (see {\hyperref[\detokenize{resources:resources}]{\sphinxcrossref{\DUrole{std,std-ref}{Further Resources / Getting Help}}}}).
\end{sphinxadmonition}


\section{Deleting a file system}
\label{\detokenize{adm:deleting-a-file-system}}
\sphinxAtStartPar
A file system can be deleted with:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm clear }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This physically deletes all the data and file system structures.


\section{Upgrading the file system}
\label{\detokenize{adm:upgrading-the-file-system}}
\sphinxAtStartPar
If you have installed a new version of S3QL, it may sometimes be
necessary to upgrade the file system metadata as well. After this operation,
the file system can no longer be accessed with older
versions of S3QL.

\sphinxAtStartPar
During the upgrade you have to make sure that the command is not
interrupted, and that no one else tries to mount, check or upgrade the
file system at the same time.

\sphinxAtStartPar
To upgrade a file system from the previous to the current revision,
execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm upgrade }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxstepscope


\chapter{Mounting}
\label{\detokenize{mount:mounting}}\label{\detokenize{mount::doc}}
\sphinxAtStartPar
A S3QL file system is mounted with the \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
command. It has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
S3QL is not a network file system like \sphinxhref{http://en.wikipedia.org/wiki/Network\_File\_System\_\%28protocol\%29}{NFS}
or \sphinxhref{http://en.wikipedia.org/wiki/CIFS}{CIFS}. It can only be
mounted on one computer at a time.
\end{sphinxadmonition}

\sphinxAtStartPar
This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/mount.log}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}compress \textless{}algorithm\sphinxhyphen{}lvl\textgreater{}]  
\sphinxAtStartPar
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma\sphinxhyphen{}6}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachesize \textless{}size\textgreater{}]  
\sphinxAtStartPar
Cache size in KiB (default: autodetect).
\item [\sphinxhyphen{}\sphinxhyphen{}max\sphinxhyphen{}cache\sphinxhyphen{}entries \textless{}num\textgreater{}]  
\sphinxAtStartPar
Maximum number of entries in cache (default:
autodetect). Each cache entry requires one file
descriptor, so if you increase this number you have to
make sure that your process file descriptor limit (as
set with \sphinxcode{\sphinxupquote{ulimit \sphinxhyphen{}n}}) is high enough (at least the
number of cache entries + 100).
\item [\sphinxhyphen{}\sphinxhyphen{}keep\sphinxhyphen{}cache]  
\sphinxAtStartPar
Do not purge locally cached files on exit.
\item [\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other]  
\sphinxAtStartPar
Normally, only the user who called \sphinxcode{\sphinxupquote{mount.s3ql}} can
access the mount point. This user then also has full
access to it, independent of individual file
permissions. If the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} option is
specified, other users can access the mount point as
well and individual file permissions are taken into
account for all users.
\item [\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root]  
\sphinxAtStartPar
Like \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}}, but restrict access to the
mounting user and the root user.
\item [\sphinxhyphen{}\sphinxhyphen{}dirty\sphinxhyphen{}block\sphinxhyphen{}upload\sphinxhyphen{}delay \textless{}seconds\textgreater{}]  
\sphinxAtStartPar
Upload delay for dirty blocks in seconds (default: 10
seconds).
\item [\sphinxhyphen{}\sphinxhyphen{}fg]  
\sphinxAtStartPar
Do not daemonize, stay in foreground
\item [\sphinxhyphen{}\sphinxhyphen{}fs\sphinxhyphen{}name FS\_NAME]  
\sphinxAtStartPar
Mount name passed to fuse, the name will be shown in
the first column of the system mount command output.
If not specified your storage url is used.
\item [\sphinxhyphen{}\sphinxhyphen{}systemd]  
\sphinxAtStartPar
Run as systemd unit. Consider specifying \textendash{}log none as
well to make use of journald.
\item [\sphinxhyphen{}\sphinxhyphen{}metadata\sphinxhyphen{}backup\sphinxhyphen{}interval \textless{}seconds\textgreater{}]  
\sphinxAtStartPar
Interval between metadata backups. Should the
filesystem crash while mounted, modifications made
after the most recent metadata backup may be lost.
During backups, the filesystem will be unresponsile.
Default: 6h
\item [\sphinxhyphen{}\sphinxhyphen{}threads \textless{}no\textgreater{}]  
\sphinxAtStartPar
Number of parallel upload threads to use (default:
auto).
\item [\sphinxhyphen{}\sphinxhyphen{}nfs]  
\sphinxAtStartPar
Enable some optimizations for exporting the file
system over NFS. (default: False)
\end{optionlist}
\end{quote}


\section{Permission Checking}
\label{\detokenize{mount:permission-checking}}
\sphinxAtStartPar
If the file system is mounted with neither the \sphinxcode{\sphinxupquote{allow\sphinxhyphen{}root}} nor
\sphinxcode{\sphinxupquote{allow\sphinxhyphen{}other}} option, the mounting user has full permissions on the S3QL file
system no matter the owner and mode of individual files. If one (or both) of the options
is used, standard unix permission checks apply, i.e. only the real root user has full
access and all other users (including the mounting user) are subject to permission checks.


\section{Compression Algorithms}
\label{\detokenize{mount:compression-algorithms}}
\sphinxAtStartPar
S3QL supports three compression algorithms, LZMA, Bzip2 and zlib (with
LZMA being the default). The compression algorithm can be specified
freely whenever the file system is mounted, since it affects only the
compression of new data blocks.

\sphinxAtStartPar
Roughly speaking, LZMA is slower but achieves better compression
ratios than Bzip2, while Bzip2 in turn is slower but achieves better
compression ratios than zlib.

\sphinxAtStartPar
For maximum file system performance, the best algorithm therefore depends on your network
connection speed: the compression should be just fast enough to saturate your network
connection.

\sphinxAtStartPar
To find the optimal algorithm and number of concurrent compression
operations for your system, S3QL ships with a program called
\sphinxcode{\sphinxupquote{benchmark.py}} in the \sphinxcode{\sphinxupquote{contrib}} directory. You should run this program
on a file that has a size that is roughly equal to the block size of
your file system and has similar contents. It will then determine the
compression speeds for the different algorithms and the upload speeds
for the specified backend and recommend the best algorithm that is
fast enough to saturate your network connection.

\sphinxAtStartPar
Obviously you should make sure that there is little other system load
when you run \sphinxcode{\sphinxupquote{benchmark.py}} (i.e., don’t compile software or encode
videos at the same time).


\section{Notes about Caching}
\label{\detokenize{mount:notes-about-caching}}
\sphinxAtStartPar
S3QL maintains a local cache of the file system data to speed up
access. The cache is block based, so it is possible that only parts of
a file are in the cache.


\subsection{Maximum Number of Cache Entries}
\label{\detokenize{mount:maximum-number-of-cache-entries}}
\sphinxAtStartPar
The maximum size of the cache can be configured with the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}cachesize}} option. In addition to that, the maximum number
of objects in the cache is limited by the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}max\sphinxhyphen{}cache\sphinxhyphen{}entries}} option, so it is possible that the cache
does not grow up to the maximum cache size because the maximum number
of cache elements has been reached. The reason for this limit is that
each cache entry requires one open file descriptor, and Linux
distributions usually limit the total number of file descriptors per
process to about a thousand.

\sphinxAtStartPar
If you specify a value for \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}max\sphinxhyphen{}cache\sphinxhyphen{}entries}}, you should
therefore make sure to also configure your system to increase the
maximum number of open file handles. This can be done temporarily with
the \sphinxstyleliteralstrong{\sphinxupquote{ulimit \sphinxhyphen{}n}} command. The method to permanently change this limit
system\sphinxhyphen{}wide depends on your distribution.


\subsection{Cache Flushing and Expiration}
\label{\detokenize{mount:cache-flushing-and-expiration}}
\sphinxAtStartPar
S3QL writes changed blocks to the backend whenever a block in the cache has not been
modified for some time, or when it runs out of cache space.  The time can set using the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}dirty\sphinxhyphen{}block\sphinxhyphen{}upload\sphinxhyphen{}delay}} option.

\sphinxAtStartPar
Cache expiration (i.e., removal of blocks from the cache) is only done when the maximum
cache size is reached. S3QL always expires the least recently used blocks first.


\section{NFS Support}
\label{\detokenize{mount:nfs-support}}
\sphinxAtStartPar
S3QL filesystems can be exported over NFS. The \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}nfs}} option
is recommended to improve performance when NFS is used, but no harm
will occur when it is not specified.

\sphinxAtStartPar
NFS supports persistence of client mounts across server restarts. This
means that if a client has mounted an S3QL file system over NFS, the
server may unmount and remount the S3QL filesystem (or even reboot)
without the client being affected beyond temporarily becoming
unavailable. This poses several challenges, but is supported by S3QL
as long as no \sphinxcode{\sphinxupquote{fsck.s3ql}} operation is run:

\begin{sphinxadmonition}{warning}{Warning:}
\sphinxAtStartPar
If \sphinxcode{\sphinxupquote{fsck.s3ql}} modifies a file system in any way, all NFS
clients must unmount and re\sphinxhyphen{}mount the NFS share before the
S3QL file system is re\sphinxhyphen{}mounted on the server.
\end{sphinxadmonition}


\section{Failure Modes}
\label{\detokenize{mount:failure-modes}}
\sphinxAtStartPar
Once an S3QL file system has been mounted, there is a multitude of
problems that can occur when communicating with the remote
server. Generally, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} always tries to keep the file
system as accessible as possible under the circumstances. That means
that if network connectivity is lost, data can still be written as
long as there is space in the local cache. Attempts to read data not
already present in the cache, however, will block until connection is
re\sphinxhyphen{}established. If any sort of data corruption is detected, the file
system will switch to read\sphinxhyphen{}only mode. Attempting to read files that
are affected by the corruption will return an input/output error
(\sphinxstyleemphasis{errno} set to \sphinxcode{\sphinxupquote{EIO}}).

\sphinxAtStartPar
In case of other unexpected or fatal problems, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
terminates, but does not unmount the file system. Any attempt to
access the mountpoint will result in a “Transport endpoint not
connected” error (\sphinxstyleemphasis{errno} set to \sphinxcode{\sphinxupquote{ESHUTDOWN}}). This ensures that a
mountpoint whose \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process has terminated can not
be confused with a mountpoint containing an empty file system (which
would be fatal if e.g. the mountpoint is automatically mirrored). When
this has happened, the mountpoint can be cleared by using the
\sphinxstyleliteralstrong{\sphinxupquote{fusermount}} command (provided by FUSE) with the \sphinxcode{\sphinxupquote{\sphinxhyphen{}u}}
parameter.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will automatically try to re\sphinxhyphen{}establish the
connection to the server if network connectivity is lost, and retry
sending a request when the connection is established but the remote
server signals a temporary problem. These attempts will be made at
increasing intervals for a period up to 24 hours, with retry intervals
starting at 20 ms and increasing up to 5 minutes. After 24 hours,
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will give up and terminate, leaving the
mountpoint inaccessible as described above.

\sphinxAtStartPar
Generally, \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} will also emit log messages for any
unusual conditions that it encounters. The destination for these
messages can be set with the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} parameter. It is highly
recommended to periodically check these logs, for example with a tool
like \sphinxhref{http://sourceforge.net/projects/logcheck/}{logcheck}. Many potential issues that \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} may
encounter do not justify restricting access to the file system, but
should nevertheless be investigated if they occur. Checking the log
messages is the only way to find out about them.


\section{Automatic Mounting}
\label{\detokenize{mount:automatic-mounting}}
\sphinxAtStartPar
If you want to mount and umount an S3QL file system automatically at
system startup and shutdown, you should do so with a dedicated S3QL
init job (instead of using \sphinxcode{\sphinxupquote{/etc/fstab}}). When using systemd,
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} can be started with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}systemd}} to run
as a systemd service of type \sphinxcode{\sphinxupquote{notify}}.

\sphinxAtStartPar
In principle, it is also possible to automatically mount an S3QL file system with an
appropriate entry in \sphinxcode{\sphinxupquote{/etc/fstab}}. However, this is not recommended for several reasons:
\begin{itemize}
\item {} 
\sphinxAtStartPar
file systems mounted in \sphinxcode{\sphinxupquote{/etc/fstab}} will be unmounted with the \sphinxstyleliteralstrong{\sphinxupquote{umount}}
command, so your system will not wait until all data has been uploaded but shutdown (or
restart) immediately (this is a FUSE limitation, cf
\sphinxurl{https://github.com/libfuse/libfuse/issues/1}).

\item {} 
\sphinxAtStartPar
There is no way to tell the system that mounting S3QL requires a Python interpreter to
be available, so it may attempt to run \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} before it has mounted the
volume containing the Python interpreter.

\item {} 
\sphinxAtStartPar
There is no standard way to tell the system that internet connection has to be up before
the S3QL file system can be mounted.

\end{itemize}

\sphinxstepscope


\chapter{Advanced S3QL Features}
\label{\detokenize{special:advanced-s3ql-features}}\label{\detokenize{special::doc}}

\section{Snapshotting and Copy\sphinxhyphen{}on\sphinxhyphen{}Write}
\label{\detokenize{special:snapshotting-and-copy-on-write}}\label{\detokenize{special:s3qlcp}}
\sphinxAtStartPar
The command \sphinxcode{\sphinxupquote{s3qlcp}} can be used to duplicate a directory tree without
physically copying the file contents. This is made possible by the
data de\sphinxhyphen{}duplication feature of S3QL.

\sphinxAtStartPar
The syntax of \sphinxcode{\sphinxupquote{s3qlcp}} is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}src\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}target\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This will replicate the contents of the directory \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} in the
directory \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}}. \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} has to be an existing directory and
\sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} must not exist. Moreover, both directories have to be
within the same S3QL file system.

\sphinxAtStartPar
The replication will not take any additional space for the file contents. Metadata usage
will increase proportional to the amount of directory entries and inodes. Only if one of
directories is modified later on, the modified data will take additional storage space.

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{s3qlcp}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}})
the root user.

\sphinxAtStartPar
After the replication, both source and target directory will still be completely ordinary
directories. You can regard \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} as a snapshot of \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} or vice versa. However,
the most common usage of \sphinxcode{\sphinxupquote{s3qlcp}} is to regularly duplicate the same source directory, say
\sphinxcode{\sphinxupquote{documents}}, to different target directories. For a e.g. monthly replication, the target
directories would typically be named something like \sphinxcode{\sphinxupquote{documents\_January}} for the
replication in January, \sphinxcode{\sphinxupquote{documents\_February}} for the replication in February etc.  In this
case it is clear that the target directories should be regarded as snapshots of the source
directory.

\sphinxAtStartPar
Exactly the same effect could be achieved by an ordinary copy program like \sphinxcode{\sphinxupquote{cp
\sphinxhyphen{}a}}. However, this procedure would be orders of magnitude slower, because \sphinxcode{\sphinxupquote{cp}} would have
to read every file completely (so that S3QL had to fetch all the data over the network
from the backend) before writing them into the destination folder (at which point S3QL
would de\sphinxhyphen{}duplicate the data).


\subsection{Snapshotting vs Hardlinking}
\label{\detokenize{special:snapshotting-vs-hardlinking}}
\sphinxAtStartPar
Snapshot support in S3QL is inspired by the hardlinking feature that
is offered by programs like \sphinxhref{http://www.samba.org/rsync}{rsync} or
\sphinxhref{http://savannah.nongnu.org/projects/storebackup}{storeBackup}.
These programs can create a hardlink instead of copying a file if an
identical file already exists in the backup. However, using hardlinks
has disadvantages:
\begin{itemize}
\item {} 
\sphinxAtStartPar
backups and restores always have to be made with a special program
that takes care of the hardlinking. The backups must not be touched
by any other programs (they may make changes that inadvertently
affect other hardlinked files)

\item {} 
\sphinxAtStartPar
special care needs to be taken to handle files which are already
hardlinked (the restore program needs to know that the hardlink was
not introduced by the backup program)

\end{itemize}

\sphinxAtStartPar
S3QL snapshots do not have these problems, and they can be used with
any backup program.


\section{Getting Statistics}
\label{\detokenize{special:getting-statistics}}\label{\detokenize{special:s3qlstat}}
\sphinxAtStartPar
You can get more information about a mounted S3QL file system with the
\sphinxcode{\sphinxupquote{s3qlstat}} command. It has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlstat }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This will print out something like this

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{Directory entries:    1488068}
\PYG{l}{Inodes:               1482991}
\PYG{l}{Data blocks:          87948}
\PYG{l}{Total data size:      400 GiB}
\PYG{l}{After de\PYGZhy{}duplication: 51 GiB (12.98\PYGZpc{} of total)}
\PYG{l}{After compression:    43 GiB (10.85\PYGZpc{} of total, 83.60\PYGZpc{} of de\PYGZhy{}duplicated)}
\PYG{l}{Database size:        172 MiB (uncompressed)}
\PYG{l}{(some values do not take into account not\PYGZhy{}yet\PYGZhy{}uploaded dirty blocks in cache)}
\end{sphinxVerbatim}

\sphinxAtStartPar
Probably the most interesting numbers are the total size of your data,
the total size after duplication, and the final size after
de\sphinxhyphen{}duplication and compression.

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{s3qlstat}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}})
the root user.

\sphinxAtStartPar
For a full list of available options, run \sphinxcode{\sphinxupquote{s3qlstat \sphinxhyphen{}\sphinxhyphen{}help}}.


\section{Immutable Trees}
\label{\detokenize{special:immutable-trees}}\label{\detokenize{special:s3qllock}}
\sphinxAtStartPar
The command \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} can be used to make a directory tree
immutable. Immutable trees can no longer be changed in any way
whatsoever. You can not add new files or directories and you can not
change or delete existing files and directories. The only way to get
rid of an immutable tree is to use the \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command (see
below).

\sphinxAtStartPar
For example, to make the directory tree beneath the directory
\sphinxcode{\sphinxupquote{2010\sphinxhyphen{}04\sphinxhyphen{}21}} immutable, execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qllock 2010\PYGZhy{}04\PYGZhy{}21}
\end{sphinxVerbatim}

\sphinxAtStartPar
Immutability is a feature designed for backups. Traditionally, backups have been made on
external tape drives. Once a backup was made, the tape drive was removed and locked away
somewhere. This means that the contents of the backup are permanently fixed. Nothing
(short of physical destruction) can change or delete files in the backup.

\sphinxAtStartPar
In contrast, when backing up into an online storage system like S3QL,
all backups are available every time the file system is mounted.
Nothing prevents a file in an old backup from being changed again
later on. In the worst case, this may make your entire backup system
worthless. Imagine that your system gets infected by a virus
that simply deletes all files it can find \textendash{} if the virus is active
while the backup file system is mounted, the virus will destroy all
backups together with the originals.

\sphinxAtStartPar
Even in the absence of malware,, being able to change a backup after it has been made is
generally not a good idea. A common S3QL use case is to keep the file system mounted at
all times and periodically create backups with \sphinxstyleliteralstrong{\sphinxupquote{rsync \sphinxhyphen{}a}}. This allows every user
to recover her files from a backup without having to call the system
administrator. However, this also allows every user to accidentally change or delete files
\sphinxstyleemphasis{in} one of the old backups.

\sphinxAtStartPar
Making a backup immutable protects you against all these problems.
Unless you happen to run into a virus that was specifically programmed
to attack S3QL file systems, backups can be neither deleted nor
changed after they have been made immutable.


\section{Fast Recursive Removal}
\label{\detokenize{special:fast-recursive-removal}}\label{\detokenize{special:s3qlrm}}
\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{s3qlrm}} command can be used to recursively delete files and
directories on an S3QL file system. Although \sphinxcode{\sphinxupquote{s3qlrm}} is faster than
using e.g. \sphinxcode{\sphinxupquote{rm \sphinxhyphen{}r}}, the main reason for its existence is that it
allows you to delete immutable trees as well. The syntax is rather
simple:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlrm }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
Be warned that there is no additional confirmation. The directory will
be removed entirely and immediately.


\section{Runtime Configuration}
\label{\detokenize{special:runtime-configuration}}\label{\detokenize{special:s3qlctrl}}\begin{quote}

\sphinxAtStartPar
s3qlctrl {[}options{]} \textless{}action\textgreater{} \textless{}mountpoint\textgreater{} …
\end{quote}

\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command performs various actions on the S3QL file system mounted
in \sphinxcode{\sphinxupquote{mountpoint}}.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} can only be called by the user that mounted the file system and (if
the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}}) the
root user.

\sphinxAtStartPar
The following actions may be specified:
\begin{quote}
\begin{quote}\begin{description}
\sphinxlineitem{flushcache}
\sphinxAtStartPar
Write all modified blocks to the backend. The command blocks until the
cache is clean.

\sphinxlineitem{dropcache}
\sphinxAtStartPar
Flush the filesystem cache and then drop all contents (i.e., make the cache
empty).

\sphinxlineitem{log}
\sphinxAtStartPar
Change the amount of information that is logged. The complete syntax is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ log }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}level\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}module\PYGZgt{} ...]}
\end{sphinxVerbatim}

\sphinxAtStartPar
here \sphinxcode{\sphinxupquote{level}} is the desired new log level and may be either of \sphinxstyleemphasis{debug},
\sphinxstyleemphasis{info} or \sphinxstyleemphasis{warn}. One or more \sphinxcode{\sphinxupquote{module}} may only be specified with the
\sphinxstyleemphasis{debug} level and allow to restrict the debug output to just the listed
modules.

\sphinxlineitem{cachesize}
\sphinxAtStartPar
Changes the cache size of the file
system. This action requires an additional argument that specifies the new
cache size in KiB, so the complete command line is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ cachesize }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}new\PYGZhy{}cache\PYGZhy{}size\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxlineitem{backup\sphinxhyphen{}metadata}
\sphinxAtStartPar
Trigger a metadata backup.

\end{description}\end{quote}
\end{quote}

\sphinxstepscope


\chapter{Unmounting}
\label{\detokenize{umount:unmounting}}\label{\detokenize{umount::doc}}
\sphinxAtStartPar
To unmount an S3QL file system, use the command:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{umount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This will block until all data has been written to the backend.

\sphinxAtStartPar
Only the user who mounted the file system with \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
is able to unmount it again. If you are root and want to unmount an
S3QL file system mounted by an ordinary user, you have to use the
\sphinxstyleliteralstrong{\sphinxupquote{fusermount \sphinxhyphen{}u}} or \sphinxstyleliteralstrong{\sphinxupquote{umount}} command instead. Note
that these commands do not block until all data has been uploaded, so
if you use them instead of \sphinxcode{\sphinxupquote{umount.s3ql}} then you should manually wait
for the \sphinxcode{\sphinxupquote{mount.s3ql}} process to terminate before shutting down the
system.

\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}lazy, \sphinxhyphen{}z]  
\sphinxAtStartPar
Lazy umount. Detaches the file system immediately,
even if there are still open files. The data will be
uploaded in the background once all open files have
been closed.
\end{optionlist}
\end{quote}

\sphinxAtStartPar
If, for some reason, the \sphinxcode{\sphinxupquote{umount.s3ql}} command does not work, the file
system can also be unmounted with \sphinxcode{\sphinxupquote{fusermount \sphinxhyphen{}u \sphinxhyphen{}z}}. Note that this
command will return immediately and the file system may continue to
upload data in the background for a while longer.

\sphinxstepscope


\chapter{Checking for Errors}
\label{\detokenize{fsck:checking-for-errors}}\label{\detokenize{fsck::doc}}
\sphinxAtStartPar
It is recommended to periodically run the \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} and
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} commands (in this order) to ensure that the
file system is consistent, and that there has been no data corruption
or data loss in the storage backend.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} is intended to detect and correct problems with
the internal file system structure, caused by e.g. a file system crash
or a bug in S3QL. It assumes that the storage backend can be fully
trusted, i.e. if the backend reports that a specific storage object
exists, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} takes that as proof that the data is
present and intact.

\sphinxAtStartPar
In contrast to that, the \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command is intended to
check the consistency of the storage backend. It assumes that the
internal file system data is correct, and verifies that all data can
actually be retrieved from the backend. Running \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}}
may therefore take much longer than running \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}.


\section{Checking and repairing internal file system errors}
\label{\detokenize{fsck:checking-and-repairing-internal-file-system-errors}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} checks that the internal file system structure is
consistent and attempts to correct any problems it finds. If an S3QL
file system has not been unmounted correctly for any reason, you need
to run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} before you can mount the file system
again.

\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{fsck.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/fsck.log}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}compress \textless{}algorithm\sphinxhyphen{}lvl\textgreater{}]  
\sphinxAtStartPar
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma\sphinxhyphen{}6}}
\item [\sphinxhyphen{}\sphinxhyphen{}keep\sphinxhyphen{}cache]  
\sphinxAtStartPar
Do not purge locally cached files on exit.
\item [\sphinxhyphen{}\sphinxhyphen{}batch]  
\sphinxAtStartPar
If user input is required, exit without prompting.
\item [\sphinxhyphen{}\sphinxhyphen{}force]  
\sphinxAtStartPar
Force checking even if file system is marked clean.
\item [\sphinxhyphen{}\sphinxhyphen{}force\sphinxhyphen{}remote]  
\sphinxAtStartPar
Force use of remote metadata even when this would
likely result in data loss.
\end{optionlist}
\end{quote}


\section{Detecting and handling backend data corruption}
\label{\detokenize{fsck:detecting-and-handling-backend-data-corruption}}\label{\detokenize{fsck:s3ql-verify}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command verifies all data in the file
system.  In contrast to \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}}
does not trust the object listing returned by the backend, but
actually attempts to retrieve every object. By default,
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} will attempt to retrieve just the metadata for
every object (for e.g. the S3\sphinxhyphen{}compatible or Google Storage backends
this corresponds to a \sphinxcode{\sphinxupquote{HEAD}} request for each object), which is
generally sufficient to determine if the object still exists. When
specifying the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}data}} option, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} will
instead read every object entirely. To determine how much data will be
transmitted in total when using \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}data}}, look at the \sphinxstyleemphasis{After
compression} row in the {\hyperref[\detokenize{special:s3qlstat}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlstat}}}} output.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} is not able to correct any data corruption that
it finds. Instead, a list of the corrupted and/or missing objects is
written to a file and the decision about the proper course of action
is left to the user. If you have administrative access to the backend
server, you may want to investigate the cause of the corruption or
check if the missing/corrupted objects can be restored from
backups. If you believe that the missing/corrupted objects are indeed
lost irrevocably, you can use the {\hyperref[\detokenize{contrib:remove-objects}]{\sphinxcrossref{\DUrole{std,std-ref}{remove\_objects.py}}}} script (from
the \sphinxcode{\sphinxupquote{contrib}} directory of the S3QL distribution) to explicitly
delete the objects from the storage backend. After that, you should
run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}. Since the (now explicitly deleted) objects
should now no longer be included in the object index reported by the
backend, \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} will identify the objects as missing,
update the internal file system structures accordingly, and move the
affected files into the \sphinxcode{\sphinxupquote{lost+found}} directory.

\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command has the following syntax:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}verify }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
This command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}missing\sphinxhyphen{}file \textless{}name\textgreater{}]  
\sphinxAtStartPar
File to store keys of missing objects.
\item [\sphinxhyphen{}\sphinxhyphen{}corrupted\sphinxhyphen{}file \textless{}name\textgreater{}]  
\sphinxAtStartPar
File to store keys of corrupted objects.
\item [\sphinxhyphen{}\sphinxhyphen{}data]  
\sphinxAtStartPar
Read every object completely, instead of checking just
the metadata.
\item [\sphinxhyphen{}\sphinxhyphen{}parallel PARALLEL]  
\sphinxAtStartPar
Number of connections to use in parallel.
\item [\sphinxhyphen{}\sphinxhyphen{}start\sphinxhyphen{}with \textless{}n\textgreater{}]  
\sphinxAtStartPar
Skip over first \textless{}n\textgreater{} objects and with verifying object
\textless{}n\textgreater{}+1.
\end{optionlist}
\end{quote}

\sphinxstepscope


\chapter{Storing Backend Information and Credentials}
\label{\detokenize{authinfo:storing-backend-information-and-credentials}}\label{\detokenize{authinfo:authinfo}}\label{\detokenize{authinfo::doc}}
\sphinxAtStartPar
Normally, S3QL reads username and password for the backend as well as
an encryption passphrase for the file system from the
terminal. Alternatively, these (and some other backend parameters) may
be specified in the \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2}} file, or a file specified by the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}authfile}} parameter.

\sphinxAtStartPar
The authinfo file consists of sections, led by a \sphinxcode{\sphinxupquote{{[}section{]}}}
header and followed by \sphinxcode{\sphinxupquote{name: value}} entries. The section headers
themselves are not used by S3QL (with the exception of the special \sphinxcode{\sphinxupquote{{[}\textless{}backend\textgreater{}\sphinxhyphen{}test{]}}}
sections used when running tests) but have to be unique within the file.

\sphinxAtStartPar
In each section, the following entries can be defined:
\begin{quote}\begin{description}
\sphinxlineitem{storage\sphinxhyphen{}url}
\sphinxAtStartPar
Specifies the storage url to which this section applies. If a
storage url starts with the string specified here, the section applies.

\sphinxlineitem{backend\sphinxhyphen{}login}
\sphinxAtStartPar
Specifies the username to use for authentication with the backend.

\sphinxlineitem{backend\sphinxhyphen{}password}
\sphinxAtStartPar
Specifies the password to use for authentication with the backend.

\sphinxlineitem{fs\sphinxhyphen{}passphrase}
\sphinxAtStartPar
Specifies the passphrase to use to decrypt the file system (if
it is encrypted).

\end{description}\end{quote}

\sphinxAtStartPar
Furthermore, command line parameters that are shared between all S3QL
commands that work with storage URL may be specified in this file as
well by omitting the leading dashes, e.g. the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options
notls}} parameter may be written into the authinfo file as
\sphinxcode{\sphinxupquote{backend\sphinxhyphen{}options: notls}}.

\sphinxAtStartPar
When reading the authinfo file, S3QL considers every applicable
section in order and uses the last value that it found for each entry.
For example, consider the following authentication file:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{g+ge}{[s3]}
\PYG{l}{storage\PYGZhy{}url: s3://}
\PYG{l}{backend\PYGZhy{}login: joe}
\PYG{l}{backend\PYGZhy{}password: notquitesecret}

\PYG{g+ge}{[fs1]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}first\PYGZhy{}bucket}
\PYG{l}{fs\PYGZhy{}passphrase: neitheristhis}

\PYG{g+ge}{[fs2]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}second\PYGZhy{}bucket}
\PYG{l}{fs\PYGZhy{}passphrase: swordfish}

\PYG{g+ge}{[fs3]}
\PYG{l}{storage\PYGZhy{}url: s3://joes\PYGZhy{}second\PYGZhy{}bucket/with\PYGZhy{}prefix}
\PYG{l}{backend\PYGZhy{}login: bill}
\PYG{l}{backend\PYGZhy{}password: bi23ll}
\PYG{l}{fs\PYGZhy{}passphrase: ll23bi}
\end{sphinxVerbatim}

\sphinxAtStartPar
With this authentication file, S3QL would try to log in as “joe”
whenever the s3 backend is used, except when accessing a storage url
that begins with “s3://joes\sphinxhyphen{}second\sphinxhyphen{}bucket/with\sphinxhyphen{}prefix”. In that case,
the last section becomes active and S3QL would use the “bill”
credentials. Furthermore, file system encryption passphrases will be used
for storage urls that start with “s3://joes\sphinxhyphen{}first\sphinxhyphen{}bucket” or
“s3://joes\sphinxhyphen{}second\sphinxhyphen{}bucket”.

\sphinxAtStartPar
The authentication file is parsed by the \sphinxhref{http://docs.python.org/library/configparser.html}{Python ConfigParser
module}.

\sphinxstepscope


\chapter{Contributed Programs}
\label{\detokenize{contrib:contributed-programs}}\label{\detokenize{contrib::doc}}
\sphinxAtStartPar
S3QL comes with a few contributed programs that are not part of the
core distribution (and are therefore not installed automatically by
default), but which may nevertheless be useful. These programs are in
the \sphinxcode{\sphinxupquote{contrib}} directory of the source distribution or in
\sphinxcode{\sphinxupquote{/usr/share/doc/s3ql/contrib}} if you installed S3QL from a package.


\section{benchmark.py}
\label{\detokenize{contrib:benchmark-py}}
\sphinxAtStartPar
This program measures S3QL write performance, uplink bandwidth and
compression speed to determine the limiting factor. It also gives
recommendation for compression algorithm and number of upload threads
to achieve maximum performance.


\section{pcp.py}
\label{\detokenize{contrib:pcp-py}}\label{\detokenize{contrib:pcp}}
\sphinxAtStartPar
\sphinxcode{\sphinxupquote{pcp.py}} is a wrapper program that starts several rsync processes to
copy directory trees in parallel. This is important because
transferring files in parallel significantly enhances performance when
copying data from an S3QL file system (see {\hyperref[\detokenize{tips:copy-performance}]{\sphinxcrossref{\DUrole{std,std-ref}{Improving copy performance}}}} for
details).

\sphinxAtStartPar
To recursively copy the directory \sphinxcode{\sphinxupquote{/mnt/home\sphinxhyphen{}backup}} into
\sphinxcode{\sphinxupquote{/home/joe}} using 8 parallel processes and preserving permissions,
you would execute

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{pcp.py \PYGZhy{}a \PYGZhy{}\PYGZhy{}processes=8 /mnt/home\PYGZhy{}backup/ /home/joe}
\end{sphinxVerbatim}


\section{s3ql\_backup.sh}
\label{\detokenize{contrib:s3ql-backup-sh}}
\sphinxAtStartPar
This is an example script that demonstrates how to set up a simple but
powerful backup solution using S3QL and \sphinxhref{http://samba.org/rsync}{rsync}.

\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{s3ql\_backup.sh}} script automates the following steps:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Mount the file system

\item {} 
\sphinxAtStartPar
Replicate the previous backup with {\hyperref[\detokenize{special:s3qlcp}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlcp}}}}

\item {} 
\sphinxAtStartPar
Update the new copy with the data from the backup source using rsync

\item {} 
\sphinxAtStartPar
Make the new backup immutable with {\hyperref[\detokenize{special:s3qllock}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qllock}}}}

\item {} 
\sphinxAtStartPar
Delete old backups that are no longer needed

\item {} 
\sphinxAtStartPar
Unmount the file system

\end{enumerate}

\sphinxAtStartPar
The backups are stored in directories of the form
\sphinxcode{\sphinxupquote{YYYY\sphinxhyphen{}MM\sphinxhyphen{}DD\_HH:mm:SS}} and the {\hyperref[\detokenize{contrib:expire-backups-py}]{\sphinxcrossref{expire\_backups.py}}} command is used to
delete old backups.


\section{expire\_backups.py}
\label{\detokenize{contrib:expire-backups-py}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups.py}} is a program to intelligently remove old
backups that are no longer needed.

\sphinxAtStartPar
To define what backups you want to keep for how long, you define a
number of \sphinxstyleemphasis{age ranges}. \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} ensures that you
will have at least one backup in each age range at all times. It will
keep exactly as many backups as are required for that and delete any
backups that become redundant.

\sphinxAtStartPar
Age ranges are specified by giving a list of range boundaries in terms
of backup cycles. Every time you create a new backup, the existing
backups age by one cycle.

\sphinxAtStartPar
Example: when \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is called with the age range
definition \sphinxcode{\sphinxupquote{1 3 7 14 31}}, it will guarantee that you always have the
following backups available:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
A backup that is 0 to 1 cycles old (i.e, the most recent backup)

\item {} 
\sphinxAtStartPar
A backup that is 1 to 3 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 3 to 7 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 7 to 14 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 14 to 31 cycles old

\end{enumerate}

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
If you do backups in fixed intervals, then one cycle will be
equivalent to the backup interval. The advantage of specifying the
age ranges in terms of backup cycles rather than days or weeks is
that it allows you to gracefully handle irregular backup intervals.
Imagine that for some reason you do not turn on your computer for
one month. Now all your backups are at least a month old, and if you
had specified the above backup strategy in terms of absolute ages,
they would all be deleted! Specifying age ranges in terms of backup
cycles avoids these sort of problems.
\end{sphinxadmonition}

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} usage is simple. It requires backups to be
stored in directories of the form \sphinxcode{\sphinxupquote{year\sphinxhyphen{}month\sphinxhyphen{}day\_hour:minute:seconds}}
(\sphinxcode{\sphinxupquote{YYYY\sphinxhyphen{}MM\sphinxhyphen{}DD\_HH:mm:ss}}) and works on all backups in the current
directory. So for the above backup strategy, the correct invocation
would be:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups.py 1 3 7 14 31}
\end{sphinxVerbatim}

\sphinxAtStartPar
When storing your backups on an S3QL file system, you probably want to
specify the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}use\sphinxhyphen{}s3qlrm}} option as well. This tells
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} to use the {\hyperref[\detokenize{special:s3qlrm}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlrm}}}} command to
delete directories.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} uses a “state file” to keep track which
backups are how many cycles old (since this cannot be inferred from
the dates contained in the directory names). The standard name for
this state file is \sphinxcode{\sphinxupquote{.expire\_backups.dat}}. If this file gets
damaged or deleted, \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} no longer knows the ages
of the backups and refuses to work. In this case you can use the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}reconstruct\sphinxhyphen{}state}} option to try to reconstruct the state
from the backup dates. However, the accuracy of this reconstruction
depends strongly on how rigorous you have been with making backups (it
is only completely correct if the time between subsequent backups has
always been exactly the same), so it’s generally a good idea not to
tamper with the state file.

\sphinxAtStartPar
For a full list of available options, run \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups.py
\sphinxhyphen{}\sphinxhyphen{}help}}.


\section{remove\_objects.py}
\label{\detokenize{contrib:remove-objects-py}}\label{\detokenize{contrib:remove-objects}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{remove\_objects.py}} is a program to remove a list of objects
from a storage backend. Since it acts on the backend\sphinxhyphen{}level, the
backend need not contain an S3QL file system.

\sphinxstepscope


\chapter{Tips \& Tricks}
\label{\detokenize{tips:tips-tricks}}\label{\detokenize{tips::doc}}

\section{SSH Backend}
\label{\detokenize{tips:ssh-backend}}\label{\detokenize{tips:ssh-tipp}}
\sphinxAtStartPar
By combining S3QL’s local backend with \sphinxhref{http://fuse.sourceforge.net/sshfs.html}{sshfs}, it is possible to store an
S3QL file system on arbitrary SSH servers: first mount the remote
target directory into the local filesystem,

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{sshfs user@my.server.com:/mnt/s3ql /mnt/sshfs}
\end{sphinxVerbatim}

\sphinxAtStartPar
and then give the mountpoint to S3QL as a local destination:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql local:///mnt/sshfs/myfsdata /mnt/s3ql}
\end{sphinxVerbatim}


\section{Permanently mounted backup file system}
\label{\detokenize{tips:permanently-mounted-backup-file-system}}
\sphinxAtStartPar
If you use S3QL as a backup file system, it can be useful to mount the
file system permanently (rather than just mounting it for a backup and
unmounting it afterwards). Especially if your file system becomes
large, this saves you long mount\sphinxhyphen{} and unmount times if you only want
to restore a single file.

\sphinxAtStartPar
If you decide to do so, you should make sure to
\begin{itemize}
\item {} 
\sphinxAtStartPar
Use {\hyperref[\detokenize{special:s3qllock}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qllock}}}} to ensure that backups are immutable
after they have been made.

\item {} 
\sphinxAtStartPar
Call {\hyperref[\detokenize{special:s3qlctrl}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlctrl backup\sphinxhyphen{}metadata}}}} right after a every
backup to make sure that the newest metadata is stored safely.

\end{itemize}


\section{Improving copy performance}
\label{\detokenize{tips:improving-copy-performance}}\label{\detokenize{tips:copy-performance}}
\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
The following applies only when copying data \sphinxstylestrong{from} an S3QL file
system, \sphinxstylestrong{not} when copying data \sphinxstylestrong{to} an S3QL file system.
\end{sphinxadmonition}

\sphinxAtStartPar
If you want to copy a lot of smaller files \sphinxstyleemphasis{from} an S3QL file system
(e.g. for a system restore) you will probably notice that the
performance is rather bad.

\sphinxAtStartPar
The reason for this is intrinsic to the way S3QL works. Whenever you
read a file, S3QL first has to retrieve this file over the network
from the backend. This takes a minimum amount of time (the network
latency), no matter how big or small the file is. So when you copy
lots of small files, 99\% of the time is actually spend waiting for
network data.

\sphinxAtStartPar
Theoretically, this problem is easy to solve: you just have to copy
several files at the same time. In practice, however, almost all unix
utilities (\sphinxcode{\sphinxupquote{cp}}, \sphinxcode{\sphinxupquote{rsync}}, \sphinxcode{\sphinxupquote{tar}} and friends) insist on copying
data one file at a time. This makes a lot of sense when copying data
on the local hard disk, but in case of S3QL this is really
unfortunate.

\sphinxAtStartPar
The best workaround that has been found so far is to copy files by
starting several rsync processes at once and use exclusion rules to
make sure that they work on different sets of files.

\sphinxAtStartPar
For example, the following script will start 3 rsync instances. The
first instance handles all filenames starting with a\sphinxhyphen{}f, the second the
filenames from g\sphinxhyphen{}l and the third covers the rest. The \sphinxcode{\sphinxupquote{+ */}} rule
ensures that every instance looks into all directories.

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{c}{\PYGZsh{}!/bin/bash}

\PYG{l}{RSYNC\PYGZus{}ARGS=\PYGZdq{}\PYGZhy{}aHv /mnt/s3ql/ /home/restore/\PYGZdq{}}

\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{}! }\PYG{g+ge}{[a\PYGZhy{}f]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}
\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{}! }\PYG{g+ge}{[g\PYGZhy{}l]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}
\PYG{l}{rsync \PYGZhy{}f \PYGZdq{}+ */\PYGZdq{} \PYGZhy{}f \PYGZdq{}\PYGZhy{} }\PYG{g+ge}{[a\PYGZhy{}l]}\PYG{l}{*\PYGZdq{} \PYGZdl{}RSYNC\PYGZus{}ARGS \PYGZam{}}

\PYG{l}{wait}
\end{sphinxVerbatim}

\sphinxAtStartPar
The optimum number of parallel processes depends on your network
connection and the size of the files that you want to transfer.
However, starting about 10 processes seems to be a good compromise
that increases performance dramatically in almost all situations.

\sphinxAtStartPar
S3QL comes with a script named \sphinxcode{\sphinxupquote{pcp.py}} in the \sphinxcode{\sphinxupquote{contrib}} directory
that can be used to transfer files in parallel without having to write
an explicit script first. See the description of {\hyperref[\detokenize{contrib:pcp}]{\sphinxcrossref{\DUrole{std,std-ref}{pcp.py}}}} for
details.

\sphinxstepscope


\chapter{Frequently Asked Questions}
\label{\detokenize{faq:frequently-asked-questions}}\label{\detokenize{faq::doc}}

\section{How can I improve the file system throughput?}
\label{\detokenize{faq:how-can-i-improve-the-file-system-throughput}}
\sphinxAtStartPar
There are three possible limiting factors for file system throughput:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Throughput from process through kernel to S3QL

\item {} 
\sphinxAtStartPar
Compression speed

\item {} 
\sphinxAtStartPar
Upload speed

\end{enumerate}

\sphinxAtStartPar
To figure out what’s the limiting factor in your case, you can use the
script \sphinxcode{\sphinxupquote{contrib/benchmark.py}} in the S3QL tarball. It will
determine the individual throughputs, and also give a recommendation
for compression algorithm and number of upload threads.


\section{Why does \sphinxstyleliteralintitle{\sphinxupquote{df}} show 1 TB of free space?}
\label{\detokenize{faq:why-does-df-show-1-tb-of-free-space}}
\sphinxAtStartPar
The file system size of S3QL is unlimited. However, most Unix
utilities (including \sphinxcode{\sphinxupquote{df}}) have no concept of an infinite size, so
S3QL has to provide some numerical value. Therefore, S3QL always
reports that the file system is 50\% free, but it also reports at least
1 TB. This means that if you store 1.5 TB you will have another 1.5 TB
free, but if you store just 2 MB, you will still have TB free rather
than just another 2 MB.


\section{Which operating systems are supported?}
\label{\detokenize{faq:which-operating-systems-are-supported}}
\sphinxAtStartPar
S3QL is developed on Linux, but should in principle work on all FUSE
supported operating systems that have the required Python modules. The
\sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} program does some tricks which may not work on
non\sphinxhyphen{}Linux systems, but you can always umount with the \sphinxcode{\sphinxupquote{fusermount
\sphinxhyphen{}u}} command provided by FUSE itself (the only difference is that it
will not block until all data has been uploaded to S3 but return
immediately). Please report your success (or failure) with other
operating systems on the mailing list, so that this information can be
extended.


\section{Is there a file size limit?}
\label{\detokenize{faq:is-there-a-file-size-limit}}
\sphinxAtStartPar
No, S3QL puts no limits on the size of your files. Any limits on
maximum object size imposed by the storage service do not matter for
S3QL, since files are automatically and transparently split into
smaller blocks.


\section{Suppose I want to make a small change in a very large file. Will S3QL download and re\sphinxhyphen{}upload the entire file?}
\label{\detokenize{faq:suppose-i-want-to-make-a-small-change-in-a-very-large-file-will-s3ql-download-and-re-upload-the-entire-file}}
\sphinxAtStartPar
No, S3QL splits files in blocks of a configurable size (default: 10 MB). So to make a small change in a big file, only one block needs to be transferred and not the entire file.


\section{I don’t quite understand this de\sphinxhyphen{}duplication feature…}
\label{\detokenize{faq:i-don-t-quite-understand-this-de-duplication-feature}}
\sphinxAtStartPar
The great thing about data de\sphinxhyphen{}duplication is that you don’t need to know anything about it and will still get all the benefits. Towards you, S3QL will behave and look exactly like a file system without data duplication. When talking to the storage backend, however, the S3QL will try to store identical data only once and thus occupy less space than an ordinary file system would.


\section{What does the “Transport endpoint not connected” error mean?}
\label{\detokenize{faq:what-does-the-transport-endpoint-not-connected-error-mean}}
\sphinxAtStartPar
It means that the file system has crashed. Please check \sphinxcode{\sphinxupquote{mount.log}} and
\sphinxcode{\sphinxupquote{mount.s3ql\_crit.log}} for a more useful error message and report a bug if
appropriate.

\sphinxAtStartPar
To make the mountpoint available again (i.e., unmount the crashed file system), use the
\sphinxcode{\sphinxupquote{fusermount \sphinxhyphen{}u}} command.

\sphinxAtStartPar
Before reporting a bug, please make sure that you’re using the most recent S3QL version.


\section{What does “Backend reports that fs is still mounted elsewhere, aborting” mean?}
\label{\detokenize{faq:what-does-backend-reports-that-fs-is-still-mounted-elsewhere-aborting-mean}}
\sphinxAtStartPar
It means that either the file systems has not been unmounted cleanly or is still
mounted. If you are sure that the file system is not mounted anywhere, run
\sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, ideally on the computer where the file system has been mounted most
recently.


\section{Can I access an S3QL file system on multiple computers simultaneously?}
\label{\detokenize{faq:can-i-access-an-s3ql-file-system-on-multiple-computers-simultaneously}}
\sphinxAtStartPar
To share a file system between several computers, you need to have a
method for these computers to communicate, so that they can tell each
other when they are changing data. However, S3QL is designed to store
data in cloud storage services which can not be used for this kind of
communication, because different computers may end up talking to
different servers in the cloud (and there is no way for S3QL to force
the cloud servers to synchronize quickly and often enough).

\sphinxAtStartPar
Therefore, the only way to share an S3QL file system among different
computers is to have one “master” computer that runs \sphinxstyleemphasis{mount.s3ql},
and then shares the mountpoint over a network file system like NFS or
CIFS.


\section{What block size should I use?}
\label{\detokenize{faq:what-block-size-should-i-use}}
\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}data\sphinxhyphen{}block\sphinxhyphen{}size}} option of the \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command determines the maximum
size of the data blocks that S3QL exchanges with the backend server.

\sphinxAtStartPar
For files smaller than the block size, this option has no effect. Files larger than the
maximum object size are split into multiple blocks. Whenever you upload or download data
from the backend, this is done in complete blocks. So if you have configured a maximum
object size of 10 MB, and want to read (or write) 5 bytes in a 100 MB file, you will still
download (or upload) roughly 10 MB of data (compression may reduce this amount). If you
decreased the maximum object size to 5 MB, you’d download/upload only about 5 MB.

\sphinxAtStartPar
On the other hand, if you want to read the whole 100 MB, and have
configured a block size of 10 MB, S3QL will have to send 10 separate
requests to the backend. With a maximum object size of 1 MB, there’d
be 100 separate requests. The larger the number of requests, the more
inefficient this becomes, because there is a fixed time associated
with the processing of each request. Also, many storage providers
charge a fixed amount for each request, so downloading 100 MB in one
request of 100 MB is cheaper than downloading it with 100 requests of
1 MB.

\sphinxAtStartPar
When choosing a block size, you have to find a balance
between these two effects. The bigger the size, the less requests will
be used, but the more data is transferred uselessly. The smaller the
size, the more requests will be used, but less traffic will be wasted.

\sphinxAtStartPar
Generally you should go with the default unless you have a good reason
to change it. When adjusting the value, keep in mind that it only
affects files larger than the block size. For example, if most of your
files are less than 1 MB, decreasing the block size from the
default 10 MB to 1 MB will have almost no effect.


\section{Is there a way to access the file system offline?}
\label{\detokenize{faq:is-there-a-way-to-access-the-file-system-offline}}
\sphinxAtStartPar
No, there is no way to do this. However, for most use\sphinxhyphen{}cases this
feature is actually not required. If you want to ensure that all data
in a directory is stored in the cloud \sphinxstyleemphasis{and} available offline, all you
need to do is store your data in a local directory that you
periodically synchronize to an S3QL mountpoint. For example, if you
would like to have an S3QL mountpoint with persistent cache at
\sphinxcode{\sphinxupquote{/mnt/data}}, you get can this as follows:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Mount a local block device at \sphinxcode{\sphinxupquote{/mnt/data}}.

\item {} 
\sphinxAtStartPar
Mount an S3QL file system at \sphinxcode{\sphinxupquote{/mnt/data\_online}} using a small
cache size and number of threads (eg. \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}threads
2 \sphinxhyphen{}\sphinxhyphen{}max\sphinxhyphen{}cache\sphinxhyphen{}entries 10}}).

\item {} 
\sphinxAtStartPar
Keep \sphinxcode{\sphinxupquote{/mnt/data\_online}} synchronized with \sphinxcode{\sphinxupquote{/mnt/data}}
by either
\begin{itemize}
\item {} 
\sphinxAtStartPar
Periodically running rsync, e.g. by calling
\sphinxcode{\sphinxupquote{rsync \sphinxhyphen{}aHA \sphinxhyphen{}\sphinxhyphen{}delete\sphinxhyphen{}during \sphinxhyphen{}\sphinxhyphen{}partial}} from cron, or

\item {} 
\sphinxAtStartPar
Continuously synchronizing using e.g. \sphinxhref{https://github.com/drunomics/syncd/blob/master/watch.sh}{watch.sh} or
\sphinxhref{https://code.google.com/p/lsyncd/}{lsyncd} (these programs use
\sphinxstyleemphasis{inotify} to constantly monitor the source directory and
immediately copy over any changes)

\end{itemize}

\item {} 
\sphinxAtStartPar
Now use \sphinxcode{\sphinxupquote{/mnt/data}} in the same way as you would use an S3QL
file system with a persistent cache.

\end{enumerate}

\sphinxstepscope


\chapter{Known Issues}
\label{\detokenize{issues:known-issues}}\label{\detokenize{issues::doc}}\begin{itemize}
\item {} 
\sphinxAtStartPar
S3QL de\sphinxhyphen{}duplicates data blocks based solely only on SHA256
checksums, without doing a byte\sphinxhyphen{}by\sphinxhyphen{}byte comparison of the data.
Since it is possible for two data blocks to have the same checksum
despite having different contents, this can lead to problems. If two
such blocks are stored in an S3QL file system, the data in one block
will be lost and replaced by the data in the other block. However,
the chances of this occurring for any two blocks are about 1 in 10\textasciicircum{}77
(2\textasciicircum{}256). For a file system that holds a total of 10\textasciicircum{}34 blocks, the
chances of a collision increase to about 1 in 10\textasciicircum{}9. Storing more
than 10\textasciicircum{}34 blocks (or about 10\textasciicircum{}25 TB with an (extremely small) block
size of 4 kB) is therefore not recommended. Being exceptionally
unlucky may also be a disadvantage.

\item {} 
\sphinxAtStartPar
S3QL does not support Access Control Lists (ACLs). There is nothing
fundamentally that prevents this, someone just has to write the
necessary code.

\item {} 
\sphinxAtStartPar
As of Linux kernel 3.5 S3QL file systems do not implement the “write
protect” bit on directories. In other words, even if a directory has
the write protect bit set, the owner of the directory can delete any
files and (empty) subdirectories inside it. This is a bug in the
FUSE kernel module
(cf. \sphinxurl{https://github.com/libfuse/libfuse/issues/23}) and needs to be
fixed in the kernel.  Unfortunately it does not look as if this is
going to be fixed anytime soon (as of 2016/2/28).

\item {} 
\sphinxAtStartPar
S3QL always updates file and directory access times as if the \sphinxcode{\sphinxupquote{relatime}}
mount option has been specified: the access time (“atime”) is only updated
if it is currently earlier than either the status change time
(“ctime”) or modification time (“mtime”).

\item {} 
\sphinxAtStartPar
S3QL directories always have an \sphinxcode{\sphinxupquote{st\_nlink}} value of 1. This may confuse
programs that rely on directories having \sphinxcode{\sphinxupquote{st\_nlink}} values of \sphinxstyleemphasis{(2 +
number of sub directories)}.

\sphinxAtStartPar
Note that this is not a bug in S3QL. Including sub directories in
the \sphinxcode{\sphinxupquote{st\_nlink}} value is a Unix convention, but by no means a
requirement. If an application blindly relies on this convention
being followed, then this is a bug in the application.

\sphinxAtStartPar
A prominent example are early versions of GNU find, which required
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}noleaf}} option to work correctly on S3QL file systems. This
bug has already been fixed in recent find versions.

\item {} 
\sphinxAtStartPar
The \sphinxcode{\sphinxupquote{umount}} and \sphinxcode{\sphinxupquote{fusermount \sphinxhyphen{}u}} commands will \sphinxstyleemphasis{not} block until all
data has been uploaded to the backend. (this is a FUSE limitation,
cf. \sphinxurl{https://github.com/libfuse/libfuse/issues/1}). If you use either
command to unmount an S3QL file system, you have to take care to
explicitly wait for the \sphinxcode{\sphinxupquote{mount.s3ql}} process to terminate before you
shut down or restart the system. Therefore it is generally not a
good idea to mount an S3QL file system in \sphinxcode{\sphinxupquote{/etc/fstab}} (you should
use a dedicated init script instead).

\item {} 
\sphinxAtStartPar
S3QL relies on the backends not to run out of space. This is a given
for big storage providers like Amazon S3 or Google Storage, but you
may stumble upon this if you use your own server or smaller providers.

\sphinxAtStartPar
If there is no space left in the backend, attempts to write more
data into the S3QL file system will fail and the file system will be
in an inconsistent state and require a file system check (and you
should make sure to make space available in the backend before
running the check).

\sphinxAtStartPar
Unfortunately, there is no way to handle insufficient space in the
backend without leaving the file system inconsistent. Since
S3QL first writes data into the cache, it can no longer return an
error when it later turns out that the cache can not be committed to
the backend.

\end{itemize}

\sphinxstepscope


\chapter{Manpages}
\label{\detokenize{man/index:manpages}}\label{\detokenize{man/index::doc}}
\sphinxAtStartPar
The man pages are installed with S3QL on your system and can be viewed
with the \sphinxstyleliteralstrong{\sphinxupquote{man}} command. For reference, they are also included
here in the User’s Guide.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command}
\label{\detokenize{man/mkfs:the-command-command}}\label{\detokenize{man/mkfs::doc}}

\subsection{Synopsis}
\label{\detokenize{man/mkfs:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mkfs.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/mkfs:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command creates a new file system in the location
specified by \sphinxstyleemphasis{storage url}. The format of the storage url depends on the backend
that is used. The S3QL User’s Guide should be consulted for a
description of the available backends.

\sphinxAtStartPar
Unless you have specified the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}plain}} option, \sphinxcode{\sphinxupquote{mkfs.s3ql}} will ask
you to enter an encryption password. This password will \sphinxstyleemphasis{not} be read
from an authentication file specified with the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}authfile}}
option to prevent accidental creation of an encrypted file system.


\subsection{Options}
\label{\detokenize{man/mkfs:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}L \textless{}name\textgreater{}]  
\sphinxAtStartPar
Filesystem label
\item [\sphinxhyphen{}\sphinxhyphen{}data\sphinxhyphen{}block\sphinxhyphen{}size \textless{}size\textgreater{}]  
\sphinxAtStartPar
Block size (in KiB) to use for storing file contents.
Files larger than this size will be stored in multiple
backend objects. Backend object size may be smaller
than this due to compression. Default: 1024 KiB.
\item [\sphinxhyphen{}\sphinxhyphen{}metadata\sphinxhyphen{}block\sphinxhyphen{}size \textless{}size\textgreater{}]  
\sphinxAtStartPar
Block size (in KiB) to use for storing filesystem
metadata. Backend object size may be smaller than this
due to compression. Default: 64 KiB.
\item [\sphinxhyphen{}\sphinxhyphen{}plain]  
\sphinxAtStartPar
Create unencrypted file system.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/mkfs:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}
\begin{quote}\begin{description}
\sphinxlineitem{3}
\sphinxAtStartPar
Invalid backend option.

\sphinxlineitem{11}
\sphinxAtStartPar
No such backend.

\sphinxlineitem{12}
\sphinxAtStartPar
Authentication file has insecure permissions.

\sphinxlineitem{13}
\sphinxAtStartPar
Unable to parse proxy settings.

\sphinxlineitem{14}
\sphinxAtStartPar
Invalid credentials (Authentication failed).

\sphinxlineitem{15}
\sphinxAtStartPar
No permission to access backend (Authorization denied).

\sphinxlineitem{16}
\sphinxAtStartPar
Invalid storage URL, specified location does not exist in backend.

\sphinxlineitem{19}
\sphinxAtStartPar
Unable to connect to backend, can’t resolve hostname.

\sphinxlineitem{45}
\sphinxAtStartPar
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/mkfs:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command}
\label{\detokenize{man/adm:the-command-command}}\label{\detokenize{man/adm::doc}}

\subsection{Synopsis}
\label{\detokenize{man/adm:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qladm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxAtStartPar
where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{passphrase}}, \sphinxstyleliteralstrong{\sphinxupquote{restore\sphinxhyphen{}metadata}},
\sphinxstyleliteralstrong{\sphinxupquote{clear}}, \sphinxstyleliteralstrong{\sphinxupquote{recover\sphinxhyphen{}key}} or \sphinxstyleliteralstrong{\sphinxupquote{upgrade}}.


\subsection{Description}
\label{\detokenize{man/adm:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command performs various operations on \sphinxstyleemphasis{unmounted} S3QL
file systems. The file system \sphinxstyleemphasis{must not be mounted} when using
\sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} or things will go wrong badly.

\sphinxAtStartPar
The storage url depends on the backend that is used. The S3QL User’s
Guide should be consulted for a description of the available backends.


\subsection{Options}
\label{\detokenize{man/adm:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}

\sphinxAtStartPar
Hint: run \sphinxcode{\sphinxupquote{s3qladm \textless{}action\textgreater{} \sphinxhyphen{}\sphinxhyphen{}help}} to get help on the additional arguments
that the different actions take.


\subsection{Actions}
\label{\detokenize{man/adm:actions}}
\sphinxAtStartPar
The following actions may be specified. Please consult the S3QL User’s Guide for more
detailed information.
\begin{description}
\sphinxlineitem{passphrase}
\sphinxAtStartPar
Changes the encryption passphrase of the file system.

\sphinxlineitem{restore\sphinxhyphen{}metadata}
\sphinxAtStartPar
Interactively restore backups of the file system metadata.

\sphinxlineitem{clear}
\sphinxAtStartPar
Delete the file system with all the stored data.

\sphinxlineitem{recover\sphinxhyphen{}key}
\sphinxAtStartPar
Recover master encryption key from external backup.

\sphinxlineitem{upgrade}
\sphinxAtStartPar
Upgrade the file system to the newest revision.

\end{description}


\subsection{Exit Codes}
\label{\detokenize{man/adm:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qladm}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}
\begin{quote}\begin{description}
\sphinxlineitem{3}
\sphinxAtStartPar
Invalid backend option.

\sphinxlineitem{10}
\sphinxAtStartPar
Could not open log file for writing.

\sphinxlineitem{11}
\sphinxAtStartPar
No such backend.

\sphinxlineitem{12}
\sphinxAtStartPar
Authentication file has insecure permissions.

\sphinxlineitem{13}
\sphinxAtStartPar
Unable to parse proxy settings.

\sphinxlineitem{14}
\sphinxAtStartPar
Invalid credentials (Authentication failed).

\sphinxlineitem{15}
\sphinxAtStartPar
No permission to access backend (Authorization denied).

\sphinxlineitem{16}
\sphinxAtStartPar
Invalid storage URL, specified location does not exist in backend.

\sphinxlineitem{17}
\sphinxAtStartPar
Wrong file system passphrase.

\sphinxlineitem{18}
\sphinxAtStartPar
No S3QL file system found at given storage URL.

\sphinxlineitem{19}
\sphinxAtStartPar
Unable to connect to backend, can’t resolve hostname.

\sphinxlineitem{45}
\sphinxAtStartPar
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/adm:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command}
\label{\detokenize{man/mount:the-command-command}}\label{\detokenize{man/mount::doc}}

\subsection{Synopsis}
\label{\detokenize{man/mount:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{mount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mount point\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/mount:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command mounts the S3QL file system stored in \sphinxstyleemphasis{storage
url} in the directory \sphinxstyleemphasis{mount point}. The format of the storage url depends on the
backend that is used. The S3QL User’s Guide should be consulted for a
description of the available backends.


\subsection{Options}
\label{\detokenize{man/mount:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/mount.log}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}compress \textless{}algorithm\sphinxhyphen{}lvl\textgreater{}]  
\sphinxAtStartPar
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma\sphinxhyphen{}6}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachesize \textless{}size\textgreater{}]  
\sphinxAtStartPar
Cache size in KiB (default: autodetect).
\item [\sphinxhyphen{}\sphinxhyphen{}max\sphinxhyphen{}cache\sphinxhyphen{}entries \textless{}num\textgreater{}]  
\sphinxAtStartPar
Maximum number of entries in cache (default:
autodetect). Each cache entry requires one file
descriptor, so if you increase this number you have to
make sure that your process file descriptor limit (as
set with \sphinxcode{\sphinxupquote{ulimit \sphinxhyphen{}n}}) is high enough (at least the
number of cache entries + 100).
\item [\sphinxhyphen{}\sphinxhyphen{}keep\sphinxhyphen{}cache]  
\sphinxAtStartPar
Do not purge locally cached files on exit.
\item [\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other]  
\sphinxAtStartPar
Normally, only the user who called \sphinxcode{\sphinxupquote{mount.s3ql}} can
access the mount point. This user then also has full
access to it, independent of individual file
permissions. If the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} option is
specified, other users can access the mount point as
well and individual file permissions are taken into
account for all users.
\item [\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root]  
\sphinxAtStartPar
Like \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}}, but restrict access to the
mounting user and the root user.
\item [\sphinxhyphen{}\sphinxhyphen{}dirty\sphinxhyphen{}block\sphinxhyphen{}upload\sphinxhyphen{}delay \textless{}seconds\textgreater{}]  
\sphinxAtStartPar
Upload delay for dirty blocks in seconds (default: 10
seconds).
\item [\sphinxhyphen{}\sphinxhyphen{}fg]  
\sphinxAtStartPar
Do not daemonize, stay in foreground
\item [\sphinxhyphen{}\sphinxhyphen{}fs\sphinxhyphen{}name FS\_NAME]  
\sphinxAtStartPar
Mount name passed to fuse, the name will be shown in
the first column of the system mount command output.
If not specified your storage url is used.
\item [\sphinxhyphen{}\sphinxhyphen{}systemd]  
\sphinxAtStartPar
Run as systemd unit. Consider specifying \textendash{}log none as
well to make use of journald.
\item [\sphinxhyphen{}\sphinxhyphen{}metadata\sphinxhyphen{}backup\sphinxhyphen{}interval \textless{}seconds\textgreater{}]  
\sphinxAtStartPar
Interval between metadata backups. Should the
filesystem crash while mounted, modifications made
after the most recent metadata backup may be lost.
During backups, the filesystem will be unresponsile.
Default: 6h
\item [\sphinxhyphen{}\sphinxhyphen{}threads \textless{}no\textgreater{}]  
\sphinxAtStartPar
Number of parallel upload threads to use (default:
auto).
\item [\sphinxhyphen{}\sphinxhyphen{}nfs]  
\sphinxAtStartPar
Enable some optimizations for exporting the file
system over NFS. (default: False)
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/mount:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}
\begin{quote}\begin{description}
\sphinxlineitem{3}
\sphinxAtStartPar
Invalid backend option.

\sphinxlineitem{10}
\sphinxAtStartPar
Could not open log file for writing.

\sphinxlineitem{11}
\sphinxAtStartPar
No such backend.

\sphinxlineitem{12}
\sphinxAtStartPar
Authentication file has insecure permissions.

\sphinxlineitem{13}
\sphinxAtStartPar
Unable to parse proxy settings.

\sphinxlineitem{14}
\sphinxAtStartPar
Invalid credentials (Authentication failed).

\sphinxlineitem{15}
\sphinxAtStartPar
No permission to access backend (Authorization denied).

\sphinxlineitem{16}
\sphinxAtStartPar
Invalid storage URL, specified location does not exist in backend.

\sphinxlineitem{17}
\sphinxAtStartPar
Wrong file system passphrase.

\sphinxlineitem{18}
\sphinxAtStartPar
No S3QL file system found at given storage URL.

\sphinxlineitem{19}
\sphinxAtStartPar
Unable to connect to backend, can’t resolve hostname.

\sphinxlineitem{30}
\sphinxAtStartPar
File system was not unmounted cleanly.

\sphinxlineitem{31}
\sphinxAtStartPar
File system appears to be mounted elsewhere.

\sphinxlineitem{32}
\sphinxAtStartPar
Unsupported file system revision (too old).

\sphinxlineitem{33}
\sphinxAtStartPar
Unsupported file system revision (too new).

\sphinxlineitem{34}
\sphinxAtStartPar
Insufficient free nodes, need to run \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}.

\sphinxlineitem{35}
\sphinxAtStartPar
Attempted to mount read\sphinxhyphen{}only, this is not supported.

\sphinxlineitem{36}
\sphinxAtStartPar
Mountpoint does not exist.

\sphinxlineitem{37}
\sphinxAtStartPar
Not enough available file descriptors.

\sphinxlineitem{39}
\sphinxAtStartPar
Unable to bind file system to mountpoint.

\sphinxlineitem{45}
\sphinxAtStartPar
Unable to access cache directory.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/mount:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command}
\label{\detokenize{man/stat:the-command-command}}\label{\detokenize{man/stat::doc}}

\subsection{Synopsis}
\label{\detokenize{man/stat:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlstat }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/stat:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command prints statistics about the S3QL file system mounted
at \sphinxcode{\sphinxupquote{mountpoint}}.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}}) the root user.


\subsection{Options}
\label{\detokenize{man/stat:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}raw]  
\sphinxAtStartPar
Do not pretty\sphinxhyphen{}print numbers
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/stat:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlstat}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/stat:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command}
\label{\detokenize{man/ctrl:the-command-command}}\label{\detokenize{man/ctrl::doc}}

\subsection{Synopsis}
\label{\detokenize{man/ctrl:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}action\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ ...}
\end{sphinxVerbatim}

\sphinxAtStartPar
where \sphinxcode{\sphinxupquote{action}} may be either of \sphinxstyleliteralstrong{\sphinxupquote{flushcache}}, \sphinxstyleliteralstrong{\sphinxupquote{dropcache}},
\sphinxstyleliteralstrong{\sphinxupquote{log}}, \sphinxstyleliteralstrong{\sphinxupquote{cachesize}} or \sphinxstyleliteralstrong{\sphinxupquote{backup\sphinxhyphen{}metadata}}.


\subsection{Description}
\label{\detokenize{man/ctrl:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command performs various actions on the S3QL file system mounted
in \sphinxcode{\sphinxupquote{mountpoint}}.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} can only be called by the user that mounted the file system and (if
the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}}) the
root user.

\sphinxAtStartPar
The following actions may be specified:
\begin{quote}
\begin{quote}\begin{description}
\sphinxlineitem{flushcache}
\sphinxAtStartPar
Write all modified blocks to the backend. The command blocks until the
cache is clean.

\sphinxlineitem{dropcache}
\sphinxAtStartPar
Flush the filesystem cache and then drop all contents (i.e., make the cache
empty).

\sphinxlineitem{log}
\sphinxAtStartPar
Change the amount of information that is logged. The complete syntax is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ log }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}level\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}module\PYGZgt{} ...]}
\end{sphinxVerbatim}

\sphinxAtStartPar
here \sphinxcode{\sphinxupquote{level}} is the desired new log level and may be either of \sphinxstyleemphasis{debug},
\sphinxstyleemphasis{info} or \sphinxstyleemphasis{warn}. One or more \sphinxcode{\sphinxupquote{module}} may only be specified with the
\sphinxstyleemphasis{debug} level and allow to restrict the debug output to just the listed
modules.

\sphinxlineitem{cachesize}
\sphinxAtStartPar
Changes the cache size of the file
system. This action requires an additional argument that specifies the new
cache size in KiB, so the complete command line is:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlctrl }\PYG{g+ge}{[options]}\PYG{l}{ cachesize }\PYG{n+nv}{\PYGZlt{}mountpoint\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}new\PYGZhy{}cache\PYGZhy{}size\PYGZgt{}}
\end{sphinxVerbatim}

\sphinxlineitem{backup\sphinxhyphen{}metadata}
\sphinxAtStartPar
Trigger a metadata backup.

\end{description}\end{quote}
\end{quote}


\subsection{Options}
\label{\detokenize{man/ctrl:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} command also accepts the following options, no matter
what specific action is being invoked:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}

\sphinxAtStartPar
Hint: run \sphinxcode{\sphinxupquote{s3qlctrl \textless{}action\textgreater{} \sphinxhyphen{}\sphinxhyphen{}help}} to get help on the additional arguments
that the different actions take.


\subsection{Exit Codes}
\label{\detokenize{man/ctrl:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlctrl}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/ctrl:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command}
\label{\detokenize{man/cp:the-command-command}}\label{\detokenize{man/cp::doc}}

\subsection{Synopsis}
\label{\detokenize{man/cp:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}source\PYGZhy{}dir\PYGZgt{}}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}dest\PYGZhy{}dir\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/cp:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command duplicates the directory tree \sphinxcode{\sphinxupquote{source\sphinxhyphen{}dir}}
into \sphinxcode{\sphinxupquote{dest\sphinxhyphen{}dir}} without physically copying the file contents.
Both source and destination must lie inside the same S3QL file system.

\sphinxAtStartPar
The replication will not take any additional space for the file contents. Metadata usage
will increase proportional to the amount of directory entries and inodes. Only if one of
directories is modified later on, the modified data will take additional storage space.

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{s3qlcp}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}})
the root user.

\sphinxAtStartPar
After the replication, both source and target directory will still be completely ordinary
directories. You can regard \sphinxcode{\sphinxupquote{\textless{}src\textgreater{}}} as a snapshot of \sphinxcode{\sphinxupquote{\textless{}target\textgreater{}}} or vice versa. However,
the most common usage of \sphinxcode{\sphinxupquote{s3qlcp}} is to regularly duplicate the same source directory, say
\sphinxcode{\sphinxupquote{documents}}, to different target directories. For a e.g. monthly replication, the target
directories would typically be named something like \sphinxcode{\sphinxupquote{documents\_January}} for the
replication in January, \sphinxcode{\sphinxupquote{documents\_February}} for the replication in February etc.  In this
case it is clear that the target directories should be regarded as snapshots of the source
directory.

\sphinxAtStartPar
Exactly the same effect could be achieved by an ordinary copy program like \sphinxcode{\sphinxupquote{cp
\sphinxhyphen{}a}}. However, this procedure would be orders of magnitude slower, because \sphinxcode{\sphinxupquote{cp}} would have
to read every file completely (so that S3QL had to fetch all the data over the network
from the backend) before writing them into the destination folder (at which point S3QL
would de\sphinxhyphen{}duplicate the data).


\subsubsection{Snapshotting vs Hardlinking}
\label{\detokenize{man/cp:snapshotting-vs-hardlinking}}
\sphinxAtStartPar
Snapshot support in S3QL is inspired by the hardlinking feature that
is offered by programs like \sphinxhref{http://www.samba.org/rsync}{rsync} or
\sphinxhref{http://savannah.nongnu.org/projects/storebackup}{storeBackup}.
These programs can create a hardlink instead of copying a file if an
identical file already exists in the backup. However, using hardlinks
has disadvantages:
\begin{itemize}
\item {} 
\sphinxAtStartPar
backups and restores always have to be made with a special program
that takes care of the hardlinking. The backups must not be touched
by any other programs (they may make changes that inadvertently
affect other hardlinked files)

\item {} 
\sphinxAtStartPar
special care needs to be taken to handle files which are already
hardlinked (the restore program needs to know that the hardlink was
not introduced by the backup program)

\end{itemize}

\sphinxAtStartPar
S3QL snapshots do not have these problems, and they can be used with
any backup program.


\subsection{Options}
\label{\detokenize{man/cp:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/cp:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlcp}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/cp:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command}
\label{\detokenize{man/rm:the-command-command}}\label{\detokenize{man/rm::doc}}

\subsection{Synopsis}
\label{\detokenize{man/rm:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qlrm }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/rm:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command recursively deletes files and directories on an S3QL file
system. Although \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} is faster than using e.g.  \sphinxstyleliteralstrong{\sphinxupquote{rm \sphinxhyphen{}r}}, the main reason
for its existence is that it allows you to delete immutable trees (which can be created
with \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}}) as well.

\sphinxAtStartPar
Be warned that there is no additional confirmation. The directory will
be removed entirely and immediately.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}}) the root user.


\subsection{Options}
\label{\detokenize{man/rm:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/rm:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/rm:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command}
\label{\detokenize{man/lock:the-command-command}}\label{\detokenize{man/lock::doc}}

\subsection{Synopsis}
\label{\detokenize{man/lock:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3qllock }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}directory\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/lock:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command makes a directory tree in an S3QL file
system immutable. Immutable trees can no longer be changed in any way
whatsoever. You can not add new files or directories and you can not
change or delete existing files and directories. The only way to get
rid of an immutable tree is to use the \sphinxstyleliteralstrong{\sphinxupquote{s3qlrm}} command.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} can only be called by the user that mounted the file system
and (if the file system was mounted with \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}other}} or
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}allow\sphinxhyphen{}root}}) the root user.


\subsection{Rationale}
\label{\detokenize{man/lock:rationale}}
\sphinxAtStartPar
Immutability is a feature designed for backups. Traditionally, backups have been made on
external tape drives. Once a backup was made, the tape drive was removed and locked away
somewhere. This means that the contents of the backup are permanently fixed. Nothing
(short of physical destruction) can change or delete files in the backup.

\sphinxAtStartPar
In contrast, when backing up into an online storage system like S3QL,
all backups are available every time the file system is mounted.
Nothing prevents a file in an old backup from being changed again
later on. In the worst case, this may make your entire backup system
worthless. Imagine that your system gets infected by a virus
that simply deletes all files it can find \textendash{} if the virus is active
while the backup file system is mounted, the virus will destroy all
backups together with the originals.

\sphinxAtStartPar
Even in the absence of malware,, being able to change a backup after it has been made is
generally not a good idea. A common S3QL use case is to keep the file system mounted at
all times and periodically create backups with \sphinxstyleliteralstrong{\sphinxupquote{rsync \sphinxhyphen{}a}}. This allows every user
to recover her files from a backup without having to call the system
administrator. However, this also allows every user to accidentally change or delete files
\sphinxstyleemphasis{in} one of the old backups.

\sphinxAtStartPar
Making a backup immutable protects you against all these problems.
Unless you happen to run into a virus that was specifically programmed
to attack S3QL file systems, backups can be neither deleted nor
changed after they have been made immutable.


\subsection{Options}
\label{\detokenize{man/lock:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/lock:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3qllock}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/lock:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command}
\label{\detokenize{man/umount:the-command-command}}\label{\detokenize{man/umount::doc}}

\subsection{Synopsis}
\label{\detokenize{man/umount:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{umount.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}mount point\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/umount:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command unmounts the S3QL file system mounted in the
directory \sphinxstyleemphasis{mount point} and blocks until all data has been uploaded to
the storage backend.

\sphinxAtStartPar
Only the user who mounted the file system with \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}
is able to unmount it with \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}}. If you are root and want to
unmount an S3QL file system mounted by an ordinary user, you have to
use the \sphinxstyleliteralstrong{\sphinxupquote{fusermount \sphinxhyphen{}u}} or \sphinxstyleliteralstrong{\sphinxupquote{umount}} command instead.
Note that these commands do not block until all data has been
uploaded, so if you use them instead of \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} then
you should manually wait for the \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process to
terminate before shutting down the system.


\subsection{Options}
\label{\detokenize{man/umount:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}lazy, \sphinxhyphen{}z]  
\sphinxAtStartPar
Lazy umount. Detaches the file system immediately,
even if there are still open files. The data will be
uploaded in the background once all open files have
been closed.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/umount:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{umount.s3ql}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/umount:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command}
\label{\detokenize{man/fsck:the-command-command}}\label{\detokenize{man/fsck::doc}}

\subsection{Synopsis}
\label{\detokenize{man/fsck:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{fsck.s3ql }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/fsck:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command checks the file system in the location specified by \sphinxstyleemphasis{storage url}
for errors and attempts to repair any problems. The format of the storage url depends on
the backend that is used. The S3QL User’s Guide should be consulted for a description of
the available backends.


\subsection{Options}
\label{\detokenize{man/fsck:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/fsck.log}}
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}compress \textless{}algorithm\sphinxhyphen{}lvl\textgreater{}]  
\sphinxAtStartPar
Compression algorithm and compression level to use
when storing new data. \sphinxstyleemphasis{algorithm} may be any of
\sphinxcode{\sphinxupquote{lzma}}, \sphinxcode{\sphinxupquote{bzip2}}, \sphinxcode{\sphinxupquote{zlib}}, or none. \sphinxstyleemphasis{lvl} may be any
integer from 0 (fastest) to 9 (slowest). Default:
\sphinxcode{\sphinxupquote{lzma\sphinxhyphen{}6}}
\item [\sphinxhyphen{}\sphinxhyphen{}keep\sphinxhyphen{}cache]  
\sphinxAtStartPar
Do not purge locally cached files on exit.
\item [\sphinxhyphen{}\sphinxhyphen{}batch]  
\sphinxAtStartPar
If user input is required, exit without prompting.
\item [\sphinxhyphen{}\sphinxhyphen{}force]  
\sphinxAtStartPar
Force checking even if file system is marked clean.
\item [\sphinxhyphen{}\sphinxhyphen{}force\sphinxhyphen{}remote]  
\sphinxAtStartPar
Force use of remote metadata even when this would
likely result in data loss.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/fsck:exit-codes}}
\sphinxAtStartPar
If \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}} found any file system errors (no matter if they were
corrected or not), the exit code will be 128 plus one of the codes
listed below. If no errors were found, the following exit codes are
used as\sphinxhyphen{}is:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}
\begin{quote}\begin{description}
\sphinxlineitem{3}
\sphinxAtStartPar
Invalid backend option.

\sphinxlineitem{10}
\sphinxAtStartPar
Could not open log file for writing.

\sphinxlineitem{11}
\sphinxAtStartPar
No such backend.

\sphinxlineitem{12}
\sphinxAtStartPar
Authentication file has insecure permissions.

\sphinxlineitem{13}
\sphinxAtStartPar
Unable to parse proxy settings.

\sphinxlineitem{14}
\sphinxAtStartPar
Invalid credentials (Authentication failed).

\sphinxlineitem{15}
\sphinxAtStartPar
No permission to access backend (Authorization denied).

\sphinxlineitem{16}
\sphinxAtStartPar
Invalid storage URL, specified location does not exist in backend.

\sphinxlineitem{17}
\sphinxAtStartPar
Wrong file system passphrase.

\sphinxlineitem{18}
\sphinxAtStartPar
No S3QL file system found at given storage URL.

\sphinxlineitem{19}
\sphinxAtStartPar
Unable to connect to backend, can’t resolve hostname.

\sphinxlineitem{32}
\sphinxAtStartPar
Unsupported file system revision (too old).

\sphinxlineitem{33}
\sphinxAtStartPar
Unsupported file system revision (too new).

\sphinxlineitem{40}
\sphinxAtStartPar
Cannot check mounted file system.

\sphinxlineitem{41}
\sphinxAtStartPar
User input required, but running in batch mode.

\sphinxlineitem{42}
\sphinxAtStartPar
File system check aborted by user.

\sphinxlineitem{43}
\sphinxAtStartPar
Local metadata is corrupted.

\sphinxlineitem{44}
\sphinxAtStartPar
Uncorrectable errors found.

\sphinxlineitem{45}
\sphinxAtStartPar
Unable to access cache directory.

\sphinxlineitem{128}
\sphinxAtStartPar
This error code will be \sphinxstyleemphasis{added} to one of the codes above if any
file system errors have been found (no matter if they were
corrected or not).

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/fsck:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command}
\label{\detokenize{man/oauth_client:the-command-command}}\label{\detokenize{man/oauth_client:oauth-client}}\label{\detokenize{man/oauth_client::doc}}

\subsection{Synopsis}
\label{\detokenize{man/oauth_client:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}oauth\PYGZus{}client }\PYG{g+ge}{[options]}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/oauth_client:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command may be used to obtain OAuth2 authentication
tokens for use with Google Storage. It requests a “user code” from
Google which has to be pasted into the browser to complete the
authentication process interactively. Once authentication in the
browser has been completed, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} displays the OAuth2 refresh
token.

\sphinxAtStartPar
When combined with the special username \sphinxcode{\sphinxupquote{oauth2}}, the refresh token
can be used as a backend passphrase when using the Google Storage S3QL
backend.


\subsection{Options}
\label{\detokenize{man/oauth_client:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/oauth_client:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_oauth\_client}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/oauth_client:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command}
\label{\detokenize{man/verify:the-command-command}}\label{\detokenize{man/verify::doc}}

\subsection{Synopsis}
\label{\detokenize{man/verify:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{s3ql\PYGZus{}verify }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}storage url\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/verify:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command verifies all data in the file system.  In
contrast to \sphinxstyleliteralstrong{\sphinxupquote{fsck.s3ql}}, \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} does not trust the object
listing returned by the backend, but actually attempts to retrieve
every object. It therefore takes a lot longer.

\sphinxAtStartPar
The format of \sphinxcode{\sphinxupquote{\textless{}storage url\textgreater{}}} depends on the backend that is
used. The S3QL User’s Guide should be consulted for a description of
the available backends.


\subsection{Options}
\label{\detokenize{man/verify:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} command accepts the following options.
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}cachedir \textless{}path\textgreater{}]  
\sphinxAtStartPar
Store cached data in this directory (default:
\sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql)}}
\item [\sphinxhyphen{}\sphinxhyphen{}backend\sphinxhyphen{}options \textless{}options\textgreater{}]  
\sphinxAtStartPar
Backend specific options (separate by commas). See
backend documentation for available options.
\item [\sphinxhyphen{}\sphinxhyphen{}authfile \textless{}path\textgreater{}]  
\sphinxAtStartPar
Read authentication credentials from this file
(default: \sphinxcode{\sphinxupquote{\textasciitilde{}/.s3ql/authinfo2)}}
\item [\sphinxhyphen{}\sphinxhyphen{}missing\sphinxhyphen{}file \textless{}name\textgreater{}]  
\sphinxAtStartPar
File to store keys of missing objects.
\item [\sphinxhyphen{}\sphinxhyphen{}corrupted\sphinxhyphen{}file \textless{}name\textgreater{}]  
\sphinxAtStartPar
File to store keys of corrupted objects.
\item [\sphinxhyphen{}\sphinxhyphen{}data]  
\sphinxAtStartPar
Read every object completely, instead of checking just
the metadata.
\item [\sphinxhyphen{}\sphinxhyphen{}parallel PARALLEL]  
\sphinxAtStartPar
Number of connections to use in parallel.
\item [\sphinxhyphen{}\sphinxhyphen{}start\sphinxhyphen{}with \textless{}n\textgreater{}]  
\sphinxAtStartPar
Skip over first \textless{}n\textgreater{} objects and with verifying object
\textless{}n\textgreater{}+1.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/verify:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{s3ql\_verify}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}
\begin{quote}\begin{description}
\sphinxlineitem{3}
\sphinxAtStartPar
Invalid backend option.

\sphinxlineitem{10}
\sphinxAtStartPar
Could not open log file for writing.

\sphinxlineitem{11}
\sphinxAtStartPar
No such backend.

\sphinxlineitem{12}
\sphinxAtStartPar
Authentication file has insecure permissions.

\sphinxlineitem{13}
\sphinxAtStartPar
Unable to parse proxy settings.

\sphinxlineitem{14}
\sphinxAtStartPar
Invalid credentials (Authentication failed).

\sphinxlineitem{15}
\sphinxAtStartPar
No permission to access backend (Authorization denied).

\sphinxlineitem{16}
\sphinxAtStartPar
Invalid storage URL, specified location does not exist in backend.

\sphinxlineitem{17}
\sphinxAtStartPar
Wrong file system passphrase.

\sphinxlineitem{18}
\sphinxAtStartPar
No S3QL file system found at given storage URL.

\sphinxlineitem{19}
\sphinxAtStartPar
Unable to connect to backend, can’t resolve hostname.

\sphinxlineitem{32}
\sphinxAtStartPar
Unsupported file system revision (too old).

\sphinxlineitem{33}
\sphinxAtStartPar
Unsupported file system revision (too new).

\sphinxlineitem{45}
\sphinxAtStartPar
Unable to access cache directory.

\sphinxlineitem{46}
\sphinxAtStartPar
The file system data was verified, and some objects were found
to be missing or corrupted.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/verify:see-also}}
\sphinxAtStartPar
The S3QL homepage is at \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxAtStartPar
The full S3QL documentation should also be installed somewhere on your
system, common locations are \sphinxcode{\sphinxupquote{/usr/share/doc/s3ql}} or
\sphinxcode{\sphinxupquote{/usr/local/doc/s3ql}}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command}
\label{\detokenize{man/pcp:the-command-command}}\label{\detokenize{man/pcp::doc}}

\subsection{Synopsis}
\label{\detokenize{man/pcp:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{pcp }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}source\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}source\PYGZgt{} ...]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}destination\PYGZgt{}}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/pcp:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command is a is a wrapper that starts several
\sphinxstyleliteralstrong{\sphinxupquote{sync}} processes to copy directory trees in parallel. This is
allows much better copying performance on file system that have
relatively high latency when retrieving individual files like S3QL.

\sphinxAtStartPar
\sphinxstylestrong{Note}: Using this program only improves performance when copying
\sphinxstyleemphasis{from} an S3QL file system. When copying \sphinxstyleemphasis{to} an S3QL file system,
using \sphinxstyleliteralstrong{\sphinxupquote{pcp}} is more likely to \sphinxstyleemphasis{decrease} performance.


\subsection{Options}
\label{\detokenize{man/pcp:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{pcp}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}a]  
\sphinxAtStartPar
Pass \sphinxhyphen{}aHAX option to rsync.
\item [\sphinxhyphen{}\sphinxhyphen{}processes \textless{}no\textgreater{}]  
\sphinxAtStartPar
Number of rsync processes to use (default: 10).
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/pcp:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{pcp}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/pcp:see-also}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{pcp}} is shipped as part of S3QL, \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxstepscope


\section{The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command}
\label{\detokenize{man/expire_backups:the-command-command}}\label{\detokenize{man/expire_backups::doc}}

\subsection{Synopsis}
\label{\detokenize{man/expire_backups:synopsis}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups }\PYG{g+ge}{[options]}\PYG{l}{ }\PYG{n+nv}{\PYGZlt{}age\PYGZgt{}}\PYG{l}{ }\PYG{g+ge}{[\PYGZlt{}age\PYGZgt{} ...]}
\end{sphinxVerbatim}


\subsection{Description}
\label{\detokenize{man/expire_backups:description}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command intelligently remove old backups that are no
longer needed.

\sphinxAtStartPar
To define what backups you want to keep for how long, you define a
number of \sphinxstyleemphasis{age ranges}. \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} ensures that you
will have at least one backup in each age range at all times. It will
keep exactly as many backups as are required for that and delete any
backups that become redundant.

\sphinxAtStartPar
Age ranges are specified by giving a list of range boundaries in terms
of backup cycles. Every time you create a new backup, the existing
backups age by one cycle.

\sphinxAtStartPar
Example: when \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is called with the age range
definition \sphinxcode{\sphinxupquote{1 3 7 14 31}}, it will guarantee that you always have the
following backups available:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
A backup that is 0 to 1 cycles old (i.e, the most recent backup)

\item {} 
\sphinxAtStartPar
A backup that is 1 to 3 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 3 to 7 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 7 to 14 cycles old

\item {} 
\sphinxAtStartPar
A backup that is 14 to 31 cycles old

\end{enumerate}

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
If you do backups in fixed intervals, then one cycle will be
equivalent to the backup interval. The advantage of specifying the
age ranges in terms of backup cycles rather than days or weeks is
that it allows you to gracefully handle irregular backup intervals.
Imagine that for some reason you do not turn on your computer for
one month. Now all your backups are at least a month old, and if you
had specified the above backup strategy in terms of absolute ages,
they would all be deleted! Specifying age ranges in terms of backup
cycles avoids these sort of problems.
\end{sphinxadmonition}

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} usage is simple. It requires backups to be
stored in directories of the form \sphinxcode{\sphinxupquote{year\sphinxhyphen{}month\sphinxhyphen{}day\_hour:minute:seconds}}
(\sphinxcode{\sphinxupquote{YYYY\sphinxhyphen{}MM\sphinxhyphen{}DD\_HH:mm:ss}}) and works on all backups in the current
directory. So for the above backup strategy, the correct invocation
would be:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{l}{expire\PYGZus{}backups.py 1 3 7 14 31}
\end{sphinxVerbatim}

\sphinxAtStartPar
When storing your backups on an S3QL file system, you probably want to
specify the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}use\sphinxhyphen{}s3qlrm}} option as well. This tells
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} to use the {\hyperref[\detokenize{special:s3qlrm}]{\sphinxcrossref{\DUrole{std,std-ref}{s3qlrm}}}} command to
delete directories.

\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} uses a “state file” to keep track which
backups are how many cycles old (since this cannot be inferred from
the dates contained in the directory names). The standard name for
this state file is \sphinxcode{\sphinxupquote{.expire\_backups.dat}}. If this file gets
damaged or deleted, \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} no longer knows the ages
of the backups and refuses to work. In this case you can use the
\sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}reconstruct\sphinxhyphen{}state}} option to try to reconstruct the state
from the backup dates. However, the accuracy of this reconstruction
depends strongly on how rigorous you have been with making backups (it
is only completely correct if the time between subsequent backups has
always been exactly the same), so it’s generally a good idea not to
tamper with the state file.


\subsection{Options}
\label{\detokenize{man/expire_backups:options}}
\sphinxAtStartPar
The \sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} command accepts the following options:
\begin{quote}
\begin{optionlist}{3cm}
\item [\sphinxhyphen{}\sphinxhyphen{}quiet]  
\sphinxAtStartPar
be really quiet
\item [\sphinxhyphen{}\sphinxhyphen{}log \textless{}target\textgreater{}]  
\sphinxAtStartPar
Destination for log messages. Specify \sphinxcode{\sphinxupquote{none}} for
standard output or \sphinxcode{\sphinxupquote{syslog}} for the system logging
daemon. Anything else will be interpreted as a file
name. Log files will be rotated when they reach 1 MiB,
and at most 5 old log files will be kept. Default:
\sphinxcode{\sphinxupquote{None}}
\item [\sphinxhyphen{}\sphinxhyphen{}debug\sphinxhyphen{}modules \textless{}modules\textgreater{}]  
\sphinxAtStartPar
Activate debugging output from specified modules (use
commas to separate multiple modules, ‘all’ for
everything). Debug messages will be written to the
target specified by the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}debug]  
\sphinxAtStartPar
Activate debugging output from all S3QL modules. Debug
messages will be written to the target specified by
the \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}log}} option.
\item [\sphinxhyphen{}\sphinxhyphen{}version]  
\sphinxAtStartPar
just print program version and exit
\item [\sphinxhyphen{}\sphinxhyphen{}state \textless{}file\textgreater{}]  
\sphinxAtStartPar
File to save state information in (default:
“.expire\_backups.dat”)
\item [\sphinxhyphen{}n]  
\sphinxAtStartPar
Dry run. Just show which backups would be deleted.
\item [\sphinxhyphen{}p \textless{}N\textgreater{}, \sphinxhyphen{}\sphinxhyphen{}proportion\sphinxhyphen{}delete \textless{}N\textgreater{}]  
\sphinxAtStartPar
Maximum proportion of backups to delete (between 0 and
1, default: 0.5)
\item [\sphinxhyphen{}\sphinxhyphen{}reconstruct\sphinxhyphen{}state]  
\sphinxAtStartPar
Try to reconstruct a missing state file from backup
dates.
\item [\sphinxhyphen{}\sphinxhyphen{}use\sphinxhyphen{}s3qlrm]  
\sphinxAtStartPar
Use \sphinxcode{\sphinxupquote{s3qlrm}} command to delete backups.
\end{optionlist}
\end{quote}


\subsection{Exit Codes}
\label{\detokenize{man/expire_backups:exit-codes}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} may terminate with the following exit codes:
\begin{quote}\begin{description}
\sphinxlineitem{0}
\sphinxAtStartPar
Everything went well.

\sphinxlineitem{1}
\sphinxAtStartPar
An unexpected error occurred. This may indicate a bug in the
program.

\sphinxlineitem{2}
\sphinxAtStartPar
Invalid command line argument or configuration file key.

\end{description}\end{quote}


\subsection{See Also}
\label{\detokenize{man/expire_backups:see-also}}
\sphinxAtStartPar
\sphinxstyleliteralstrong{\sphinxupquote{expire\_backups}} is shipped as part of S3QL, \sphinxurl{https://github.com/s3ql/s3ql/}.

\sphinxstepscope


\chapter{Further Resources / Getting Help}
\label{\detokenize{resources:further-resources-getting-help}}\label{\detokenize{resources:resources}}\label{\detokenize{resources::doc}}
\sphinxAtStartPar
If you have questions or problems with S3QL that you weren’t able to
resolve with this manual, you might want to consider the following other resources:
\begin{itemize}
\item {} 
\sphinxAtStartPar
The \sphinxhref{https://github.com/s3ql/s3ql/wiki}{S3QL Wiki}

\item {} 
\sphinxAtStartPar
The \sphinxhref{http://groups.google.com/group/s3ql}{S3QL Mailing List}. You
can subscribe by sending a mail to
\sphinxhref{mailto:s3ql+subscribe@googlegroups.com}{s3ql+subscribe@googlegroups.com}.

\end{itemize}

\sphinxAtStartPar
Please report any bugs you may encounter in the \sphinxhref{https://github.com/s3ql/s3ql/issues}{Issue Tracker}.

\sphinxstepscope


\chapter{Implementation Details}
\label{\detokenize{impl_details:implementation-details}}\label{\detokenize{impl_details:impl-details}}\label{\detokenize{impl_details::doc}}
\sphinxAtStartPar
This section provides some background information on how S3QL works
internally. Reading this section is not necessary to use S3QL.


\section{Metadata Storage}
\label{\detokenize{impl_details:metadata-storage}}
\sphinxAtStartPar
Like most unix filesystems, S3QL has a concept of inodes.

\sphinxAtStartPar
The contents of directory inodes (aka the names and inodes of the
files and sub directories contained in a directory) are stored
directly in an \sphinxhref{http://www.sqlite.org/}{SQLite} database. This database is split into a number
of storage objects that are downloaded when the file
system is mounted and uploaded periodically in the background and when
the file system is unmounted. This has two implications:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
The entire file system tree can be read from the
database. Fetching/storing storage objects from/in the storage
backend is only required to access the contents of files (or, more
precisely, inodes). This makes most file system operations very
fast because no data has to be send over the network.

\item {} 
\sphinxAtStartPar
An S3QL filesystem can only be mounted on one computer at a time,
using a single \sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}} process. Otherwise changes made in
one mountpoint will invariably be overwritten when the second mount
point is unmounted.

\end{enumerate}

\sphinxAtStartPar
Sockets, FIFOs and character devices do not need any additional
storage, all information about them is contained in the database.


\section{Data Storage}
\label{\detokenize{impl_details:data-storage}}
\sphinxAtStartPar
The contents of file inodes are split into individual blocks. The
maximum size of a block is specified when the file system is created
and cannot be changed afterwards. Every block is stored as an
individual object in the backend, and the mapping from inodes to
blocks and from blocks to objects is stored in the database.

\sphinxAtStartPar
While the file system is mounted, blocks are cached locally.

\sphinxAtStartPar
Blocks can also be compressed and encrypted before they are stored in
the storage backend. This happens during upload, i.e. the cached data
is unencrypted and uncompressed.

\sphinxAtStartPar
If some files have blocks with identical contents, the blocks will be
stored in the same backend object (i.e., the data is only stored
once).


\section{Data De\sphinxhyphen{}Duplication}
\label{\detokenize{impl_details:data-de-duplication}}
\sphinxAtStartPar
Instead of uploading every block, S3QL first computes a checksum (a
SHA256 hash) to check if an identical blocks has already been stored
in an backend object. If that is the case, the new block will be
linked to the existing object instead of being uploaded.

\sphinxAtStartPar
This procedure is invisible for the user and the contents of the block
can still be changed. If several blocks share a backend object and one
of the blocks is changed, the changed block is automatically stored in
a new object (so that the contents of the other block remain
unchanged).


\section{Caching}
\label{\detokenize{impl_details:caching}}
\sphinxAtStartPar
When an application tries to read or write from a file, S3QL
determines the block that contains the required part of the file and
retrieves it from the backend or creates it if it does not yet exist.
The block is then held in the cache directory. It is committed to S3
when it has not been accessed for more than a few seconds. Blocks are
removed from the cache only when the maximum cache size is reached.

\sphinxAtStartPar
When the file system is unmounted, all modified blocks are written to
the backend and the cache is cleaned.


\section{Encryption}
\label{\detokenize{impl_details:encryption}}
\sphinxAtStartPar
When the file system is created, \sphinxstyleliteralstrong{\sphinxupquote{mkfs.s3ql}} generates a 256 bit
master key by reading from \sphinxcode{\sphinxupquote{/dev/random}}. The master key is
encrypted with the passphrase that is entered by the user, and then
stored with the rest of the file system data. Since the passphrase is
only used to access the master key (which is used to encrypt the
actual file system data), the passphrase can easily be changed.

\sphinxAtStartPar
Data is encrypted with a new session key for each object and each
upload. The session key is generated by appending a nonce to the
master key and then calculating the SHA256 hash. The nonce is
generated by concatenating the object id and the current UTC time as a
32 bit float. The precision of the time is given by the Python \sphinxhref{http://docs.python.org/library/time.html\#time.time}{time()} function and
usually at least 1 millisecond. The SHA256 implementation is included
in the Python standard library.

\sphinxAtStartPar
Once the session key has been calculated, a SHA256 HMAC is calculated
over the data that is to be uploaded. Afterwards, the data is
compressed (unless \sphinxcode{\sphinxupquote{\sphinxhyphen{}\sphinxhyphen{}compress none}} was passed to
\sphinxstyleliteralstrong{\sphinxupquote{mount.s3ql}}) and the HMAC inserted at the beginning. Both HMAC
and compressed data are then encrypted using 256 bit AES in CTR
mode using \sphinxhref{http://www.pycrypto.org/}{PyCrypto}.  Finally, the nonce is
inserted in front of the encrypted data and HMAC, and the packet is
send to the backend as a new S3 object.



\renewcommand{\indexname}{Index}
\printindex
\end{document}