Introduction to AF's Backup system
          ==================================


This file gives a short introduction into AF's backup system,
it's architecture (wow !), the ideas behind it and how it can
be used to serve the individual needs.


Client-Server
-------------

Yes, it's a client-server system. The server is a machine, that
has access to some kind of storage media, usually a tape drive.
This server side knows, how to handle this device, how many
cartridges are in use and what to do, when a cartridge gets full
or other special cases occur. What the serverside does not
know is anything, that has to do with directory structures, files,
attributes of the stored files or even their names. This is all
handled on the client side. The server only knows data streams.
The client demands from the server the functionality to pass him
e.g. the next bunch of data from the tape, to write a packet to
tape or to put in a certain cartridge and the server performs
the actions (hopefully) as he can. The client knows, how to pack
and unpack the files and directories and on which cartridges and
files they are written and can be found. For this purpose he
maintains file lists containing the names of all stored filesystem
entries in a local directory. These filelists are not saved to the
backup if not explicitely configured. Compression is done on the
client side to reduce network traffic and to get more safety. If
the packed stream would be compressed, a single wrong bit in the
stream would make the whole rest of the backup useless. Thus the
files are compressed individually on the client side.


Server
''''''

Storage Devices

Any kind of streamer can be used with this backup system. The
only restriction is, that it must have a block-device entry
in the /dev-directory. The blocksize of the device must be
set correctly in the serverside configuration.
Besides of streamers also filesystems can be used to save the
data to. The configuration to do this is a little special and
can be taken from the HOWTO.FAQ.DO-DONT 3.
With streamers several tape cartridges can and should be used.
There are two basic cases: Either you have a robot or have not.
If you have no robot, you must change the cartridges manually.
The server supplies you with e-mail, whenever another cartridge
should be put into the drive. Having done this, a special
command (cartready) must be run on the server to let him know,
that he can continue now.
If you have a robot, but no special software (commands) to tell
it, which cartridge to insert into the drive, you must set the
robot to sequential mode. That is, the robot puts the next tape
into the drive once the actual one is ejected. Such a mode is
usually supported and then the server ejects the cartridges out
of the drive until the right tape is found.
If you have commands to set the cartridge directly, you can
configure them in the configuration file.

Cartridges

The cartridges are identified by numbers beginning with 1. Once
they are in use, they automatically get a magnetic label, that
they keep until overwritten with a special command (label_tape).
It is a good idea also to use adhesive labels and to write the
numbers onto the tapes, so they can't be mixed up. The server
recognizes, if there is a tape in the drive, that has the wrong
label. What then happens, is depending on the cases mentioned
above. If you have no robot, you get another email, that points
you to the problem. If you have a robot in sequential mode, the
server tries all cartridges, he has, to find the right one. If
it finds a tape with the right label, this is used. If it does
not, it stops and writes errors to the serverside error logfile.
If you have a robot and commands to set the cartridge and the
robot inserts the wrong one, this is considered a severe problem
and the server stops, again writing errors to it's error log.
Once all tapes of a set (for cartridge sets see the next section)
have been used up, the server starts again with the first of the
set overwriting it silently. This could be considered problematic,
but up to now i never got a negative response to this behaviour.
If someone thinks, it is a must to inform an administrator before
overwriting older backups, i'll build this feature in.
More cartridges than the robot can juggle can be used. For this
case see the HOWTO.FAQ.DO-DONT Q6.

Cartridge Sets

You can devide your cartridges into sets for several purposes.
A set comprises of a range of cartridges, e.g. the cartridges
number 1-3. There must be no unused cartridges inbetween sets
and in fact the configuration characteristics do systematically
not allow this. On the other hand you can omit cartridges at
the upper end of the numbers e.g. use 8 cartrdiges of 10. The
upper two will be unused. Nonetheless if you have a robot, you
must tell the backup system, that there are 10 cartridges. The
configuration parameter is named LastCartridges. This implies,
that you have only to give the last cartridge of each set,
separated by whitespace e.g. 3 8. If you do not configure sets
of cartridges, all available will be used as the one and only
set. Which set to use, can be configured on the client side
configuration file or overriding this setting by using
commandline options.
Trials of answers to the question, why to use sets, can be
found in the HOWTO.FAQ.DO-DONT Q4.

Storage units

This is a term i use for the combination of a backup server's
hostname, the port number to connect and a valid cartridge set
number on this server. Several storage units can be configured
on each client. When starting a backup, the client side first
checks the servers for availability one after the other and
starts the backup to the first available one found. Running a
restore the user or administrator does not have to remember,
to which storage unit the backup went. This is all handled
transparently.
For each of the three clientside configuration parameters
(server hostname, port number and cartridge set) several
entries are allowed now. They are associated by position.
For further information, see the HOWTO.FAQ.DO-DONT HOWTO Q9.

Read and Write

The read position can be requested by clients without any
restriction. With the write position it is different. The server
always continues to write at the position, where it stopped the
most recent write operation. This handling is maintained
individually for each cartridge set. For each set the writing
position is stored. Writing position means not only the cartridge
used before, but also the number of the file on tape. Thus not
another tape is started for each and every backup.
The data stream is written to the tape in pieces (tape files) of
a configurable maximum size. This is done to find the position
faster, where certain data has been stored. The client knows for
every file, on which cartridge the file is and in which tape file
it can be found. Thus a restore of certain requested files is much
faster than starting one only file for each backup in the worst
case using up a whole cartridge.

Locking

The server process generates a lockfile when starting to prevent
several server processes to access the same tape drive. Nontheless
if several processes start, the one started later waits a confi-
gurable time for the device to become available sending email
after configurable intervals.


Client
''''''

Full and Incremental backup

A full backup means, that everything requested is stored on the
server. An incremental backup stores only these files and direc-
tories, that have changed or are newer than the start of the
previous full or incremental backup. For determining, what has
changed, the mtime in the inode is used. Why this is done, can
be read from the HOWTO.FAQ.DO-DONT, Q2. Problems may arise, if
mechanisms are used that create files and directories while
resetting the mtime to an earlier date and time (e.g. using
tar -x or cpio -i). Then these files are not saved during the
next incremental backup.
The full backup can be divided into several parts. This is useful
for example, if the duration for backing up everything is too
long for the intended timepoint. If the full backup should run
during the weekend, but the about 60 hours from friday evening
to monday morning are not sufficient, this is a case for several
parts. These can be configured listing the files and directories
for each part in different clientside configuration parameters.
Before the parameter NumBackupParts must be set appropriately.

Packing and Unpacking

The client packs the files and directories and sends them to the
server or receives a data stream from the server and unpacks it.
If compression is requested, a configurable compression program
can be used to process each packed file individually. The name of
the appropriate uncompress-program is packed into the stream, so
it needs not be supplied during restore. Nontheless it must be
insured, that compress and uncompress-programs are within the
search path of the client processes or supplied with the full path.

Filelists

For each stored filesystem entry the client writes a line to a
file containing the cartridge number, where it can be found for
restore, also the tape file number and (since 2.9) the ID of the
user owning the filesystem entry. This listfile is lying locally
in a filesystem on the client. It serves for determining, where
the data for restore can be found on the server and to determine,
whether a user is allowed to restore the corresponding file, if
this possibility is configured (see HOWTO.FAQ.DO-DONT Q7).
Each time a full backup is done, a new filelist is started.
Incremental backups result in appending lines to this file.
The filelist files are identified by the number of the full
backup, that is increased continuously each full backup.
A configurable number of such filelist files is kept for later
use. The restore utility needs these lists, when certain files
are requested for restore. If these filelists get lost, you are
in trouble. You should use the mechanism described in the next
section to avoid this.

Minimum restore information

The minimum information to restore everything (the stored data
and the filelists) is, to which cartridges and - on those - in
which tape files the data can be found. This information is
passed each start of a full or incremental backup to a configurable
program, that can read this information (3 lines of a special
format) as the standard input. It should write this to a place
where it is safe from crashes of the local machine. One possi-
bility is a NFS-mounted directory, another one is to send this
information via email. In any case it must be made sure, that
the information is not mixed with those from different clients.
When you mix them up, restore of the wrong data may occur.
This minimum information can be passed to the restore utility
for restoring either everything or only the filelist files.

Clientside program structure

There is a two-layer program hierarchy on the client side. The
program "client" in the client's binary directory is performing
the packing, unpacking and the communication with the server
side. It does not read the configuration file and does not
maintain the filelists described above. This is done by programs
with higher functionality, that in turn call the "client"-program
setting it into different modes. The entry "backup" in the same
directory is identical with "client".


How to use it
-------------

Here a typical case is described. Individual needs can be
satisfied reading the rest of the documentation.

We assume, you have a DAT-cartridge handler with 12 tapes
connected to some machine without the possibility to set a
cartridge directly via commandline interface. So the first
thing to do is set the robot to sequential mode. It is no bad
idea to make this machine "fetch" the backup from the clients
using a small script, that is started via crond. Thus the
first step is to setup this backup server. Unpack the
distribution, run the Install script and select installation
option number 1 for installing and configuring the server side.
The last question to be answered is, whether to run the
serverside configuration program. Answer with yes, cause we
should do this now. The most entries can be accepted as they
are by default. Which device should be used, you must know
yourself, you have for sure already done some archiving with
tar to the connected streamer. The blocksize can be looked
up usually in the man-pages of rmt or tar. Set the parameter
"CartridgeHandler" to 1 and the number of cartridges to 12.
Cause you must use the sequential mode of the robot, clear
the "SetCartridgeCommand", if not already done entering the
parameter number and a single space for the value. Set the
"ChangeCartridgeCommand" to a command, that ejects a tape
from the drive (usually mt -f %d rewoffl - the %d stands for
the configured devicename, just for convenience, that it needs
not to be typed in several times). The UserToInform-parameter
should be set to some administrator or group of administrators,
that should receive important informations. The rest of the para-
meters is quite self-explanatory. Use the online-help, the
CONFIG and the HOWTO.FAQ.DO-DONT, if you don't know, what they
mean. Especially the CONFIG should give valuable hints. Finish
the server configuration entering "ok".
Now make the robot to have cartridge number 1 present inside
the streamer. If you do not want to do this or can't convince
the device to do, what you desire, run the following command:
/the/path/to/server/bin/cartis <cartno>
where <cartno> should be the integer number specifying the
cartridge actually present in the drive.

Now install the client side on each client, that should do
backups to the server. Again, unpack the distribution and
run the Install script, choosing the installation option
number 3 for client with remote start possibility. Again, all
defaults should be ok, so the last question, whether to run
the clientside configuration program can be affirmed.
Reasonable defaults are present for each parameter. What you
have to configure in any case are the files and directories,
that are to be stored. Furthermore, the startup information
program should be configured to a reasonable command (see above
under "Minimum restore information" for details). If you
are upgrading to a newer version and some of the parameters
are not set, see the client.config (or server.config, respec-
tively) of the actual distribution for defaults, that i consider
sufficient for normal cases. This also might give you better
ideas, what the parameters are good for.

Finally write small scripts starting the backup one after the
other on the clients (the server may off course also be a
client). An example script is provided in the HOWTO.FAQ.DO-DONT
under 2, just replace the list of hosts in the shell-variable
BACKUPCLIENTS. Now add entries to root's crontab-file. Entering
crontab -l > my_crontab
as root writes the actual crontab to the file my_crontab. To
run e.g. a full backup each friday evening and incremental
backups each evening from monday to thursday (all at 8 pm), add
these two lines:

0 20 * * 5   /usr/backup/client/bin/full_backup_cycle
0 20 * * 1-4 /usr/backup/client/bin/incr_backup_cycle

assumed the named programs are the scripts you wrote according
to the above description. Then run the command
crontab my_crontab
to install the modified crontab file.

That's it. Maybe, you want to start the first full backup manually
to see, whether everything is correctly installed and working.
In this case just start the script yourself and watch, what
happens. Especially watch the client- and serverside logfiles in
/path/to/client/var/backup.log and /path/to/server/var/backup.log.
If something goes wrong, also the clientside command
/path/to/client/bin/print_errors
tells more details.



----

If this introduction together with the rest of the documentation
is not sufficient, please let me know, but PLEASE give me
instructive hints, what you are missing or what should be
clarified.

Albert Fluegel

Oct 13 1997