1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918
|
AF's Backup HOWTO
=================
Index
-----
1. How to optimize the performance to obtain a short backup time ?
2. How to start the backup on several hosts from a central machine ?
3. How to store the backup in a filesystem instead of a tape ?
4. How to use several streamer devices on one machine ?
5. How to recover from a server crash during backup ?
6. How to port to other operating systems ?
7. How to provide recovery from hard crashes (disk crash, ...) ?
8. How to make differential backups ?
9. How to use several servers for one client ?
D. Do-s and Dont-s
F. The FAQ
--------------------------------------------------------------------------
1. How to optimize the performance to obtain a short backup time ?
Basically since version 2.7 the client side tries to optimally adapt
to the actually maximum achievable throughput, so the administrator
doesn't have to do much here.
The crucial point is the location of the bottleneck for the throughput
of the backup data stream. This can be one of:
- The streamer device
- The network connection between backup client and server
- The CPU on the backup client (in case of compression selected)
What usually is not a problem:
- The CPU load of the server
The main influence the administrator has on a good backup performance
is the compression rate on the client side. In most cases the bottleneck
for the data stream will be the network. If it is based on standard
ethernet, the maximum throughput without any other network load will be
around 1 MB/sec. With 100 MBit ethernet or a similar technology about
10 MB/sec might be achieved, so the streamer device is probably the
slowest part (with maybe 5 MB/sec for a Exabyte tape). To use this
capacity it is not clever to plug up the client side CPU with heavy
data compression load. This might be unefficient and thus lead to a
lousy backup performance. The influence of the compression rate on the
backup performance can be made clear with the following table. The
times in seconds have been measured with the (unrepresentative)
configuration given below the table. The raw backup duration gives the
pure data transmission time without tape reeling or cartridge loading
or unloading.
compression program | raw backup duration
-----------------------+----------------------
gzip -1 | 293 seconds |
gzip -5 | 334 seconds |
compress | 440 seconds | increasing duration
<no compression> | 560 seconds |
gzip -9 | 790 seconds V
Configuration:
Server/Client machine:
586, 133/120MHz (server/client), 32/16 MB (server/client)
Network:
Standard ethernet (10 MB, 10BASE2 (BNC/Coax), no further load)
Streamer:
HP-<something>, 190 kByte/sec
Obviously the bottleneck in this configuration is the streamer.
Anyway it shows the big advantage compression can have on the
overall performance. The best performance is here achieved with
the lowest compression rate and thus the fastest compression
program execution. I would expect, that the performance optimum
shifts towards a somewhat better compression with a faster client
CPU (e.g. the latest Alpha-rocket).
So to find an individual performance optimum i suggest to run some
backups with a typical directory containing files and subdirectories
of various sizes. Run these backups manually on the client-side machine
with different compression ratios using the "client"-command as
follows:
/the/path/to/bin/afclient -cvnR -h your_backuphost -z "gzip -1" gunzip \
/your/example/directory
Replace "gzip -1" and "gunzip" appropriately for the several runs.
--------------------------------------------------------------------------
2. How to start the backup on several hosts from a central machine ?
For this purpose serves the remote startup utility. To implement this
as fast as possible, a part of the serverside installation must be
made on the client side, where it is requested to start the backup from
a remote site. Choose the appropriate option when running the Install-
script.
To start a backup on another machine, use the -X option of the
client-program. A typical invocation is
/the/path/to/client/bin/afclient -h <hostname> -X incr_backup
This starts an incremental backup on the supplied host. Each
program on the remote host lying in the directory configured
as Program-Directory in the configuration file of the serverside
installation part of the remote host (default: $BASEDIR/server/rexec)
can be started, but no other. The entries may be symlinks, but
they must have the same filename like the programs, they point to.
The machine, where this script is started may be any machine in
the network having the client side of the backup system installed.
--------------------------------------------------------------------------
3. How to store the backup in a filesystem instead of a tape ?
* Option 1 (using symbolic links)
Assumed the directory, where you'd like to store the backup, is
/var/backup/server/vol, change to that directory and create a
symbolic link like this:
ln -s data.0 data
and create the file data.0 with a
touch data.0
Then edit your serverside configuration file and make the following
entries (assuming /usr/backup/server/bin is the directory, where the
programs of the server side reside):
Backup-Device: /var/backup/server/vol/data
Tape-Blocksize: 1024
Cartridge-Handler: 0
Max Bytes Per File: 10485760
Cart-Insert-Gracetime: 0
SetFile-Command: /bin/rm -f %d;touch %d.%m; ln -s %d.%m %d; exit 0
SkipFiles-Command: /usr/backup/server/bin/__inc_link -s %d %n
Set-Cart-Command:
Change-Cart-Command: exit 0
Erase-Tape-Command: /bin/rm -f %d.[0-9]* %d ; touch %d.0 ; ln -s %d.0 %d ; exit 0
If the directory /var/backup/server/vol/data is on a removable media,
you can supply the number of media you would like to use and an
eject-command as follows:
Number Of Cartridges: 10
# or whatever
Change-Cart-Command: your_eject_command
If a suitable eject-command does not exist, try to write one yourself.
See below for hints.
Furthermore you can add the appropriate umount command before the eject-
command like this:
Change-Cart-Command: umount /var/backup/server/vol/data; your_eject_command
To get this working the backup serverside must run as root. Install the
backup stuff supplying the root-user when prompted for the backup user.
Or edit /etc/inetd.conf and replace backup (or whatever user you configured)
(5th column) with root, sending a kill -1 to the inetd afterwards.
Actually you must mount the media manually after having it inserted into
the drive. Afterwards run the command /path/to/server/bin/cartready to
indicate, that the drive is ready to proceed. This is the same procedure
like having a tape drive.
Each media you will use must be prepared creating the file "data.0" and
setting the symbolic link "data" pointing to data.0 like described above.
* Option 2 (supply a directory name as device)
Using one of the serverside configuration programs or editing the
configuration file, supply a directory name as the backup device.
The directory must be writable for the user, under whose ID the
server process is started (whatever you configured during
installation, see /etc/inetd.conf). The backup system then writes
files with automatically generated names into this directory.
The rest of the configuration should e. g. be set as follows:
Backup-Device: /var/backup/server/vol
Tape-Blocksize: 1024
Cartridge-Handler: 0
Max Bytes Per File: 10485760
Cart-Insert-Gracetime: 0
SetFile-Command: exit 0
SkipFiles-Command: exit 0
Set-Cart-Command:
Change-Cart-Command: exit 0
Erase-Tape-Command: /bin/rm -f %d/* ; exit 0
A SetFile-Command is mandatory, so this exit 0 is a dummy.
For the further options (using mount or eject commands) refer
to the explanations under * Option 1.
(
How to write an eject command for my removable media device ?
If the information in the man-pages is not sufficient or you don't
know, where to search, try the following:
Do a grep ignoring case for the words "eject", "offline" and
"unload" over all system header-files like this:
egrep -i '(eject|offl|unload)' /usr/include/sys/*.h
On Linux also try /usr/include/linux/*.h and /usr/include/asm/*.h.
You should find macros defined in headers with names giving hints
to several kinds of devices. Look into the header, whether the
macros could be used with the ioctl system call. The comments
should tell details. Then you can eject the media with the
following code fragment:
#include <sys/ioctl.h>
#include <your_device_related_header>
{
int res, fd;
char *devicefile = "/dev/whatever";
fd = open(devicefile, O_RDONLY);
if(fd < 0){
/* catch error */
...
}
res = ioctl(fd, YOUR_EJECT_MACRO);
if(res < 0){
/* catch error */ ...
}
close(fd);
}
You might want to extend the utility obtainable via ftp from:
ftp://ftp.zn.ruhr-uni-bochum.de/pub/Linux/eject.c and related
files. Please send me any success news. Thanks !
--------------------------------------------------------------------------
4. How to use several streamer devices on one machine ?
Run an installation of the server side for each streamer device,
install everything into a separate directory and give a different
port number to each installed server. This can be done giving each
server an own service name. For the default installation, the
service is named "afbackup" and has port number 2988. Thus, entries
are provided in files in /etc:
/etc/services:
afbackup 2988/tcp
/etc/inetd:
afbackup stream tcp nowait ...
For a second server, you may add appropriate lines, e.g.:
/etc/services:
afbackup2 2989/tcp
/etc/inetd.conf:
afbackup2 stream tcp nowait ...
Note, that the paths to the configuration files later in the inetd.conf-
lines must be adapted to each installation, respectively. To get the
services active, send a Hangup-Signal to the inetd.
(ps ..., kill -HUP <PID>)
The relations between backup clients and streamer devices on the
server must be unique. Thus the /etc/services on the clients must
contain the appropriate port number for the backup entry, e.g.:
afbackup 2990/tcp
Note, that on the clients the service name must always be "afbackup"
and not "afbackup2" or whatever.
As an alternative, you can supply the individual port number in
the clientside configuration. If you do so, no changes must be
made in any clientside system file, here /etc/services.
Do not use NIS (YP) for maintaining the afbackup-services-entry, i.e.
do not add the entry with "afbackup" above to your NIS-master-services-file.
It is anyway better not to use the files /etc/passwd ... as sources
for your NIS-master-server, but to use a copy of them in a separate
directory (as usually configured on Solaris and other Unixes).
--------------------------------------------------------------------------
5. How to recover from a server crash during backup ?
With some devices there will be the problem, that the end-of-tape mark
is not written on power-down during writing to the tape. Even worse,
when power is up again, the position, where the head is actually placed,
gets corrupt, even if no write access has been applied at power-down.
Some streamers are furthermore unable to start to write at a tape
position, where still records follow, e.g. if there are 5 files on tape,
it is e.g. impossible to go to file 2 and start to write there. An
I/O-error will be reported.
The only way to solve this is to tell the backup system to start to
write at the beginning of the next cartridge. If the next cartridge
has e.g. the label-number 5, log on to the backup server, become root
and type:
/your/path/to/server/bin/cartis -i 5 1
--------------------------------------------------------------------------
6. How to port to other operating systems ?
* Unix-like systems *
This is not that difficult. The GNU-make is mandatory, but this is
usually no problem. A good way to start is to grep for AIX or sun
over all .c- and .h-files, edit them as needed and run the make.
You might want to run the prosname.sh to find out a specifier for
your operating system. This specifier will be defined as a macro
during compilation (exactly: prepocessing).
An important point is the x_types.h-file. Here the types should be
adapted as described in the comments in this file, lines 28-43.
Insert ifdef-s as needed like for the OSF 1 operating system on alpha
(macros __osf__ and __alpha). Note, that depending on the macro
USE_DEFINE_FOR_X_TYPES the types will be #define-d instead of
typedef-d. This gives you more flexibility, if one of those
possibilities is making problems.
The next point is the behaviour of the C-library concerning the
errno-variable in case the tape comes to it's physical end. In most
cases errno is set to ENOSPC, but not always (e.g. AIX is special).
This can be adapted modifying the definition of the macro
END_OF_TAPE (in budefs.h). This macro is only used in if-s as shown:
if(END_OF_TAPE) ...
Consult your man-pages for the behaviour of the system calls on
your machine. It might be found under rmt, write or ioctl.
The next is the default name of the tape device. Define the macro
DEFAULT_TAPE_DEVICE (in budefs.h) appropriately for your OS.
A little pathological is the statfs(2) system call. It has a different
number of arguments depending on the system. Consult your man-pages,
how it should be used. statfs is only used in write.c
There may be further patches to be done, but if your system is close
to POSIX this should be easy. The output of the compiler and/or the
linker should give the necessary hints.
Please report porting successes to af@muc.de. Thanks.
Good luck !
* Win-whatever *
This is my point of view:
Porting to Microsoft's Features-and-bugs-accumulations is systematically
made complicated by the Gates-empire. They spend a lot of time on taking
care, that it is as difficult as possible to port to/from Win-whatever.
This is one of their monopolization strategies. Developpers starting to
write programs shall have to make the basic decision: "Am i gonna hack
for Micro$oft's "operating systems", or for the others ?" Watching the
so-called market this decision is quite easy: Of couse they will program
for the "market leader". And as few as possible of what they produce
should be usable on other ("dated") platforms. Companies like Cygnus
are providing cool tools (e. g. a port of the GNU-compiler) to make
things easier but due to the fact, that M$ are not providing so many
internals to the public, in my opinion porting is nonetheless an
annoying job. Thank Bill Gates for his genious strageties.
In short, at the moment i'm not gonna provide information how to port
to Micro$oft-platforms. If a port will be available at all, i'll do it
myself and as normal in the M$-world, i wanna get a lot of money for it.
This is my actual point of view. Maybe i'll be forced to change due to
the increasing censorship upcoming in the computer business, but i pray,
this won't ever be the case.
--------------------------------------------------------------------------
7. How to provide recovery from hard crashes (disk crash, ...) ?
A key to this is the clientside StartupInfoProgram parameter. This
command should read the standard input and write it to some place
outside of the local machine - to be more precise - not to a disk
undergoing backups or containing the clientside backup log files.
The information written to the standard input of this program is
the minimum information required to restore everything after a
complete loss of the saved filesystems and the client side of the
backup system. Recovery can be achieved using the restore-utility
with the -e flag (See: PROGRAMS) and supplying the minimum recovery
information to the standard input of restore. Several options exist:
- Write this information to a mail-program (assumed the mail folders
are outside of the filesystems undergoing backup) and sending this
information to a backup-user. Later the mailfile can be piped into
the restore-utility (mail-related protocol lines and other unneeded
stuff will be ignored). For each machine, that is a backup client,
an individual mail user should be configured, cause the minimum
restore information does NOT contain the hostname (to be able to
restore to a different machine, what might make perfect sense in
some situations)
- Write the information into a file (of course: always append),
that resides on an NFS-mounted filesystem, eventually for security
reasons exported especially to this machine only. To be even more
secure, the exported directory might be owned by a non-root-user,
who is the only one, who may write to this directory. This way it
can be avoided to export a directory with root-access. Then the
StartupInfoProgram should be something like:
su myuser -c "touch /path/to/mininfo; cat >> /path/to/mininfo"
The mininfo-file should have a name, that allows to deduce the
name of the backup-client, that wrote it. E.g. simply use the
hostname for this file.
- Write the information to a file on floppy disk. Then the floppy
disk must always be in the drive, whenever a backup runs. The
floppy could be mounted using the amd automounter as explained in
ftp://ftp.zn.ruhr-uni-bochum.de/pub/linux/README.amd.floppy.cdrom
or using the mtools usually installed for convenience. In the
former case the command should contain a final sync. In the
latter case the file must be first copied from floppy, then
appended the information, finally copied back to floppy e.g. like
this:
mcopy -n a:mininfo /tmp/mininfo.$$; touch /tmp/mininfo.$$; \
cat >> /tmp/mininfo.$$; mcopy -o /tmp/mininfo.$$ a:mininfo; \
/bin/rm -f /tmp/mininfo.$$; exit 0
Note, that the whole command must be entered in one line using
the (x)afclientconfig command. In the configuration file lineend
escaping is allowed, but not recognized by (x)afclientconfig. An
alternative is to put everything into one script, that is started
as StartupInfoProgram (Don't forget to provide a good exit code
on successful completion)
My personal favourite is the second option, but individual preferences
or requirements might lead to different solutions. There are more
options here. If someone thinks, i have forgotten an important one,
feel free to email me about it.
It might be a good idea to compile afbackup linked statically with
all required libraries (building afbackup e.g. using the command
make EXTRA_LD_FLAGS=-static when using gcc), install it, run the
configuration program(s), if not yet done, tar everything and
put it to a floppy disk (if enough space is available).
To recover from a heavy system crash perform the following steps:
- Replace bad disk(s) as required
- Boot from floppy or cdrom (the booted kernel must be network-able)
- Add the backup server to /etc/hosts and the following line to
/etc/services: afbackup 2988/tcp
- Mount your new disk filesystem(s) e.g. in /tmp/a and in a way, that
this directory reflects your original directory hierarchy below
/ (like usually most system setup tools do)
- Untar your packed and statically linked afbackup-distribution, but
NOT to the place, where it originally lived (e.g. /tmp/a/usr/backup),
cause it will be overwritten, if you also saved the clientside
afbackup-installation, what i strongly recommend.
- Run the restore-command with -e providing the minimum restore
information saved outside of the machine to stdin:
/path/to/staticlinked/restore -R /tmp/a -e < /path/to/mininfo-file
Bootsector stuff is NOT restored in this procedure. For Linux
you will have to reinstall lilo, but this is usually no problem.
--------------------------------------------------------------------------
8. How to make differential backups ?
A differential backup means for me: Save all filesystem entries modified
since the previous full backup, not only those modified since the last
incremental backup.
This task can be accomplished using the -a option of the incr_backup
command. It tells incr_backup to keep the timestamp. If -a is omitted
one time, another differential backup is no longer possible since the
timestamp is modified without -a. So if differential backups are required,
you have to do without incremental backups.
--------------------------------------------------------------------------
9. How to use several servers for one client ?
Several storage units can be configured for one client. A storage unit
is a combination of a hostname, a port number and a cartridge set number.
Several servers can be configured on one machine, each operating an own
streamer device or directory for storing the data.
The storage units are configured by the first three parameters of the
client side. These are hostnames, port numbers and cartridge set numbers,
respectively. Several entries can be made for each of these parameters.
The port numbers and/or cartridge set numbers can be omitted or fewer
than hostnames can be supplied, then the defaults will apply. If more
port or cartridge set numbers than hostnames are given, the superfluous
ones are ignored. The lists of hostnames and numbers can be seperated
by whitespace and/or commas.
When a full or incremental backup starts on a client, it tests the
servers, one after the other, whether they are ready to service them.
If none is ready, it waits for a minute and tries again.
With each stored filesystem entry, not only the cartridge number and
file number on tape is stored, but now also the name of the host,
where the entry is stored to, and the appropriate port number. Thus
they can be restored without the necessarity, that the user or adminis-
trator knows, where they are now. This all happens transparently and
without additional configuration efforts. For older backups, the first
entry of each list (hostname and port) is used. Therefore, in case of
an upgrade, the first entries MUST be those, that applied for this
before the upgrade.
If there are several clients, the same order of server entries should
not be configured for all of them. This would probably cause most of
the backups to go to the first server, while the other(s) are not
explioted. The entries should be made in a way, that a good balancing
of storage load is achieved. Other considerations are:
- Can the backup be made to a server in the same subnet, where the
client is
- Has this software been upgraded ? Then the first entry should be
the same server as configured before (see above)
- The data volume on the clients to be saved (should be balanced)
- The tape capacity of the servers
- other considerations ...
--------------------------------------------------------------------------
D. Do-s and Dont-s
ALWAYS
------
- configure a Startup-Info-Command on the clientside
to save the crucial information for the emergency
restore after a hard crash losing the filename logfiles
i.e. the filelists
- write the numbers of the cartridges onto their cases
or use adhesive labels
- be happy and forgive me the bugs
- try to keep things simple. Unnecessarily making things
complicated will strike back mercilessly sooner or later,
according to Murphy sooner, if not ASAP
... to be continued
NEVER
-----
- kill any afbackup-related programs with -9 (== -KILL)
It may take a while until the program has cleaned up
but it does clean up and it will terminate. If it
takes longer than 10 minutes, then you might THINK of
kill -9. But first watch the processor time consumed
by the process !
- use fewer cartridges than with a capacity of at least
two times the size required for one full backup including
subsequent incremental backups. Otherwise stored files
could be lost due to an unsuccessful backup overwriting
previously stored data.
... to be continued
--------------------------------------------------------------------------
F. AF's Backup FAQ
===============
Index
-----
Q1: How do I tell how much space is left on the tape?
Q2: Why is the mtime used for deciding what files to save during incremental
backup and not the ctime or both ?
Q3: Do my current configurations get overwritten during an upgrade ?
Q4: Why should I and how do I use sets of cartridges ?
Q5: How many cartridges should I use ?
Q6: I have a robot with n cartridges. Can i use more than n tapes ?
Q7: Can ordinary users restore their own files and directories ?
Q8: Why does afbackup not have a GUI ?
Q9: What does the warning mean: "Filelist without user-ID information ..." ?
Q10: The whole backup systems hangs in the middle of a backup, what's up ?
Q11: Tape reels back and forth, mail sent "tape not ready ...", what's up ?
Q12: The server seems to have a wrong tape file count, what's wrong ?
Q13: When using crond, the client seems not to start correctly ... ?
Q14: What does AF mean ?
Q15: Though client backup works, remote start does not. Why ?
Q16: My server does not work, tape operates, but nothing seems to be written ?
Q17: I have a ADIC 1200G autoloading DAT and no docs. Can i use it with HPUX ?
Q18: What is a storage unit and how and why should i use it ?
Questions and Answers
=====================
Q1: How do I tell how much space is left on the tape?
A1: This is hard to tell due to the problem to determine exactly, how many
bytes can be written to a certain physical tape. To get an estimate, how
many bytes are already on tape, ask the server for the actual writing
position from any client site:
/path/to/client/bin/afclient -h <your_backup_server> -Q
This will tell you also the file to that will be written next time.
This number times your configured Max Bytes Per File (in the serverside
configuration file) gives you an upper limit of the written bytes. It
is an upper limit, cause not all files on tape have the full length.
To get one more *estimate*, how many bytes can be written to a tape,
you might search the logfiles for the highest occuring file number.
Go to your (clientside) /path/to/client/var and look at your
backup_log.XXX or backup_log.XXX.z -file. Type more (or zmore,
respectively) and scan the entries. They are of the form
CC.FF: /files/and/dirs
where CC is the cartridge, where this file is stored and FF is the
file number on tape. Look for the highest FF (Assumed of course, that
you have already several tapes written and at least one full tape).
This number times your configured Max Bytes Per File results in the
*estimate* for the number of bytes, that fit on tape.
Q2: Why is the mtime used for deciding what files to save during incremental
backup and not the ctime ?
A2: First: The ctime changes any time a chmod, a chown, or other operations
modifying the inode are performed. A change like this is not worth
selecting this file for backup, cause the file itself did not change.
Second: After backup of a file, afbackup restores the atime, because
i found the atime a quite worth information. A restore of the atime
changes the ctime, no way around this. If the ctime was evaluated
for choosing the files for incremental backup, a file stored once
would be saved again all the following backups, cause at any
backup time the ctime changes. Incremental backup would be senseless,
cause all files would be saved all the time a backup runs.
Q3: Do my current configurations get overwritten during an upgrade ?
A3: No. Nothing gets overwritten or lost. Newly introduced parameters have
the old non-configurable behaviour as default. The defaults are applied,
if the appropriate parameters are not given explicitely in the
configuration files.
Q4: Why should I and how do I use sets of cartridges ?
A4: The question, why, is not that easy to answer. Maybe you have groups
of hosts you would like to save to distinguished cartridges, maybe
you would like to make the full backups to other cartridges than the
incremental backup. Maybe you have the requirement, that you want to
use an infinite number of cartridges for the full backup and reuse
the ones for the incremental backup each time another full backup
has finished. Maybe you have more exotic requirements ...
The answer to the how is easy: Set the serverside parameter
LastCartridges supplying the last cartridge of each set. E.g. if
your cartridge sets should be 1-3, 4-8 and 9-10, set the parameter
to 3 8 10. A set may be a single cartridge, but i do not recommend
this, cause writing to the beginning of that cartridge destroys the
rest stored on it and making these data unaccessible. The last
number is usually the number of cartridges you have, but not
necessarily. Cartridges at the upper end of the numbers might be
omitted. If the last number is not equal to the number of cartridges,
this number is NOT automatically added. The many numbers are given
with this parameter, the many cartridge sets you have. The default,
if this parameter is not present, is one cartridge set with all
available cartridges.
Q5: How many cartridges should I use ?
A5: The cartridges should be have enough capacity for at least two times
a full backup including subsequent incremental backups. Otherwise
files could get lost due to an unsuccessful backup overwriting
previously stored data.
Q6: I have a robot with n cartridges. Can i use more than n tapes ?
A6: Yes, you can use any number of tapes, if your robot is in the
sequential mode. Simply fake a higher number to the backup system
in the serverside configuration file. The only point is, that you
have to change the cartridges in the robot manually in time. If
you have e.g. a robot with 10 cartridges and would like to use 20,
then you have to watch, when it is time to insert other cartridges
to the appropriate positions. E.g. when cartridge number 8 is in
the drive, take out cartridges 1-7 and insert number 10-16 into
the appropriate slot. Later, when they are in use, you can replace
8-10 by 17-19 and so on.
When you want to do a restore, the restore-program tells you,
where it wants to read from like this:
Going to restore from cartridge X, file Y ...
Insert manually the right cartridges into slots, the robot will
access next time. The system will automatically recognize by the
label on the tape, that it has found the right cartridge now. A
warning is written to the serverside log telling, that another
cartridge was found than expected, but this is just a warning and
we know, how this happened ...
Q7: Can ordinary users restore their own files and directories ?
A7: Yes, they can, but this feature must be enabled. The restore-
utility must be installed executable for all users and setuid
root. This can be achieved entering as administrator root:
rm -f $BASEDIR/client/bin/afrestore
cp $BASEDIR/client/bin/full_backup $BASEDIR/client/bin/afrestore
chmod 4755 $BASEDIR/client/bin/afrestore
Thus ordinary users can run this program. Built-in safety checks
provide, that they can only restore or list their own files and
directories. Changing the restore-directory using the option -R
allows them only to restore to directories owned by themselves.
Q8: Why does afbackup not have a GUI ?
A8: My ideal imagination of a backup system is, that i do not have
to care about it at all, once it is installed and configured
properly. It should do it's job in the background, only noticing
me, if something goes wrong. Thus i would not want any icons,
clocks or meters pop up on my workspace plugging me up with
unnecessary and unimportant stuff. The installation procedure
is simple enough and would not get better having a graphical
frontend. My opinion.
Q9: What does the warning mean: "Filelist without user-ID information ..." ?
A9: You are running restore with the list-option as non-root and
the filelists are of the pre-version-2.9-format. Thus they do
not contain the user-ID of the owners of the files. The program
does not know, whether it is permittable to show you the names
of the files. For security reasons it is hardcoded not to show
them. With the new format containing the user-IDs, you will see
the matching names of the files owned by you.
Q10: The whole backup systems hangs in the middle of a backup, what's up ?
A10: This phenomenon has been reported only on Linux, Kernel Version
2.0.30 and seems to be the result of a bug in this kernel. I
never experienced this problem on my 2.0.29 Kernel or on other
platforms.
Rumors told me, that the 2.0.30 kicks out about every 10000th
to 20000th Process, i.e. the process is started, appears in the
process list, but does not do anything and never terminates.
Thus parent processes wait forever, when this happens. Afbackup
compresses each saved file separately i.e. starts the
configured compression program for each file. When the problem
described above arises, this compression program hangs and
thus the whole chain up to the server process, that waits for
requests until eternity.
Solutions: (i'm aware, these are no real solutions)
- switch off compression for the saved files or
- change your Linux Kernel
Q11: Tape reels back and forth, mail sent "tape not ready ...", what's up ?
A11: The current state of investigations is, that this is probably
a problem of a dirty read-write head. This may sound weird,
but i'll try to explain.
I experienced this problem without any warning. One day when
starting a new backup i watched the tape reeling back and forth,
later sending an email to the person in charge telling, that
the device is not ready for use, and requesting to correct this
problem. Compiling everything with debugging turned on i caught
the server process during the initiliazation phase and found
the Set-File-Command (mt -f ... fsf ...) failing. Then i found
out, that there were fewer files on tape than the backup system
expected (1 too few) and thus the mt failed. I had no idea, how
this could happen. I corrected everything manually by decrementing
the writing position on tape. The next backup, that i started,
worked fine and another one immediately following, too. A verify
also succeeded. So i took out the tape and decided to ignore the
fault for the moment. Before i ran the next backup, i started a
verify to see, what has changed within the meantime, but now
again: tape reels back and forth endlessly. Looking onto the
tape manually using mt and dd once again: too few files on tape.
Seemed like some file (not the last one on tape !) was lost.
Strange. The only thing i could imagine causing all the trouble
was an error of the tape drive, e.g. a dirty read-write-head.
So i put in the next tape and started all over at the beginning
of the new tape. Everything worked perfectly from now on. The
phenomenon has been reported to me only on Linux with HP-DAT-
streamers. This could mean a correlation and/or a problem of a
Linux driver, but the reported number of 2 is in my opinion too
small for a conclusion like this. Furthermore i guess, the com-
bination Linux + HP-DAT-drive is very common, so the probability,
that problems might arise in such an environment is quite high
simply due to the number of installations of this kind.
Admittedly i had been too lazy for a notable while to use any
cleaning cartridge, so i guess this had been the problem.
Solution (to get out of the temporary inconvenience):
- Use a new cartridge and tell the backup system to use it with
the serverside command
/path/to/cartis -i <nexttape> 1
where <nexttape> is the number of the next cartridge
Conclusion:
- Feel urged to use cleaning cartridges regularly
Q12: The server seems to have a wrong tape file count, what's wrong ?
A12: Probably you experienced the following: The last filename in
the filename log is preceded by a different pair of cartridge
number / tape file number than the pair named in the report
email, written to the tape-position file on the server or
queried with the client-program option -Q.
This is perfectly possible. The last saved file can make the
tape file exceed the configured maximum length. Then one or
more further tape files are opened appropriately.
Q13: When using crond, the client seems not to start correctly ... ?
A13: You probably get the message "Connection to client lost ..."
in the clientside logfile. This is a weird problem i only
experienced on IRIX. The program gets a SIGPIPE and i have no
clue, why. You might start full_backup or incr_backup with the
option -d, which causes the program to detach from the terminal
and to wait for 30 Seconds before continuing. Maybe this solves
your problem.
Q14: What does AF mean ?
A14: Another F......
Q15: Though client backup works, remote start does not. Why ?
A15: The problem is in most cases, that during the remote start
the configured (un)compression programs, usually gzip and the
corresponding gunzip, are not found in the search path. Cause
the remotely started backup is some child of the inetd, it
gets of course the inetd's command search path. If this does
not contain the path to gzip, the start fails.
Q16: My server does not work, tape operates, but nothing seems to be written ?
A16: There seems to be a problem on some platforms. Try to start the
server with the -b option: Edit /etc/inetd.conf and add -b before
the last argument of afserver in the line starting with afbackup.
Then send a hangup signal to the inetd (ps ... |grep inetd -> PID,
kill -HUP <PID>). Then try again. If it works, be happy, but be
aware, that the performance is reduced in this mode. This problem
is worked on.
Q17: I have a ADIC 1200G autoloading DAT and no docs. Can I use it with HPUX ?
A17: Thanks to Gian-Piero Puccioni (gip@ino.it) you can. You will find
the mtx.c program he wrote helpful. Check out this file how to
build and use his mtx command. It enables you to load/unload/handle
certain cartridges.
Q18: What is a storage unit and how and why should i use it ?
A18: See in the HOWTO, Q9
|