File: keates.html

package info (click to toggle)
lg-issue91 1-2
links: PTS
area: main
in suites: sarge
size: 3,084 kB
ctags: 266
sloc: ansic: 1,343; perl: 104; sh: 98; makefile: 34
file content (620 lines) | stat: -rw-r--r-- 24,243 bytes
<!--startcut  ==============================================-->
<!-- *** BEGIN HTML header *** -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML><HEAD>
<title>Easy Backup and Restore LG #91</title>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#0000AF"
ALINK="#FF0000">
<!-- *** END HTML header *** -->

<!-- *** BEGIN navbar *** -->
<A HREF="collinge.html">&lt;&lt;&nbsp;Prev</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="index.html">TOC</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="../index.html">Front Page</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue91/keates.html">Talkback</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="../faq/index.html">FAQ</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="kruk.html">Next&nbsp;&gt;&gt;</A>
<!-- *** END navbar *** -->

<!--endcut ============================================================-->

<TABLE BORDER><TR><TD WIDTH="200">
<A HREF="http://www.linuxgazette.com/">
<IMG ALT="LINUX GAZETTE" SRC="../gx/2002/lglogo_200x41.png" 
	WIDTH="200" HEIGHT="41" border="0"></A> 
<BR CLEAR="all">
<SMALL>...<I>making Linux just a little more fun!</I></SMALL>
</TD><TD WIDTH="380">


<CENTER>
<BIG><BIG><STRONG><FONT COLOR="maroon">Easy Backup and Restore</FONT></STRONG></BIG></BIG>
<BR>
<STRONG>By <A HREF="../authors/keates.html">Alan Keates</A></STRONG>
</CENTER>

</TD></TR>
</TABLE>
<P>

<!-- END header -->



<h2>Introduction</h2>

<p>
Until recently the extent of my backup efforts were to take the
occasional CD copy of my home directory and keep copies of important files
somewhere else, usually on another disk partition, or a floppy disk.
</p>

<p>All this changed with the need to run some Windows legacy
applications. The only machine really suitable for this work was
my main workstation, a 1.2 GHz Athlon machine, multiboot with four 
distributions.
I decided to free up the 1st primary partition, which held Mandrake 9.0, and 
set up a Windows partition.</p>

<p>I freed up the 1st primary partition by transferring the contents of that
to the 7th partition, overwriting an expendable Vector Linux 3.0 Distribution.
To be totally safe I booted into Debian 3.0, mounted
both partitions to individual mount points in /mnt and as root used tar and a 
pipe to copy everything including
all links and permissions from the source partition to the target partition.
A few minutes later, after changing my grub boot menu, I was able to boot into 
Mandrake 9.0 Linux in the 7th partition and verify that everything worked as 
expected.
</p>

<p>
At this point one would normally just DOS format the now free first
partition and install Windows. However I began to feel a little uneasy.
Windows could just format the whole darn drive, or some other similar screwup 
could happen, in which
case I would be placed in the position of fdisk'ing the partitions and
reinstalling everything from scratch. The original disks would, of course have
all the applications except for those extra packages installed by me, but
any custom configurations would all be lost.
</p>

<p>The machine was now running Mandrake 9.0, Debian 3.0 and Slackware 8.1.
Of these, only losing my Slackware install would cause me grief.
This has been running like a top, boots to KDE 3.0 in less than 30 seconds, 
including
my sign on, and is absolutely rock solid stable. It also has the CUPS print 
system set up perfectly for all my printers on the LAN. So I must retain this 
setup at all costs. The solution of
course is to fully back up everything from the Slackware install.
</p>

<p>
At that point the desire to have a simple, easy and foolproof backup and
recovery method took hold.
</p>

<h2>What do we really need for a backup and recovery system?</h2>

<p>
If we are a home or SOHO Linux user I would suggest the following, it should:

<ul>
<li>  Require no equipment or software other than that we already have

<li>Be cost effective in backup media

<li>Be really easy to use regularly, or it will not be used at all

<li>Be easy to verify, or it may be useless when the time comes</li>

<li>Require only the media and a working machine, in the hardware sense

<li>Require only minimal knowledge of the recovery process when the crunch comes
</ul>
</p>

<p>A quick review of past Gazette articles and a search of the web will turn up 
hundreds of backup solutions.  Many are specifically aimed at the
backup function, many at the repair and system recovery part of the overall
effort to get back to some predefined state. Virtually none are customized to
your system, or your specific requirements, so why not roll your own solution?
That is what we do here.
</p>

<h3>What can we use</h3>
<p>
Most home or SOHO users do not have a tape drive system and are unlikely to 
purchase one for the sole purpose of backup, given that the cost of the tape
system and software most probably exceeds that of the computer itself.
This essentially leaves just backup to removable disk, backup to the same or
another hard drive, backup to CD and backup over a network to some other hard 
drive.
This last is essentially just a more complicated backup to local hard drive
except there is zero chance of it being lost when your system goes down.
So let us look at these options.
</p>

<p>
<b>Floppy</b> - Good for incremental backups on a daily basis and perhaps the 
best solution for saving work as it progresses, but useless for system wide 
restoration. The LS120 Disk and the Zip disk are not large enough or common
enough to be considered for the sort of simple but complete backup considered 
here.
</p>

<p>
<b>Hard Drive</b> - One can back up to a separate partition on the same drive, 
which of course is of little use if that drive fails, or one can backup to 
another hard drive in the same computer. This is good except there is a fair 
chance that a power supply failure or nearby lightning strike could fry both 
drives (or somebody could steal the computer), leaving nothing to restore.
</p>

<p>
<b>Network File System Transfer</b> - This is a good solution to backup and
restore of the files, for one interested enough to correctly install it, however
it does nothing for the process of getting the system up again to the point
where one can restore the files. Too complicated for most to institute.
</p>

<p>
<b>CD-ROM</b> - This is where it begins to look interesting. These days most 
Linux users have installed a CD burner and the availability of cheap CD-RW
disks means that the cost of maintaining something akin to the traditional 
rotating backup system is definitely on. This is the one for us.
</p>

<h2>CD-ROM Backup</h2>

<p>
The most essential requirement is to have a working and reliable CD burner.
Any current Linux distribution will have the tools required, and to minimize 
media costs, about $4 will
supply two good quality CD-RW disks. For daily backups these will last for about
five and a half years and used weekly a machine eternity!
</p>

<p>The scheme proposed here is to use the two CD-RW disks to take backups in
rotation; in my actual implementation I have color coded the spine of the disk
covers Red and Green respectively, to aid in the correct rotation.</p>

<p>
We also require the backup disk to self boot into a minimal working
Linux system. This is to ensure that we can re-establish the Master Boot 
Record (MBR) and the rest of the original partition information if required. 
This rules out
using a boot disk image as commonly supplied with the majority of distributions.
These supply just a boot method and a Linux kernel, and usually boot straight to
the partition they are customized to boot.
</p>

<p>
After a quick perusal of the small Linux on self boot CDs I settled on using
the classic and well tried <a href="www.toms.net/rb">TomsRtBt</a> disk in 2.88
MB image format. This 
is not an ISO image, but is suitable for being the boot image of an ISO we will
burn.
It is also to be found at various other sources on the web. I have used this in 
the floppy format
and it is very good and quite complete. Note that it also includes a Toms FAQ.
</p>

<p>
In order to restore our working Linux system to a given state we will require 
records of all of the current directory contents which are changing on a day to
day basis or have changed as customizations since initial install. This can be 
done laboriously by inspection and detailed lists, which will minimize what must
be restored, or accomplished very easily by backing up the entire contents of 
these
directories.
</p>

<p>
In my case I have decided to back up the entire contents of <b>/home /etc
/usr/local /opt /var /root /boot</b> of the Slackware 8.1 partition.
</p>

<p>
<ul>
  <li> /home of course holds all the files of each user</li>
  <li> /etc holds all of the configuration information</li>
  <li> /usr/local normally holds any extra programs added since install</li>
  <li> /opt is also commonly used by applications to install files</li>
  <li> /var holds all data of a variable nature</li>
  <li> /root belongs to the root user and has essential customizations</li>
  <li> /boot has all the files for booting the system and boot .conf files </li>
</ul>
</p>

<p>
In addition to the contents of each of the identified directories above there 
are some more very important pieces of information one wouldn't want to be
without if a sudden failure to boot occured. These are a binary copy of the MBR, 
a text list of the Partition Table, a copy of the fstab file in case you have forgotten which
partitions correspond to what filesystem, and optionally a copy of the current
XF86Config file and/or the text output of commands like lsdev and lspci for full 
system information.
</p>

<p>
Also how are we going to structure all of this information to ensure it gets 
onto the CD in such a way as to be completely self contained and usable for the
task at hand?
</p>

<p>
Here is what I did. Firstly create a directory to hold all of the information to
backup. As root: <i>mkdir /tmp/backup</i>. Note here that I am using /tmp as 
repository for the constant part of the backup CD. This is safe in Slackware, but might not
be in other distributions, choose a safe location and one not itself backup up by the
tar file.
</p>

<p>
Put into the backup directory a copy of TomsRtBt Img file :
<i>cp ./tomsrtbt288.img /tmp/backup/tomsrtbt288.img</i>, here the img file is in
my home directory.
</p>

<p>
Put into the backup directory a copy of the Master Boot Record:
<i>dd if=/dev/hda bs=512 count=1 &gt; /tmp/backup/MBR.bin</i>. The MBR holds
the first stage of the boot mechanism you employ, in my case stage1 of Grub, the
GRand Unified Boot Loader, or LILO, and also the partition information for the
Primary Partitions. The Extended Partition information is held elsewhere on the
disk and can if required be restored with the information you will store from 
the fdisk command detailed next.
</p>

<p>
Put into the backup directory a list of the Partition Information :
<i>fdisk -l &gt; /tmp/backup/Partition_Table</i>, this will be used to compare
with
a Toms listing of the partition table before any restoration takes place.
</p>

<p>
Put into the backup directory a copy of fstab which defines the file system 
mount points, any errors here and the files and devices will not be 
accessible.
<i>cp /etc/fstab /tmp/backup/fstab.bak</i>
</p>

<p>
Optionally copy any other information you wish available to you before you are 
able to boot into your newly restored Linux system. For easy accessability I
keep a copy of XF86Config on the disk to ensure that I can always set up X the 
way I like even if installing a new system upgrade, and a copy of 
menu.lst as I use Grub as my boot loader of choice.
<i>cp /etc/X11/XF86Config /tmp/backup/XF86Config.bak </i>...
<i>cp /boot/grub/menu.lst /tmp/backup/menu.lst.bak</i>
</p>

<p>
These files will be added to every copy of the backup disk that is burned, and 
need only be changed if one of them changes, when of course it should be copied 
over.
</p>

<h2>What do we need to do to create our self-booting backup disk</h2>

<ol>
  <li>Create a compressed TAR file of chosen directories, add to 
/tmp/backup</li>
  <li>Create bootable ISO of backup directory using mkisofs</li>
  <li>Check that size of ISO will fit on chosen CD-RW disk</li>
  <li>Burn to CD-RW using cdrecord</li>
  <li>At appropriate stages echo messages to standard out, md5sums, etc</li>
  <li>Clean up files no longer needed</li>
</ol>

<p>
The script to accomplish this is shown below, for a text copy see <a 
href="misc/keates/backup.sh.txt">backup.</a>
Be sure to rename the file without the .sh.txt part and to make it executable - 
<i>chmod 755
./backup </i>- and copy to somewhere in roots PATH, /usr/local/bin is a good 
place,
and do the same for the next script.
</p>

<pre>
#!/bin/bash
#  backup
#------------------------------------------------------------------------------
#  Script to enable easy backup of all important Linux files
#  and also creates a customized system repair disk.
#  Uses two CD-RW Disks labled "RED" and "GREEN to rotate backups
#------------------------------------------------------------------------------
# The backup directory already contains files for boot and recovery.
# One can add more - my Slackware 8.1 system backup is < 580MB.

Backup_Dirs="/home /etc /usr/local /opt /var /root /boot"
Backup_Dest_Dir=/tmp/backup
Backup_Date=`date +%b%d%Y`
Image_File=/tmp/backup.iso
declare -i Size

# Create tar file with todays Month Day Year prepended for easy identification
tar cvzf $Backup_Dest_Dir/$Backup_Date.tar.gz $Backup_Dirs &> /dev/null

# Start backup process to local CD-RW drive
echo "Backing up $Backup_Dest_Dir to CD-RW Drive - $Backup_Date"
echo "Creating ISO9660 file system image ($Image_File)."

mkisofs -b toms288.img -c boot.cat -r \
        -o $Image_File $Backup_Dest_Dir  &amp;&gt; /dev/nul

# Check size of directory to burn in MB
Size=`du -m $Image_File | cut -c 1-3`
if [ $Size -lt 650 ]
then
   echo "Size of ISO Image $Size MB, OK to Burn"
else
   echo "Size of ISO Backup Image too Large to burn"
   rm $Backup_Dest_Dir/$Backup_Date.tar.gz # Remove dated tar file
   rm $Image_File   # ISO is overwritten next backup but cleanup anyway
   exit 1
fi

# Burn the CD-RW
Speed=4                 # Use best speed for CD-RW disks on YOUR system
echo "Burning the disk."
                              # Set dev=x,x,x from cdrecord -scanbus
cdrecord -v speed=$Speed blank=fast dev=1,0,0 $Image_File &amp;&gt; /dev/null
Md5sum_Iso=`md5sum $Image_File`
echo "The md5sum of the created ISO is $Md5sum_Iso"

# Could TEST here using Md5sum_Iso to verify md5sums but problem is tricky.
echo "To verify use script md5scd, this will produce the burned CD's md5sum"
echo "run this as User with backup CD in drive to be used for recovery."
echo "This verifies not only the md5sum but that disk will read OK when needed."

# Remove image file and tar file
echo "Removing $Image_File"
rm $Image_File
echo "Removing : $Backup_Dest_Dir/$Backup_Date.tar.gz"
rm $Backup_Dest_Dir/$Backup_Date.tar.gz
echo "END BACKUP $Backup_Date"
echo "Be sure to place this backup in the RED CD case and previous CD in GREEN"
echo "------------------------------------------------------------------------"
exit 0
</pre>

<h2>Using the backup system</h2>

<p>
In use the process is simple, I usually backup every day, if not doing much on 
the system then every week. At the start of every backup I place the cdrom from 
the Green marked
case into the CD burner. In an xterm I su to root and issue the command <i>nohup 
backup &amp;&gt; /tmp/backup.log &</i>, close the xterm and go to bed. The backup only
takes about 15 minutes and so can also be done at any convenient time in a work day.
When next at the computer I <i>cat /tmp/backup.log</i> and check that backup
went well.
</p>

<p>
If I also want to verify the backup ISO I note down the first and last four or 
five letters of the listed
ISO md5sum. As my burner will not reliably read back the CD just written I 
transfer the CD to my cdrom
and verify that the md5sums are identical using the script <a 
href="misc/keates/md5scd.sh.txt">md5scd</a>, see below for a listing.
If they are, I put that newly burned CD into the red case and the last burned CD 
into the green case ready
for the next backup cycle. If any possibility of confusion exists one can always 
check the date on the tar file.
Note that because of the burner not reliably reading the backup CD, that I have 
not included an automatic check of the md5sums, as failure to validate does not mean the CD
is in error, just the read from  the burner was. In fact, I have never 
experienced a md5sum compare failure when using my cdrom. I consider the MD5 checksum
essential as even a single bit error could conceivably corrupt the whole compressed archive.
</p>

<pre>
#!/bin/bash
#------------------------------------------------------------------------
# md5scd ---- Data CD md5sum Verifier
# Script to automate determining Md5sum for ISO9660 CDs
# NOTE - This script assumes that correct md5sum is known and one wishes
# to verify that a particular CD copy has been burnt correctly.
# If working from a downloaded ISO image use "md5sum ISO" at command line
#------------------------------------------------------------------------
# Requires standard tools found in all Linux Distributions
# If script invoked as user, check all permissions, groups, etc.

# Missing arguments?
if [ $# -ne 2 ]
then
  echo "Usage - md5scd mountpoint device, ex - md5scd /mnt/cdrom /dev/hdc"
  exit 1
else
    : OK have arguments
fi

# Loop over md5sum determination ...100 good copies for install-fest?
again=yes
while [ "$again" = yes ]
   do
     echo "Please insert CD at $1 and press ENTER when ready"
     read                        #Wait for insertion of disk
     mount $1
     Block_Count=`df -k $1 | grep $1 | cut -c 25-30`
               #700Mb disk cannot exceed this ^^^^^ column limit in 1k blocks
     umount $1
     Md5sum_Cd=`dd if=$2 count=$Block_Count bs=1024 | md5sum`
     echo "The md5sum of the CD at $1 is " $Md5sum_Cd
     echo
     echo -n "Verify another CD? [yes/no]"
     read again   # Wait for "yes" -&gt; repeat, anything else -&gt; drop through'
   done
exit 0
</pre>

<h2>What do I do if my system crashes?</h2>

<p>
Before that eventuality one should make sure the backup disk will boot, make 
sure one understands the 
limitations of tomsrtbt and as the only editor available is VI, practice reading 
the various files placed
on the backup disk. The disk will have to be mounted first  <i>mount -t iso9660
/dev/xxx /mnt</i>.
It is possible to unzip and untar the tarred file using tomsrtbt by first using 
gzip
and then tar.
</p>

<p>
 However it is probably better to first check that the partition table is 
correct by <i>fdisk -l</i> and
reference to the stored table, and if not to restore the MBR 
<i>dd if=/mnt/MBR.bin of=/dev/hda bs=1 count=512</i>.
This will restore the primary partitions and the bootloader.
Then use fdisk and the partition table file to manually restore the extended 
partition and the logical
partitions within. This all requires nerve and practice! However any changes can 
be abandoned if not sure or only practicing.
</p>

<p>
Next do a clean install to the now proper partitions. Reboot to the point where 
one has a root console,
mount the backup CD and execute <i>tar xvzf /Mount_Dir/Backup_Tar_Filename</i>.
This will restore all of the
previous directories into their correct places, and should leave you with an 
almost
fully restored system.
</p>

<p>
Note that if the problem is only lost or corrupted files, one can also restore
any of the saved directories at any time by extracting with <i>tar xvzf 
/Mount_Dir/Backup_Tar_Filename /home</i>, for example.
</p>

<h2>The Proof of the Pudding</h2>

<p>
The test of any system is, "Does it work?" I initially verified that the backup 
CD does boot into Toms wonderful Linux system, that all of the text material was 
readable, of course fdisk -l did correspond to the stored version. I did not reinstate the MBR from
the binary image file, however it is there if I ever need it.
</p>

<p>The final test was to restore my Slackware 8.1 system in place of
the original Mandrake 9.0 system, before installing Windows and perhaps 
<i>needing</i> to restore.

<p>
In brief, 
<ol>
<li>I changed my menu.lst to reflect the fact that now we would boot Slackware 
not Mandrake
and formatted the partition i.e <i>mke2fs -j /dev/hda1</i>.
<li>Rebooted with the Slackware install disk in the drive and
the bios set to boot from cdrom. In 15 minutes everything was installed.
<li>Rebooted into the new system and at a root console mounted the last backup 
cd to /mnt and issued
<i>tar xvzf /mnt/last_dated_backup.tar.gz</i>.
</ol>
</p>
In a further five minutes this reinstalled all of the backed up
partition contents, and a reboot now brought me into a restored slackware 8.1 
with X set up and a kde login.
Because of not saving /dev some of the device permissions had to be reset, 
sound, etc, but this was trivial.
The whole process took less than half an hour. A far cry from a normal install, 
and then adding all the missing favourite
programs followed by a lengthy reconfiguration.
</p>

<h2>Conclusion</h2>

<p><b>Backup is easy to do and easy to keep doing</b>.
In use there are a number of small improvements that could be made. The manual 
backup and verification commands 
could be made shell variables and invoked with a single word. Also if the total 
file size becomes a factor one could
use the --exclude flag of tar to not include large sections of invariant code in 
the tar file, or use bzip2 compression.
As it is now, complete directories from root are saved.
</p>

<p>
The urgent need for the Windows applications turned out not to be so urgent, but 
provided the prod to actually backup regularly.
Perhaps my next project will be to install Wine and try to get those pesky 
applications to run within Linux, without
the need to always reboot.
</p>

<p>
I would be interested in any comments for improvement, indeed any feedback
would be welcome, particularly if glaring
flaws or omissions are evident. I can be reached at <a href="mailto:%6B%65%61%74%65%73%61%40%77%69%67%68%74%6D%61%6E%2E%63%61">this address</a>.
This scheme has been in use for only a short time but so far I'm very satisfied,
I encourage you to also institute regular backups. If you want a quick ready 
made approach try this one,a few changes to the scripts should have you backing up today and everyday after
that .
</p>


























<!-- *** BEGIN author bio *** -->
<P>&nbsp;
<P>
<!-- *** BEGIN bio *** -->
<P>
<img ALIGN="LEFT" ALT="[BIO]" SRC="../gx/2002/note.png">
<em>
Retired Control Systems Engineer, spent much of career designing and
implementing Computerized Control and Shutdown Systems for Canada's CANDU
Nuclear Reactors.  A programmer for over 40 yrs and a Linux enthusiast since
1994, first log entry shows 7.83 Bogomips on a 386 DX33 machine still running.
Linux and Golf are in first and second place among many other hobbies.
</em>
<br CLEAR="all">
<!-- *** END bio *** -->

<!-- *** END author bio *** -->


<!-- *** BEGIN copyright *** -->
<hr>
<CENTER><SMALL><STRONG>
Copyright &copy; 2003, Alan Keates.
Copying license <A HREF="../copying.html">http://www.linuxgazette.com/copying.html</A><BR> 
Published in Issue 91 of <i>Linux Gazette</i>, June 2003
</STRONG></SMALL></CENTER>
<!-- *** END copyright *** -->
<HR>

<!--startcut ==========================================================-->
<CENTER>
<!-- *** BEGIN navbar *** -->
<A HREF="collinge.html">&lt;&lt;&nbsp;Prev</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="index.html">TOC</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="../index.html">Front Page</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue91/keates.html">Talkback</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="../faq/index.html">FAQ</A>&nbsp;&nbsp;|&nbsp;&nbsp;<A HREF="kruk.html">Next&nbsp;&gt;&gt;</A>
<!-- *** END navbar *** -->
</CENTER>
</BODY></HTML>
<!--endcut ============================================================-->