%
% Copyright (C) 2003-2009 FAUmachine Team <info@faumachine.org>.
% This program is free software. You can redistribute it and/or modify it
% under the terms of the GNU General Public License, either version 2 of
% the License, or (at your option) any later version. See COPYING.
%

This chapter tells you how to make the hardware of your virtual machine
fail. Section \ref{s:injectingfaults.faults} gives you an overview over
available failures. You can use the graphical user interface to inject
faults interactively (section \ref{s:injectingfaults.gui} or you can
include fault injection into your scripts for automated testing
(sections \ref{s:injectingfaults.auto} and \ref{s:testautomation}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Available Failures}
\label{s:injectingfaults.faults}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The following sections show in which ways you can make your virtual
hardware fail.

We distinguish between transient and permanent faults. An example for a
transient fault is a bitflip, an example for a permant fault is a
stuck-at-one fault.

In a bitflip, a single bit (e.g.\ in a processor register or in main
memory) flips to zero (if it was one, before) or to one (if it was zero,
before).
That is all, nothing more happens. So if the next thing you do is write
a new value to the register or memory location containing the flipped
bit, this bit will happily change state to the new value and you won't
even notice anything bad has happened.
Of course, if the next thing you do is read from this register or memory
location, you will get the faulty value and bad things may or may not
happen.

Things are different for a permant fault. Permant in this case does not
mean ``forever''. It just means, that this fault is not transient and
may persist for a certain duration of time.
A stuck-at-one fault is such a permant fault. A bit becomes stuck at
the value one and does not change its value any more. Even if the next
thing you do is write a new value to the register or memory location
containing the flipped bit, and the bit should really be zero after the
write, it won't be, because it is stuck at the value one.

%%======================================================================
\subsection{Processor Failures}
\label{s:injectingfaults.faults.cpu}
%%======================================================================
The only failures you can currently induce in the processor are random
transient bitflips in processor registers. 

You will have to specify the processor (on single processor machines
the processor usually has the ID 0), the register and the bit to flip.
Since the processor of the virtual machines are all i386 architecture
processors, the possible registers are
``eax'', ``ebx'', ``ecx'', ``edx'', ``edi'', ``esi'', ``ebp'', ``esp'',
``eip'', ``eflags'', ``ds'', ``es'', ``fs'', and ``gs''.

Legal bit-IDs to flip are 0 to 31 for the long registers and 0 to 15 for
the short ones.

%%======================================================================
\subsection{Main Memory Failures}
\label{s:injectingfaults.faults.mem}
%%======================================================================
The only failures you can currently induce in the main memory are random
transient bitflips.

You will have to specify the address of the byte in which you want to
flip the bit as well as the bit to flip.
%%======================================================================
\subsection{Block Device Failures}
\label{s:injectingfaults.faults.block}
%%======================================================================
The only failures you can currently induce in a block device are
permanent failures of random bytes on this device.

You will have to specify the device, the first affected byte and the
number of affected bytes (range). Any time the operating systems tries
to read or write from the affected parts of this block device, the
controller will return an error code.

Of course you can specify several ranges for the same device.
%%======================================================================
\subsection{Network Failures}
\label{s:injectingfaults.faults.net}
%%======================================================================
You can make the network interfaces lose packets when sending and
receiving. These are permanent faults, i.e.\ the affected network
interface will lose packets as long as this fault is active (turned on).

You will have to specify the network interface, whether it is a send or
receive problem and the percentage of packages lost (0\% means no
packets are lost which is equivalent to no failure).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Injecting Faults Using the GUI}
\label{s:injectingfaults.gui}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Fault injection via the graphical user interface lets you inject faults
interactively at your leisure using the \GUIMenu{Inject}-menu.

%%======================================================================
\subsection{Creating a New Fault Description}
\label{s:injectingfaults.gui.new}
%%======================================================================
To create a new fault description, you will need to specify the
necessary information conerning your fault (see section
\ref{s:injectingfaults.faults}) in the \GUIWindow{Fault
Description}-window. You can open this window using the
entry \GUIMenu{Show List} from the \GUIMenu{Inject}-menu and then choosing
\GUIButton{Add}. You can also choose \GUIMenu{Edit New} from the
\GUIMenu{Inject}-menu directly.

This window will give you a list of the hardware devices available for
fault injection to the left. If you choose a device, a new list or text
entry field will appear to let you enter more details.

If you choose a memory device, you will have to enter the address of the
affected byte as a hexadecimal number. After you have entered this
number and hit the Return-Key, you can choose the number of the bit to
flip from a list. After you have chosen a number, an
\GUIButton{OK}-button will appear in the lower left hand corner of the
window.
Choose \GUIButton{OK} to add the new fault to the list of
currently loaded fault descriptions.
Choose \GUIButton{Cancel} anytime
to abort creating a new fault description.

%%======================================================================
\subsection{Editing an Existing Fault Description}
\label{s:injectingfaults.gui.edit}
%%======================================================================
To edit an existing fault description choose entry \GUIMenu{Show List}
from the \GUIMenu{Inject}-menu to display the list of loaded fault
descriptions. Then choose a fault description and click
\GUIButton{Edit}. The \GUIWindow{Fault Description}-window filled with
the details of the chosen fault description will appear. 
Choose \GUIButton{OK} to commit the changes to this fault description.
Choose \GUIButton{Cancel} anytime
to abort the changes.

%%======================================================================
\subsection{Deleting an Existing Fault Description}
\label{s:injectingfaults.gui.delete}
%%======================================================================
To delete an existing fault description choose entry \GUIMenu{Show List}
from the \GUIMenu{Inject}-menu to display the list of loaded fault
descriptions. Then choose a fault description and click
\GUIButton{Delete}. You will not be asked to confirm the deletion!
If the fault is currently active, this
will \emph{not} deactivate the fault! Use with care!

%%======================================================================
\subsection{Saving Fault Descriptions to a File}
\label{s:injectingfaults.gui.save}
%%======================================================================
You can save the currently loaded list of fault descriptions to a file.
Choose entry \GUIMenu{Show List} from the \GUIMenu{Inject}-menu to
display the list of loaded fault descriptions.
Click \GUIButton{Save} and a file selection box will pop up to let you
select a filename and location for saving. Click \GUIButton{OK} in the
file selection box to save the list of fault descriptions.
%%======================================================================
\subsection{Loading Fault Descriptions from a File}
\label{s:injectingfaults.gui.load}
%%======================================================================
If you have previously saved a fault description list (see section
\ref{s:injectingfaults.gui.save}, you can load it again. 
Choose entry \GUIMenu{Show List} from the \GUIMenu{Inject}-menu to
display the list of currently loaded fault descriptions.
Click \GUIButton{Load} and a file selection box will pop up to let you
select the filename and location of the file to load. Click \GUIButton{OK} in the
file selection box to actually load the file. The fault descriptions
from the file will be appended to the list of currently loaded fault
descriptions. If you do not want this, use \GUIButton{Clear} first, to
clear the current list of fault descriptions. If there are
currently active faults, this will \emph{not} deactivate these
faults! Use with care!

%% FIXME
Don't try to load a file which does not contain fault descriptions in
the right format! This will currently crash the GUI.
%%======================================================================
\subsection{Miscellaneous and Summary}
\label{s:injectingfaults.gui.misc}
%%======================================================================
Use the \GUIMenu{Show List} to look at the list of faults currently
loaded. This list tells you whether the fault is currently active
and shows you the fault description.

There are buttons for the following actions along the bottom of the
\GUIWindow{Show Fault List}-window:
\begin{description}
\item[\GUIButton{Load}]{
	Load a previously saved fault description list from a file, appending it to
	the list of currently loaded fault descriptions.
}
\item[\GUIButton{Save}]{
	Save a fault description list to a file.
}
\item[\GUIButton{Add}]{
	Add a fault description to the list.
}
\item[\GUIButton{Edit}]{
	Edit the selected fault description.
}
\item[\GUIButton{Delete}]{
	Delete the selected fault description. If the fault is currently active, this
	will \emph{not} deactivate the fault! Use with care!
}
\item[\GUIButton{Clear}]{
	Clear the fault description list. This unloads the fault
	description list. If there are
	currently active faults, this will \emph{not} deactivate these
	faults! Use with care!
}
\item[\GUIButton{Close}]{
	Close the window. This does not unload or clear the fault list
	and does not deactivate any faults. It simply gets the window
	out of your way.
}
\end{description}

You can create or edit a faultload file even when the virtual machine
is not turned on. Choose \GUIMenu{Inject}\rArrow\GUIMenu{Show List} to
show the list of currently loaded fault descriptions. Choose
\GUIButton{Add} to add a new fault description.

\removed{%
%%======================================================================
\subsection{ Interpreting Tracer Output}
%%======================================================================
To be added ...
}%/removed


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Automated Fault Injection}
\label{s:injectingfaults.auto}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Please refer to section \ref{s:testautomation} for general information
on how to use Expect for test automation and automatic experiment
control. This section only treats fault injection via Expect.

\command{expect} is an automatic experiment controller, which can  wait for
certain events to occur on a virtual machine and in return send some
input  to this virtual machine. The input \command{expect} sends will usually
simulate a user sitting in front of the machine. Events occurring on a
virtual machine are such things as screen output (e.g. the login-prompt
appearing). \command{expect} is generating a workload.

Additionally, \command{expect} may activate/deactivate faults in a
virtual machine. \command{expect} is generating a faultload.
\command{expect} is able to activate all faults defined in
\Filename{example/faultloads/README}.

Currently, we use it as experiment
controller creating a workload in a client/server system, see example
in \Filename{example/oracle}. For a short description:
\command{serial\_interface} is used to translate the input/output
of a serial port of a node into \command{expect}-events. Using this
Babel-fish it is possible to respond to characters on a serial line
in \command{expect}-protocol.
Then, \command{expect} logs in at four virtual machines each over serial
port \Filename{ttyS0} starting application servers on the serial
ports (\Filename{ttyS1}, \Filename{ttyS2} and \Filename{ttyS3}).
\command{expect} simulates the user by creating
partly random input at the masks of the application servers at all
serial interface simultanously. In detail: there are 4 virtual
machines each running 3 application servers at its serial ports.
Thus, there are 12 serial interfaces controlled with \command{expect}.

Additionally \command{expect} creates a faultload at the virtual
machine running an Oracle database server. The fault is selectable
via setting a generic from the command line starting \command{expect}
For this setup \command{expect} seems to work well.

If you want \command{expect} to wait for the login-prompt, for example,
your virtual machine will have to fulfill the following requirements:
\begin{itemize}
\item{
        Configure the virtual machine to have a serial interface
        (\ref{s:configmanual.ser}).
}
\item{
        Configure your virtual machine to run a \command{getty}
        listening on the serial interface. (Don't use \command{mgetty}
        for the moment!) Debian-\command{getty} has been tested and
        works, whereas Caldera-\command{getty} does not work.
        
        It should probably work to insert a line such as
\begin{alltt}
T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
\end{alltt}
        into \Filename{/etc/inittab} on your virtual machine. On
        Debian-machines, for example, you will just need to uncomment this line.
}
\item{
	Connect a serial\_terminal or serial\_pattern to the
	corresponding serial interface of that virtual machine
	(\Filename{system.vhdl}).
}
\item{
	Write a login script in a VHDL-process, sending the login name,
	after recognition of the login: prompt.
}
\end{itemize}

Ready to go. Good Luck!

To inject faults clamp shortcuts to the fault injector signals
of the components. E.g. if you want to make the harddisk inoperatable
a piece of VHDL source could be:

\begin{alltt}
    SIGNAL disk\_fault : boolean;
    .
    .
    .
    hda : idedisk PORT MAP(ide => ide\_bus);
    .
    .
    .
    inject : PROCESS BEGIN
        disk\_fault'shortcut\_out\_add("hda\%byte\_fault/0/0");
        disk\_fault <= 1=1;
        WAIT;
    END PROCESS inject;
\end{alltt}

Here is some rough information on the syntax of the fault descriptions
which you can use in the shortcut (printf-like notation):
You will have to substitute the VHDL-name of the component instead of
"cpu", "eth", "hd", "memory".
\begin{alltt}
    /cpu/%d/%s/%d
    /eth/%d/%s/%d
    /hd/%c/%d/%d
    /memory/%d/%d

Semantics:
    /cpu/<cpu-id>/<register>/<bit>
        <cpu-id>: The ID of the CPU. For monoprocessors this is 0, possible
            values for multiprocessors depend on the setup.
        <register>: Which register. Possible registers are most
            intel-architecture registers, eg. eax, ebx, ecx, esp, eip etc.
        <bit>: Which bit to flip. Must be between 1 and 32 (inclusive).

    /eth/<interface>/<type>/<percent>
        <interface>: Interface number, possible values are usually between 0 and
            15. Which interfaces are available depends on the configuration.
        <type>: "receive" or "send"
        <percent>: Percentage of packages to drop.

    /fd/<drive>/<start>/<range>
        <drive>: Which drive (usually 0 or 1 are possible). Availability depends
            on the node configuration.
        <start>: First defect byte.
        <range>: How many defect bytes. All bytes falling within
            <start> .. <start>+<range> are defect.
            if range is -1, the whole disk is defect

    /hd/<drive>/<start>/<range>
        <drive>: Which drive (usually a, b, c etc. are possible). Availability depends
            on the node configuration.
        <start>: First defect byte.
        <range>: How many defect bytes. All bytes falling within
            <start> .. <start>+<range> are defect.
            if range is -1, the whole disk is defect

    /memory/<address>/<bit>
        <address>: Address of the defect word (must be word aligned).
        <bit>: Which bit to flip. Must be between 1 and 32 (inclusive).
\end{alltt}