FxT - Fast User or Kernel Tracing
                       University of New Hampshire

FxT is a fast tracing engine that can be used either in user land, in kernel
land, or both, maintained mostly by samuel.thibault@ens-lyon.org



                        FUT - Fast User Tracing

        FUT is the Fast User Tracing, the User part of FxT.  It permits
recording the execution trace along functions within applications (without any
kernel patching). To install FUT on your system, simply use the usual GNU way:

$ ./configure
$ make
$ sudo make install

This will notably install a fxt.pc file in /usr/local/lib/pkgconfig so that to
build your application you just need to add $(pkg-config --cflags fxt) to your
CFLAGS and $(pkg-cnfig --libs fxt) to your LDFLAGS.

tools/test.c shows an example of the use of FUT.



                        FKT - Fast Kernel Tracing



        FKT is the Fast Kernel Tracing, the Kernel part of FxT. It permits
recording the execution trace along functions within the kernel. Its purpose is
originately to instrument parts of the kernel, but it also revealed to be useful
for debugging or teaching.

	Please read 'INSTALL.FKT' first to setup FKT on your system for your kernel.
	Note that you do _not_ need to install FKT if you only need the Userland
FUT part.


	Once FKT is properly installed, running, the tools built and installed
in the PATH, tracing is quite easy:

	fkt_record -f output -k 0xf -- scp machine:file .

	Will trace the execution of 'scp machine:file .' and write the result
in file output. Then you can use fkt_print to analyze the recording (you
don't need to be running the fkt kernel, but having the proper modules in
/lib/modules might sometimes help).

	fkt_print -f output -t output.timedat > stats

	Then you can have a look at the stats file.

	fkt_timestats output,timedat

	Will produce output.eps, which will show you how long did each page of
data to get filled, hence, to some extent, the load of the system.


Table of contents
-----------------

- fkt_record
- fkt_setmask, fkt_enable and fkt_disable
- fkt_print
- Adding probes


fkt_record
----------

	fkt_record opens /dev/fkt and the output file and calls sendfile()
between them, which triggers the tracing. Before calling sendfile(), it can
fork() to exec() a user program as needed. The recording stops whenever the
file system is full or the given size (-s option) is reached, or a signal is
caught. As a result, one can stop the recording by pressing ^C for instance,
or as soon as the son exits if any (it sends a SIGCLD to fkt_record).

Options:
	-f file	precises the output record file. The location of that file
		should be carefully chosen: its recording shouldn't compete
		with what is actually traced, since not only will it interfere
		with the mere measures, but it may not be able to actually flush
		the recording when some blasting is running for instance.
		The best thing to do might be using a separate drive, with a
		single big partition, and use its block device as file name,
		since it would avoid all the filesystem stuff.

	-k mask	chooses which keymask will be set for the recording.
		It defaults to 0x1, which is only system calls, irqs and traps,
		but setting it to 0x3 activates probes from network device
		drivers, setting it to 0x7 also activates the ip probes, while
		setting it to 0x5 only activates ip probes.
		include/linux/fkt.h holds the definition of KEYMASKs for sets
		of probes which can be enabled and disable this way.
		Avoiding including useless layers gives better measurements.
		It can even be set to 0, which will prevent fkt from recording
		anything before fkt_enable is used for instance.

	-s size	restricts the recording to a given size. Suffixes like G, M,
		k are accepted. By default, the available size of the
		filesystem or the block device will be used

	-n	prevents fkt_record from fork()ing, so pressing ^C will be
		the only way to stop fkt_record.

	-S System.map	will load the given System.map file to include addresses
			of every function of the kernel, enabling proper display
			by fkt_print when -finstrument-functions in the built-in
			kernel. If not given, only addresses available in
			/proc/ksyms will be included in the trace. this option
			may be set several times.

	-p pow	will ask fkt to allocate 2^pow pages for its recording buffer.
		it is useful if load pikes are expected during the recording.
		the default is 7.
	
	-h	prints some reference help

fkt_setmask, fkt_enable, fkt_disable
------------------------------------

	These tools are particularly useful when fkt_record was given the -n
option: they modify the current keymask `on the fly' by doing an ioctl() on
/dev/fkt. For instance,

	fkt_record -k 0 -n &
	fkt_setmask 0xf
	some network stuff
	fkt_disable 1
	some other network stuff
	killall fkt_record

	Will record both network stuff in the same record file, but the second
one will not have any device driver (bit 1, i.e. mask 0x2) probe entry.

fkt_print
---------

	This tool analyzes a trace produced by fkt_record.

	It first shows the recording header.

	Then the full recording is dumped: for each probe triggered during the
recording, the following is shown:

	- the time since last probe record for the same cpu
	- the cpu on which it was running
	- the pid of the current process on this cpu
	- the code of the probe
	- the name corresponding to the code
	- the parameters of the probe, if any

	It then shows some statistics computed during reading the recording.

	- Cpu cycles: it should be quite similar between cpus

	- Process cycles: the number of cycles spent between every process,
both user&kernel level, so this adds to 100%.

	- Functions cycles: the time each function spent to achieve its work.
Note that testing is not computed here, so this doesn't adds to kernel time.

	- Nesting summary: for each encountered function, the functions it
called are displayed along with the time they spent to achieve their work. The
cycles this function spent itself is also displayed, so this is a good summary
to know the balance of the whole work for this function.

	- Accounting for every cycle exactly once: this summary shows each spent
cycle, either in system calls (S), traps (T), Irqs handling (I), kernel
functions (F) or processes: idle (i), fkt_record (f), son of fkt_record (u), or
other (o). Note that nesting is taken into account here, so the total should
be exactly 100%.

	- Histogram account for every cycle exactly once: this is the same as
above, except it is drawn as an histogram, allowing fast comparisons, especially
between records.

Options:
	-f file	precises the input record file

	-s pid	asks fkt_print to start taking statistics when the first entry
		for this pid is encountered.
		setting pid to 0 asks for no statistics at all, and setting it
		to -1 asks for full statistics (default).

	-p pid	asks fkt_print to start printing trace items when the first
		entry for this pid is encountered. With a prepended '+', only
		items for this pid will be printed.
		setting pid to 0 asks for no printing at all, and setting it to
		-1 asks for full printing (default).

	-d	enables debugging. It shouldn't be useful unless an analysis
		error happens (prefixed with ===)

	-k	asks fkt_print to have the histogram only show kernel cycles,
		getting rid of process time which might flatten it.
	
	-b file	asks fkt_print to print statistics about the fkt buffering
		mecanism in the given file. fkt_bufstats can then be used to get
		an .eps file from it, provided gnuplot is installed.

	-t file	asks fkt_print to print the time each page took to get filled.
		fkt_timestats can be used to get an .eps from it. fkt_timestats
		does the same, with a logarithmic scale.
		To some extent, it quickly shows the load of the machine during
		the recording.

	-h	prints some reference help

fkt_extract, fkt_select
-----------------------

	The problem with big traces is that it takes a long time to analyze
them. fkt_extract can extract a subtrace from a trace:

	fkt_extract trace_file1 trace_file2 23 4053

	Will extract 4031 pages from trace file trace_file1, starting from page
23 and write them to trace_file2. All the headers are copied as well, so
fkt_print will still work on trace_file2.

	To know the page numbers, you can use fkt_select.

	fkt_select trace.timedat

	Will open a window where the activity is drawn: it shows how fast pages
were filled during the trace, the x axis being absolute time in processor
cycles. a part of it can then be selected, zoomed, unzoomed, and finaly pressing
's' will print the extrema pages numbers on standard output, which can be used
as are.
	trace.timedat is obtained by fkt_print thanks to the -t option.
Hopefully, -p 0 -s 0 can be set to fasten its run, if only the .timedat file is
wanted for extracting a small part of the trace for a thorough analysis for
instance.

Adding probes
-------------

	The fastest way to add probes to file.c for instance is adding

	CFLAGS_file.o += -finstrument-functions

	to the Makefile, which tells gcc to automatically call probes at entry
and exit of every function of file.c. If file.o is loaded as a module, it should
be loaded *just before* launching fkt_record, for it needs both its address and
the file itself, to have the base address of the module in memory and the offset
of functions within the module.

	The drawback is that you can't get parameters or return code for
instance. Adding probes to every function might also be too much.

	Adding __attribute__ ((no_instrument_function)) before the name of a
function in its definition will prevent gcc from adding the call for this
function. Probes can then be added by hand:

void __attribute__ ((no_instrument_function)) myfun(int param1, int param2)
{
	int ret=0;
	FKT_PROBE2(FKT_SOME_KEYMASK,FKT_MYFUN_ENTRY_CODE,param1,param2);

	work... Do some ret = -ESOMETHING if needed...

	FKT_PROBE1(FKT_SOME_KEYMASK,FKT_MYFUN_EXIT_CODE,ret);
	return ret;
}

	FKT_MYFUN_ENTRY_CODE and FKT_MYFUN_EXIT_CODE must be uniquely defined in
include/linux/fkt.h and fkt/tools/names.c to be properly printed by fkt_print.


	But -finstrument-functions and hand-inserted probes should be removed
from any production release, since it will fail to be built in any non-fkt
patched kernel.

-- 
vim: textwidth=80