Open Fabrics Enterprise Distribution (OFED)
		      Tips for Working with OFED 1.4

		 	     December 2008

===============================================================================
Table of Contents
===============================================================================
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
4. External Module Compilation Over OFED-1.4
5. Adding/Deleting a patch to OFED package
6. Adding vendor specific actions to the installation of OFED
7. How to compile OFED sources manually

===============================================================================
1. OFED Utilities
===============================================================================

The OFED package includes utilities under <prefix>/bin, where <prefix> stands
for the OFED installation path. To retrieve this path, run the script 
"/etc/infiniband/info" as explained in Section 2.2 below.

Notes:
------
1. This document includes descriptions for a subset of the existing utilities.
   To learn about other utilities, use their --help flag.

2. The sources for all utilities are not part of the RPM installation. However,
   all sources exist in the openib-1.4.tgz tarball.


1.1 Device Information
----------------------
Device information can be obtained using several utilities:

a. ibv_devinfo

   ibv_devinfo prints the ca attributes.

   usage:
     ibv_devinfo

     Options:
       -d, --ib-dev=<dev>    use IB device <dev> (default: first device found)
       -i, --ib-port=<port>  use port <port> of IB device (default: all ports)
       -l, --list            print only the IB devices names
       -v, --verbose         print all the attributes of the IB device(s)

b. ibstat

	usage:
	ibstat [OPTIONS] <ca_name> [portnum]

	Options:
	-d	debug
	-l	list all IB devices
	-s	print short device summary
	-p	print port GUIDs
	-V	print ibstat version information and exit
	-h	print usage

	Examples:
	ibstat -l 		# list all IB devices
	ibstat mthca0 2 	# stat port 2 of mthca0

c. Using sysfs file system
	The driver supports the sysfs file system under: /sys/class/infiniband

	Examples:

	> ls /sys/class/infiniband/mthca0/
	board_id device fw_ver hca_type hw_rev node_desc node_guid node_type
	ports sys_image_guid

	> cat /sys/class/infiniband/mthca0/board_id
	MT_0200000001

	> ls /sys/class/infiniband/mthca0/ports/1/
	cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid
	sm_sl state

	> cat /sys/class/infiniband/mthca0/ports/1/state
	4: ACTIVE

1.2 Performance Tests
---------------------
	The following performance tests are provided with the OFED release:

	1. Latency tests:
	   - ib_read_lat: RDMA read
	   - ib_write_lat: RDMA write
	   - ib_send_lat: UD, UC and RC (default) send

	2. Bandwidth tests:
	   - ib_read_bw: RDMA read
	   - ib_write_bw: RDMA write
	   - ib_send_bw: UD, UC and RC (default) send

	Usage:
	Server: <test name> <options>
	Client: <test name> <options> <server IP address>
		<server IP address> is an Ethernet or IPoIB address.
	--help	  lists the available <options>. The same options must be
		  passed to both server and client.	
	
	Note: See PERF_TEST_README.txt for more information on the performance
	      tests.
	
	Example: ib_send_bw
	Usage:
	ib_send_bw 			start a server and wait for connection
	ib_send_bw <host>		connect to server at <host>

	options:
	-p, --port=<port> 		listen on/connect to port <port>
						(default: 18515)
	-d, --ib-dev=<dev> 		use IB device <dev>
						(default: first device found)
	-i, --ib-port=<port> 		use port <port> of IB device
						(default: 1)
	-c, --connection=<RC/UC/UD> 	connection type RC/UC/UD (default: RC)
	-m, --mtu=<mtu> 		mtu size (default: 1024)
	-s, --size=<size> 		size of message to exchange
						(default: 65536)
	-a, --all 			run sizes from 2 up to 2^23
	-t, --tx-depth=<dep> 		size of tx queue (default: 300)
	-n, --iters=<iters> 		number of exchanges
						(at least 2, default: 1000)
	-b, --bidirectional 		measure bidirectional bandwidth
						(default: unidirectional)
	-V, --version display version number

1.3 Ping-pong Example Tests
---------------------------
	The ping-pong example tests provide basic connectivity tests. Each test
	has a help message (-h).
	- ibv_ud_pingpong
	- ibv_rc_pingpong
	- ibv_srq_pingpong
	- ibv_uc_pingpong

	Example: ibv_ud_pingpong --h
	Usage:
	ibv_ud_pingpong  	start a server and wait for connection
	ibv_ud_pingpong <host>	connect to server at <host>

	options:
	-p, --port=<port> 	listen on/connect to port <port>
					(default: 18515)
	-d, --ib-dev=<dev> 	use IB device <dev>
					(default: first device found)
	-i, --ib-port=<port> 	use port <port> of IB device (default: 1)
	-s, --size=<size> 	size of message to exchange (default: 2048)
	-r, --rx-depth=<dep> 	number of receives to post at a time
					(default: 500)
	-n, --iters=<iters> 	number of exchanges (default: 1000)
	-e, --events 		sleep on CQ events (default: poll)


===============================================================================
2. Debug HOWTOs
===============================================================================

2.1 OFED Components and Version Information
-------------------------------------------
The text file BUILD_ID provides data on all OFED components (whether installed
or not). This file is a part of the ofed-docs RPM and installed under 
/usr/share/doc/ofed-docs-1.4 on RedHat, and under 
/usr/share/doc/packages/ofed-docs-1.4 on SuSE.

The same information can be obtained by executing the 'ofed_info' command. For
example:

> ofed_info
OFED-1.4
libibverbs:
git://git.openfabrics.org/ofed_1_4/libibverbs.git ofed_1_4
commit b00dc7d2f79e0660ac40160607c9c4937a895433
libmthca:
git://git.kernel.org/pub/scm/libs/infiniband/libmthca.git master
commit be5eef3895eb7864db6395b885a19f770fde7234
libmlx4:
git://git.openfabrics.org/ofed_1_4/libmlx4.git ofed_1_4
commit fd418d6ee049afe76bb769aff87c303b96848495
libehca:
git://git.openfabrics.org/ofed_1_4/libehca.git ofed_1_4
commit e0c2d7e8ee2aa5dd3f3511270521fb0c206167c6
libipathverbs:
git://git.openfabrics.org/~ralphc/libipathverbs ofed_1_4
commit 65e5701dbe7b511f796cb0026b0cd51831a62318
libcxgb3:
git://git.openfabrics.org/~swise/libcxgb3.git ofed_1_4
commit f685c8fe7e77e64614d825e563dd9f02a0b1ae16
libnes:
git://git.openfabrics.org/~glenn/libnes.git master
commit 07fb9dfbbb36b28b5ea6caa14a1a5e215386b3e8
libibcm:
git://git.openfabrics.org/~shefty/libibcm.git master
commit 7fb57e005b3eae2feb83b3fd369aeba700a5bcf8
librdmacm:
git://git.openfabrics.org/~shefty/librdmacm.git master
commit e0b1ece1dc0518b2a5232872e0c48d3e2e354e47
libsdp:
git://git.openfabrics.org/ofed_1_4/libsdp.git ofed_1_4
commit 02404fb0266082f5b64412c3c25a71cb9d39442d
sdpnetstat:
git://git.openfabrics.org/~amirv/sdpnetstat.git ofed_1_4
commit 75a033a9512127449f141411b0b7516f72351f95
srptools:
git://git.openfabrics.org/ofed_1_3/srptools.git ofed_1_3
commit d3025d0771317584e51490a419a79ab55650ebc9
perftest:
git://git.openfabrics.org/~orenmeron/perftest.git master
commit ca629627c7a26005a1a4c8775cc01f483524f1c4
qlvnictools:
git://git.openfabrics.org/~ramachandrak/qlvnictools.git ofed_1_4
commit 1dc6e51a728cbfbdd2018260602b8bebde618da9
tvflash:
git://git.openfabrics.org/ofed_1_4/tvflash.git ofed_1_4
commit e1b50b3b8af52b0bc55b2825bb4d6ce699d5c43b
mstflint:
git://git.openfabrics.org/~orenk/mstflint.git master
commit 9ddeea464e946cd425e05b0d1fdd9ec003fca824
qperf:
git://git.openfabrics.org/~johann/qperf.git/.git master
commit bee05d35b09b0349cf4734ae43fc9c2e970ada8c
ibutils:
git://git.openfabrics.org/~orenk/ibutils.git master
commit 6516d16e815c68fa405562ea773b0c5215c1b70c
ibsim:
git://git.openfabrics.org/~sashak/ibsim.git master
commit eff83c7a522dea41c21e15746b1c58ff21fdecaa

ofa_kernel-1.4:
Git:
git://git.openfabrics.org/ofed_1_4/linux-2.6.git ofed_kernel
commit 60ca4b0e03aa5acccb01a3e0430ba240ad521547

# MPI
mvapich-1.1.0-3143.src.rpm
mvapich2-1.2p1-1.src.rpm
openmpi-1.2.8-1.src.rpm
mpitests-3.1-891.src.rpm


2.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
on the machine.

For example:

 > /etc/infiniband/info
   prefix=/usr
   Kernel=2.6.9-78.ELsmp

   MODULES:  CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m
   CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y
   CONFIG_INFINIBAND_MTHCA=m CONFIG_IPATH_CORE=m CONFIG_INFINIBAND_IPATH=m
   CONFIG_INFINIBAND_IPOIB=m

   User level:  --kernel-version 2.6.9-78.ELsmp --kernel-sources
   /lib/modules/2.6.9-78.ELsmp/build --with-libibcm --with-libibverbs
   --with-libipathverbs --with-libmthca --with-mstflint --with-perftest

2.3 Building/Installing InfiniBand (IB) Modules With Debug Information
----------------------------------------------------------------------
To compile/build/install the IB modules so that they will contain debug
information, set OPENIB_KERNEL_EXTRA_CFLAGS="-g" in your environment
before running OFED's install.pl/build.sh .


===============================================================================
3. Pinning (Locking) User Memory Pages
===============================================================================

Memory locking is managed by the kernel on a per user basis. Regular users (as
opposed to root) have a limited number of pages which they may pin, where
the limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory than
it is allowed to pin.

The user can change the system per-process memory lock limit by adding
the following two lines to the file /etc/security/limits.conf:

  *  soft  memlock  <number>
  *  hard  memlock  <number>

  where <number> denotes the number of KBytes that may be locked by a
  user process.

The above change to /etc/security/limits.conf will allow any user process in the
system to lock up to <number> KBytes of memory.

On some systems, it may be possible to use "unlimited" for the size to disable
these limits entirely.
	      
Note: The file /etc/security/limits.conf contains further documentation.


===============================================================================
4. External Module Compilation Over OFED-1.4
===============================================================================

To build kernel modules depending on OFED's modules, take the Modules.symvers
file from <prefix>/src/openib/Module.symvers (part of the kernel-ib-devel RPM)
and copy it to the modules subdir and then compile your module.

If <prefix>/src/openib/Module.symvers does not exist or it is empty, use the
create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the
Module.symvers file.

See "Module versioning & Module.symvers" in the modules.txt from kernel
documentation (e.g. linux-2.6.20/Documentation/kbuild/modules.txt).


===============================================================================
5. Adding/Deleting a patch to OFED package
===============================================================================
If there is a need to add or delete a patch to OFED package one should use the 
ofed_patch.sh script, that is available under the docs directory.

This script supports kernel sources only.

Usage:
Add patch to OFED:
ofed_patch.sh  --add
               --ofed|-o <path_to_ofed>
               --patch|-p <path_to_patch>
               --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>

Remove patch from OFED:
ofed_patch.sh  --remove
               --ofed|-o <path_to_ofed>
               --patch|-p <patch name>
               --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>

Examples:
ofed_patch.sh --add --ofed /tmp/OFED-1.4/ --patch /tmp/cma_fix.patch --type kernel

ofed_patch.sh --remove --ofed /tmp/OFED-1.4/ --patch cma_fix.patch --type kernel

===============================================================================
6. Adding vendor specific actions to the installation of OFED
===============================================================================

Vendors that want to add actions to the install/uninstall process of OFED can
bind external scripts to hooks in install.pl and ofed_uninstall.sh.

6.1  Specifying vendor scripts and configuration parameters
-----------------------------------------------------------
This option is only available when installing ofed in non interactive mode.
Edit the OFED configuration file (ofed.conf ) and add the lines below (you don't have to use all of them).

# Script to run before install process starts
vendor_pre_install=my_pre_install.sh
# Script to run after install process finishes
vendor_post_install=my_post_install.sh
# Script to run before uninstall process starts
vendor_pre_uninstall=my_pre_uninstall.sh
# Script to run after uninstall process finishes
vendor_post_uninstall=my_post_uninstall.sh

You can also also add vendor specific configuration parameters. Lines that start
with vendor_config_ will not be parsed by install.pl and can be parsed by one
of the vendor scripts

vendor_config_something=value_for_that_something

Running ./install.pl -c ofed.conf in the OFED directory will now invoke the
relevant vendor specific actions.

6.2. Requirements from vendor scripts
-------------------------------------
The script files that are given to install.pl in ofed.conf should
	- be located in the root directory of OFED
	- return zero on success

If vendor script fails it fails the entire installation.

6.3 Skeleton for pre/post install vendor script
-----------------------------------------------
install.pl passes some useful installation variables to the vendor pre/post
install scripts environment. See the example below for a typical usage.

#!/bin/bash

eval $*

# The following env. parameters are set at this point
# 
# CONFIG: full path filename of the OFED configuration file
# RPMS: directory of binary RPMs
# SRPMS: directory of source RPMS
# PREFIX: prefix of installation
# TOPDIR: root of OFED package
# QUIET: quiet operation indicator

function readconf() {
        local config=$1
        while read line; do
            # skip comments
            [[ ${line:0:1} == "#" ]] && continue

            # skip empty lines
            [[ -z "$line" ]] && continue

            # parse line
            token=$(echo $line|cut -f1 -d=)
            value=$(echo $line|cut -f2 -d=)

        done < "$config"
}

readconf $CONFIG

exit 0

===============================================================================
7. How to compile OFED sources manually
===============================================================================

These are the instructions how to compile and install kernel and user parts "manually" 
meaning without building the RPMs and without using the install.pl script.

7.1 Compiling the kernel modules
--------------------------------
1. tar xzf OFED-1.4.tgz
2. rpm -ihv OFED-1.4/SRPMS/ofa_kernel-1.4-ofed1.4.src.rpm
3. cd /usr/src/redhat/SOURCES
4. tar xzvf ofa_kernel-1.4.tgz
5. cd ofa_kernel-1.4
6. configure: 
   run ./configure --help for a list of options.
   basic invocation is:
   ./configure --with-core-mod --with-ipoib-mod --with-mthca-mod --with-mlx4_core-mod --with-mlx4_inf-mod
7. make
   make install
     
NOTES: 
  1. configure applies the patches to the source code according to the current 
     kernel. If you wish to rerun configure it is recommend to untar the source 
     code tree from the beginning and start with a clean state.
     An alternative is to pass the option: --without-patch to the configure invocation.
      
  2. The modules select for install are written to configure.mk.kernel

7.2 Compiling the user space libraries
--------------------------------------
To install user space library from the source RPM provided by OFED-1.4 manually,
do the following:

Example for libibverbs:
 
1. tar xzf OFED-1.4.tgz
2. rpm -ihv SRPMS/libibverbs-1.1.2-1.ofed1.4.src.rpm
3. cd /usr/src/redhat/SOURCES (for RedHat)
   or
   cd /usr/src/packages/SOURCES (for SuSE)
4. tar xzf libibverbs-1.1.2.tgz
5. cd libibverbs-1.1.2
6. ./configure (specify parameters, if required)
7. make
8. make install