1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
Open Fabrics Enterprise Distribution (OFED)
Version 1.4
README
December 2008
This is the OpenFabrics Enterprise Distribution (OFED) version 1.4
software package supporting InfiniBand and iWARP fabrics. It is composed
of several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet or an iWARP network.
*** Note: If you plan to upgrade OFED on your cluster, please upgrade all
its nodes to this new version.
This document includes the following sections:
1. HW and SW Requirements
2. OFED Package Contents
3. Installing OFED Software
4. Starting and Verifying the IB Fabric
5. MPI (Message Passing Interface)
6. Related Documentation
OpenFabrics Home Page: http://www.openfabrics.org
The OFED rev 1.4 software download available in
http://www.openfabrics.org/builds/ofed-1.4/release/
Please email bugs and error reports to your InfiniBand vendor, or use bugzilla
https://bugs.openfabrics.org/
1. HW and SW Requirements:
==========================
1) Server platform with InfiniBand HCA or iWARP RNIC (see OFED Distribution
Release Notes for details)
2) Linux operating system (see OFED Distribution Release Notes for details)
3) Administrator privileges on your machine(s)
4) Disk Space: - For Build & Installation: 300MB
- For Installation only: 200MB
5) For the OFED Distribution to compile on your machine, some software
packages of your operating system (OS) distribution are required. These
are listed here.
OS Distribution Required Packages
--------------- ----------------------------------
General:
o Common to all gcc, glib, glib-devel, glibc, glibc-devel,
glibc-devel-32bit (to build 32-bit libraries on x86_86
and ppc64), zlib-devel
o RedHat, Fedora kernel-devel, rpm-build
o SLES 9.0 kernel-source, udev, rpm
o SLES 10.0 kernel-source, rpm
Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit
glibc-devel should be installed.
Specific Component Requirements:
o Mvapich a Fortran Compiler (such as gcc-g77)
o Mvapich2 libstdc++-devel, sysfsutils (SuSE),
libsysfs-devel (RedHat5.0, Fedora C6)
o Open MPI libstdc++-devel
o ibutils tcl-8.4, tcl-devel-8.4, tk, libstdc++-devel
o tvflash pciutils-devel
o mstflint libstdc++-devel (32-bit on ppc64), gcc-c++
Note: The installer will warn you if you attempt to compile any of the
above packages and do not have the prerequisites installed.
*** Important Note for open-iscsi users:
Installing iSER as part of OFED installation will also install open-iscsi.
Before installing OFED, please uninstall any open-iscsi version that may
be installed on your machine. Installing OFED with iSER support while
another open-iscsi version is already installed will cause the installation
process to fail.
2. OFED Package Contents
========================
The OFED Distribution package generates RPMs for installing the following:
o OpenFabrics core and ULPs
- HCA drivers (mthca, mlx4, ipath, ehca)
- iWARP driver (cxgb3, nes)
- core
- Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER
Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA.
o OpenFabrics utilities
- OpenSM: InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
o MPI
- OSU MVAPICH stack supporting the InfiniBand and iWARP interface
- Open MPI stack supporting the InfiniBand and iWARP interface
- OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface
- MPI benchmark tests (OSU BW/LAT, Intel MPI Benchmark, Presta)
o Extra packages
- open-iscsi: open-iscsi initiator with iSER support
- ib-bonding: Bonding driver for IPoIB interface
o Sources of all software modules (under conditions mentioned in the
modules' LICENSE files)
o Documentation
3. Installing OFED Software
============================
The default installation directory is: /usr
Install Quick Guide:
1) Download and extract: tar xzvf OFED-1.4.tgz file.
2) Change into directory: cd OFED-1.4
3) Run as root: ./install.pl
4) Follow the directions to install required components. For details, please see
OFED_Installation_Guide.txt under OFED-1.4/docs.
Notes:
1. The install script removes previously installed IB packages and
re-installs from scratch. You will be prompted to acknowledge the deletion
of the old packages. However, configuration files (.conf) will be
preserved and saved with a ".rpmsave" extension.
2. After the installer completes, information about the OFED
installation such as the prefix, the kernel version, and
installation parameters can be found by running
/etc/infiniband/info.
3. Information on the driver version and source git trees can be found
using the ofed_info utility
4. Starting and Verifying the IB Fabric
=======================================
1) If you rebooted your machine after the installation process completed,
IB interfaces should be up. If you did not reboot your machine, please
enter the following command: /etc/init.d/openibd start
2) Check that the IB driver is running on all nodes: ibv_devinfo should print
"hca_id: <linux device name>" on the first line.
3) Make sure that a Subnet Manager is running by invoking the sminfo utility.
If an SM is not running, sminfo prints:
sminfo: iberror: query failed
If an SM is running, sminfo prints the LID and other SM node information.
Example:
sminfo: sm lid 0x1 sm guid 0x2c9010b7c2ae1, activity count 20 priority 1
To check if OpenSM is running on the management node, enter: /etc/init.d/opensmd status
To start OpenSM, enter: /etc/init.d/opensmd start
Note: OpenSM parameters can be set via the file /etc/opensm/opensm.conf
4) Verify the status of ports by using ibv_devinfo: all connected ports should
report a "PORT_ACTIVE" state.
5) Check the network connectivity status: run ibchecknet to see if the subnet
is "clean" and ready for ULP/application use. The following tools display
more information in addition to IB info: ibnetdiscover, ibhosts, and
ibswitches.
6) Alternatively, instead of running steps 3 to 5 you can use the ibdiagnet
utility to perform a set of tests on your network. Upon finding an error,
ibdiagnet will print a message starting with a "-E-". For a more complete
report of the network features you should run ibdiagnet -r. If you have a
topology file describing your network you can feed this file to ibdiagnet
(using the option: -t <file>) and all reports will use the names they
appear in the file (instead of LIDs, GUIDs and directed routes).
7) To run an application over SDP set the following variables:
env LD_PRELOAD='stack_prefix'/lib/libsdp.so
LIBSDP_CONFIG_FILE='stack_prefix'/etc/libsdp.conf <application name>
(or LD_PRELOAD='stack_prefix'/lib64/libsdp.so on 64 bit machines)
The default 'stack_prefix' is /usr
5. MPI (Message Passing Interface)
==================================
In Step 2 of the main menu of install.pl, options 2, 3 and 4 can
install one or more MPI stacks. Multiple MPI stacks can be installed
simultaneously -- they will not conflict with each other.
Three MPI stacks are included in this release of OFED:
- MVAPICH 1.1.0-3355
- Open MPI 1.3.2
- MVAPICH2 1.2p1
OFED also includes 4 basic tests that can be run against each MPI
stack: bandwidth (bw), latency (lt), Intel MPI Benchmark and Presta. The tests
are located under: <prefix>/mpi/<compiler>/<mpi stack>/tests/.
Please see MPI_README.txt for more details on each MPI package and how to run
the tests.
6. Related Documentation
========================
1) Release Notes for OFED Distribution components are to be found under
OFED-1.4/docs and, after the package installation, under
/usr/share/doc/ofed-docs-1.4 for RedHat
/usr/share/doc/packages/ofed-docs-1.4 for SuSE.
2) For a detailed installation guide, see OFED_Installation_Guide.txt.
3) For more information, please visit the OFED web-page http://www.openfabrics.org
For more information contact your vendor.
|
