1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
|
Open Fabrics Enterprise Distribution (OFED)
ConnectX driver (mlx4) in OFED 1.4.2 Release Notes
July 2009
===============================================================================
Table of Contents
===============================================================================
1. Overview
2. Supported Firmware Versions
3. VPI (Virtual Process Interconnect)
4. Infiniband new features and bug fixes since OFED 1.3.1
5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
7. New features and bug fixes since OFED 1.4.1
8. Known Issues
9. mlx4 Available Parameters
===============================================================================
1. Overview
===============================================================================
mlx4 is the low level driver implementation for the ConnectX adapters designed
by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter,
as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports
Infiniband and Ethernet NIC configurations. To accommodate the supported
configurations, the driver is split into three modules:
- mlx4_core
Handles low-level functions like device initialization and firmware
commands processing. Also controls resource allocation so that the
InfiniBand and Ethernet functions can share the device without
interfering with each other.
- mlx4_ib
Handles InfiniBand-specific functions and plugs into the InfiniBand
midlayer
- mlx4_en
A new 10G driver named mlx4_en was added to drivers/net/mlx4.
It handles Ethernet specific functions and plugs into the netdev mid-layer.
===============================================================================
2. Supported Firmware Versions
===============================================================================
- This release was tested with FW 2.6.000.
- The minimal version to use is 2.3.000.
- To use both IB and Ethernet (VPI) use FW version 2.6.0
===============================================================================
3. VPI (Virtual Protocol Interconnect)
===============================================================================
VPI enables ConnectX to be configured as an Ethernet NIC and/or an Infiniband
adapter.
o Overview:
The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and
Infiniband drivers.
It supplies the user with the ability to run Infiniband and Ethernet
protocols on the same HCA (separately or at the same time).
For more details on the Ethernet driver see MLNX_EN_README.txt.
o Firmware:
The VPI driver works with FW 25408 version 2.6.000 or higher.
One needs to use INI files that allow different protocols over same HCA.
o Installing the mlnx_en driver:
To install the OFED 1.4 with the Ethernet driver make sure that "mlx4_en=y"
in file ofed.conf
o Loading drivers:
To load the Ethernet driver one should set "MLX4_EN_LOAD=yes"
in file /etc/infiniband/openib.conf.
If "MLX4_EN_LOAD" is not marked as "yes", the Ethernet driver can be loaded
by running "/sbin/modprobe mlx4_en".
o Port type management:
By default both ConnectX ports are initialized as Infiniband ports.
If you wish to change the port type use the connectx_port_config script after
the driver is loaded.
Running "/sbin/connectx_port_config -s" will show current port configuration
for all ConnectX devices.
Port configuration is saved in file: /etc/infiniband/connectx.conf.
This saved configuration is restored at driver restart only if done via
"/etc/init.d/openibd restart".
Possible port types are:
"eth" - Always Ethernet.
"ib" - Always Infiniband.
"auto" - Link sensing mode - detect port type based on the attached
network type. If no link is detected, the driver retries link
sensing every few seconds.
Port link type can be configured for each device in the system at run time
using the "/sbin/connectx_port_config" script.
This utility will prompt for the PCI device to be modified (if there is only
one it will be selected automatically).
At the next stage the user will be prompted for the desired mode for each port.
The desired port configuration will then be set for the selected device.
Note: This utility also has a non interactive mode:
"/sbin/connectx_port_config [[-d|--device <PCI device ID>] -c|--conf <port1,port2>]".
- The following configurations are supported by VPI:
Port1 = eth Port2 = eth
Port1 = ib Port2 = ib
Port1 = auto Port2 = auto
Port1 = ib Port2 = eth
Port1 = ib Port2 = auto
Port1 = auto Port2 = eth
Note: the following options are not supported:
Port1 = eth Port2 = ib
Port1 = eth Port2 = auto
Port1 = auto Port2 = ib
===============================================================================
4. Infiniband new features and bug fixes since OFED 1.3.1
===============================================================================
Features that are enabled with FW 2.5.0 only:
- Send with invalidate and Local invalidate send queue work requests.
- Resize CQ support.
Features that are enabled with FW 2.6.0 only:
- Fast register MR send queue work requests.
- Local DMA L_Key.
- Raw Ethertype QP support (one QP per port) -- receive only.
Non FW dependent features:
- Allow 4K messages for UD QPs.
- Allocate/free fast register MR page lists.
- More efficient MTT allocator.
- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1).
- Pass congestion management class MADs to the HCA.
- Enable firmware diagnostic counters available via sysfs.
- Enable LSO support for IPOIB.
- IB_EVENT_LID_CHANGE is generated more appropriately.
- Fixed race condition between create QP and destroy QP (bugzilla 1389)
===============================================================================
5. Infiniband new features and bug fixes since OFED 1.4
===============================================================================
- Enable setting via module param (set_4k_mtu) 4K MTU for ConnectX ports.
- Support optimized registration of huge pages backed memory.
With this optimization, the number of MTT entries used is significantly
lower than for regular memory, so the HCA will access registered memory with
fewer cache misses and improved performance.
For more information on this topic, please refer to Linux documentation file:
Documentation/vm/hugetlbpage.txt
- Do not enable blueflame sends if write combining is not available
- Add write combining support for for PPC64, and thus enable blueflame sends.
- Unregister IB device before executing CLOSE_PORT.
- Notify and exit if the kernel module used does not support XRC. This is done
to avoid libmlx4 compatibility problem.
- Added a module parameter (log_mtts_per_seg) for number of MTTs per segment.
This enable to register more memory with the same number of segments.
===============================================================================
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
===============================================================================
6.1 Changes and New Features
----------------------------
- Added Tx Multi-queue support which Improves multi-stream and bi-directional
TCP performance.
- Added IP Reassembly to improve RX bandwidth for IP fragmented packets.
- Added linear skb support which improves UDP performance.
- Removed the following module parameters:
- rx/tx_ring_size
- rx_ring_num - number of RX rings
- pprx/pptx - global pause frames
The parameters above are controlled through the standard Ethtool interface.
Bug Fixes
---------
- Memory leak when driver is unloaded without configuring interfaces first.
- Setting flow control parameters for one ConnectX port through Ethtool
impacts the other port as well.
- Adaptive interrupt moderation malfunctions after receiving/transmitting
around 7 Tera-bytes of data.
- Firmware commands fail with bad flow messages when bringing an interface up.
- Unexpected behavior in case of memory allocation failures.
===============================================================================
7. New features and bug fixes since OFED 1.4.1
===============================================================================
- Added support for new device ID: 0x6764: MT26468 ConnectX EN 10GigE PCIe gen2
===============================================================================
8. Known Issues
===============================================================================
- mlx4_en driver is not supported on PPC64 and IA64
- The mlx4_en module uses a Linux implementation for Large Receive Offload
(LRO) in kernel 2.6.24 and later. These kernels require installing the
"inet_lro" module.
- The SQD feature is not supported:
- To load the driver on machines with 64KB default page size UAR bar must be
enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium with
SLES 11 or when 64KB page size enabled.
Perform the following three steps:
1. Add the following line in the firmware configuration (INI) file under the
[HCA] section:
log2_uar_bar_megabytes = 5
2. Burn a modified firmware image with the changed INI file.
3. Reboot the system.
- Ethernet in MLNX_OFED 1.4 is not supported for the following OPNs:
MHQH29-XTC
MHGH29-XTC
MHGH29-XSC
MHGH28-XTC
MHGH28-XTC
MHGH28-XSC
MHGH28-XSC
MHEH28-XTC
MHEH28-XSC
MHQH19-XTC
Attempting to use these cards as NICs will yield the following error
in /var/log/messages:
mlx4_core 0000:0d:00.0: command 0x9 failed: fw status = 0x8
<date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed Initializing port
<date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed starting port:1
As a workaround, use the MLNX_EN driver instead of OFED for Linux.
===============================================================================
9. mlx4 Available Parameters
===============================================================================
In order to set mlx4 parameters, add the following line(s) to /etc/modpobe.conf:
options mlx4_core parameter=<value>
and/or
options mlx4_ib parameter=<value>
and/or
options mlx4_en parameter=<value>
mlx4_core parameters:
set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (default 0)
msi_x: attempt to use MSI-X if nonzero (default 1)
enable_qos: Enable Quality of Service support in the HCA if > 0, (default 0)
block_loopback Block multicast loopback packets if > 0 (default: 1)
internal_err_reset: Reset device on internal errors if non-zero (default 1)
debug_level: Enable debug tracing if > 0 (default 0)
log_num_qp: log maximum number of QPs per HCA (default is 17; max is 20)
log_num_srq: log maximum number of SRQs per HCA (default is 16; max is 20)
log_rdmarc_per_qp: log number of RDMARC buffers per QP (default is 4; max is 7)
log_num_cq: log maximum number of CQs per HCA (default is 16 max is 19)
log_num_mcg: log maximum number of multicast groups per HCA (default is 13; max is 21)
log_num_mpt: log maximum number of memory protection table entries per HCA
(default is 17; max is 20)
log_num_mtt: log maximum number of memory translation table segments per HCA
(default is 20; max is 20)
log_num_mac: log maximum number of MACs per ETH port (1-7) (int)
log_num_vlan: log maximum number of VLANs per ETH port (0-7) (int)
log_mtts_per_seg Log2 number of MTT entries per segment (1-5; default is 3)
use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool)
mlx4_ib parameters:
debug_level: Enable debug tracing if > 0 (default 0)
mlx4_en parameters:
rss_xor: Use XOR hash function for RSS 0 (default is xor)
rss_mask: RSS hash type bitmask (default is 0xf)
num_lro: Number of LRO sessions per ring or disabled (0) (default is 32)
pfctx: Priority based Flow Control policy on TX[7:0].
Per priority bit mask (default is 0)
pfcrx: Priority based Flow Control policy on RX[7:0].
Per priority bit mask (default is 0)
inline_thold: threshold for using inline data (default is 128)
|
