1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
Open Fabrics Enterprise Distribution (OFED)
SDP in OFED 1.4.1 Release Notes
May 2009
===============================================================================
Table of Contents
===============================================================================
1. Overview
2. Bug Fixes and Enhancements since OFED 1.3
3. Bug Fixes and Enhancements since OFED 1.4
4. Known Issues
5. Verification Applications/Flows/Tests
===============================================================================
1. Overview
===============================================================================
SDP in OFED is at GA level for OFED 1.4.1
===============================================================================
2. Bug Fixes and Enhancements since OFED 1.3
===============================================================================
* Cleanup
- Compilation warnings
- New kernel support
* New function
- sdpnetstat supply information about socket state and process name.
* Bug fixes
- open/close connection mechanism overhaul - many issue related to that are
fixed.
- No known kernel crash is caused by SDP.
- Many small bugs fixed - see bugzilla.
- Having now full windows interoperability.
===============================================================================
2. Bug Fixes and Enhancements since OFED 1.4
===============================================================================
SDP:
- BUG1311 Netpipe fails with a IB_WC_LOC_LEN_ERR.
- BUG1472 - clean socket timeouts and refcount when device is removed
- BUG1502 - scheduling while atomic
- BUG1309 - SDP close is slow + fix recv buffer initial size setting
- BUG1087 - fixed recovery from failing rdma_create_qp()
===============================================================================
3. Known Issues
===============================================================================
- BUG1444 - setsockopt(SO_RCVBUF) is not working in sdp socket. To limit top
system wide sdp memory usage for recv use module parameter top_mem_usage.
- There are some issues regarding PPC and IA64 that were not fixed for this
release. check bugzilla for more info.
- TCP allows connecting to IP_ANY - 0.0.0.0 (as a destination address!). SDP
does not allow - and will reject the connection.
- Each SDP socket currently consumes up to 2 MBytes of memory. If this value
is high for your installation, it is possible to trade off performance
for lower memory utilization per socket by reducing the value of the
"rcvbuf_scale" module parameter (default: 16).
Note: the minimum legal value for this parameter is 1.
At this parameter value, each socket will consume approximately 128 KBytes.
- Small message size performance is low when messages are sent by client
at a rate lower than the rate at which they are consumed by server,
and when TCP_CORK is not set. This is observed, for example, with iperf
benchmark. As a workaround, set the TCP_CORK socket option
to ensure data is sent in at least 32K byte chunks.
- Performance is low on 32-bit kernels, as SDP utilizes high memory
to ease memory pressure. Moving to a 64-bit kernel solves this
problem even if the application remains a 32-bit one.
- By default, SDP utilizes a 2 Kbyte MTU size. This may cause PCI-X cards
using Mellanox Technologies "Infinihost" HCAs to experience low bandwidth.
Workaround: reset the MTU size to 1K in this situation, using either of
the two methods below:
1. Activate the "tavor quirk" workaround in opensm:
a. Create an opensm options cache file (/var/cache/osm/opensm.opts):
> opensm --cache-options -o
b. Add the following line to /var/cache/osm/opensm.opts:
enable_quirks TRUE
c. Rerun opensm using your usual command line options to activate
the opensm quirk option.
2. Activate the "tavor quirk" workaround in cma:
set the tavor_quirk module parameter of the rdma_cm module to value 1
(default: 0).
- BZCOPY mode is only effective for large block transfers.
By setting the /sys parameter 'sdp_zcopy_thresh' to a non-zero value, a
non-standard SDP speedup is enabled. All messages longer than
'sdp_zcopy_thresh' bytes in length will cause the user space buffer to
be pinned and the data sent directly from the original buffer. This
results in less CPU use and, on many systems, much better bandwidth.
The default 64K value for 'sdp_zcopy_thresh' is sometimes too low for
some systems. You must experiment with your hardware to select the
best value. when mixing bzcopy and bcopy on the same socket the socket
could get stucked in rare situations - BUG1324.
===============================================================================
4. Verification Applications/Flows/Tests
===============================================================================
- ssh/sshd
- wget/netscape/firefox/apache
- netpipe
- netperf
- LTP socket tests
- iperf-2.0.2
- ttcp
- Threaded and forking echo client server examples
- Various Java client server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj)
- Many UNIX utilities to verify that pre-load did not harm the applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Open Fabrics Enterprise Distribution (OFED)
libsdp v. 9382 in OFED 1.4.1 Release Notes
May 2009
===============================================================================
Table of Contents
===============================================================================
1. Overview
2. New Features
3. Bug Fixes
4. Bug Fixes and Enhancements since OFED 1.4
5. Known Issues
6. Verification Applications/Flows/Tests
===============================================================================
1. Overview
===============================================================================
This document describes the contents of the libsdp OFED 1.3 release.
libsdp is a LD_PRELOAD-able library that can be used to migrate existing
applications to use InfiniBand Sockets Direct Protocol (SDP) instead of
TCP sockets, transparently and without recompilations. To setup libsdp
please follow the instructions below. The libsdp version for this release
is 1.3.
===============================================================================
2. New Features
===============================================================================
* Add support for new kernel options
- SIOCOUTQ ioctl support
- Add keepalive support
- New options: SOCK_KEEPALIVE, TCP_KEEPIDLE
* Add libsdp-devel sub-package
===============================================================================
3. Bug Fixes
===============================================================================
The following list of bugs were fixed. Note that other less critical
or visible bugs were also fixed.
* Multi-threaded applications that are opening and closing many SDP sockets
quickly using the 'both' flow would incorrectly close a valid socket.
* Attempt to bind to an invalid socket using 'both' flow returns the wrong
errno.
* Applications using signal driven IO (FASYNC) on 'both' flow sockets would
fail because the second accep() would be executed on the TCP socket.
* Update error handling to set errno to a valid error code prior to
returning -1.
===============================================================================
4. Bug Fixes and Enhancements since OFED 1.4
===============================================================================
libsdp:
* Enable building libsdp on Solaris
* BUG1256 - Add epoll support
sdpnetstat:
* BUF1513 - sdpnetstat is not showing all the listening processes on ipv6 sockets.
===============================================================================
5. Known Issues
===============================================================================
* libsdp cannot provide its socket switch functionality for executables
statically linked with libc.
* When using server to listen on both SDP and TCP, the number of sockets is
doubled.
===============================================================================
6. Verification Applications/Flows/Tests
===============================================================================
See the corresponding section in the SDP release notes above.
|
