1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
# Testing in rdma-core
rdma-core now offers an infrastructure for quick and easy additions of feature-
specific tests.
## Design
### Resources Management
`BaseResources` class is the basic objects aggregator available. It includes a
Context and a PD.
Inheriting from it is `TrafficResources` class, which also holds a MR, CQ and
QP, making it enough to support loopback traffic testing. It exposes methods for
creation of these objects which can be overridden by inheriting classes.
Inheriting from `TrafficResources` are currently three classes:
- `RCResources`
- `UDResources`
- `XRXResources`
The above subclasses add traffic-specific constants. For example, `UDResources`
overrides create_mr and adds the size of the GRH header to the message size.
`RCResources` exposes a wrapper to modify the QP to RTS.
### Tests-related Classes
`unittest.TestCase` is a logical test unit in Python's unittest module.
`RDMATestCase` inherits from it and adds the option to accept parameters
(example will follow below) or use a random set of valid parameters:
- If no device was provided, it iterates over the existing devices, for each
port of each device, it checks which GID indexes are valid (in RoCE, only
IPv4 and IPv6 based GIDs are used). Each <dev, port, gid> is added to an array
and one entry is selected.
- If a device was provided, the same process is done for all ports of this
device, and so on.
### Traffic Utilities
tests/utils.py offers a few wrappers for common traffic operations, making the
use of default values even shorter. Those traffic utilities accept an
aggregation object as their first parameter and rely on that object to have
valid RDMA resources for proper functioning.
- get_[send, recv]_wr() creates a [Send, Recv]WR object with a single SGE. It
also sets the MR content to be 'c's for client side or 's's for server side
(this is later validated).
- post_send() posts a single send request to the aggregation object's QP. If the
QP is a UD QP, an address vector will be added to the send WR.
- post_recv() posts the given RecvWR <num> times, so it can be used to fill the
RQ prior to traffic as well as during traffic.
- poll_cq() polls <num> completions from the CQ and raises an exception on a
non-success status.
- validate() verifies that the data in the MR is as expected ('c's for server,
's's for client).
- traffic() runs <num> iterations of send/recv between 2 players.
## How to run rdma-core's tests
#### Developers
The tests can be executed from ./build/bin:
```
./build.sh
./build/bin/run_tests.py
```
#### Users
The tests are not a Python package, as such they can be found under
/usr/share/doc/rdma-core-{version}/tests.
In order to run all tests:
```
python /usr/share/doc/rdma-core-<version>/tests/run_tests.py
```
#### Execution output
Output will be something like:
```
$ ./build/bin/run_tests.py
..........................................ss...............
----------------------------------------------------------------------
Ran 59 tests in 13.268s
OK (skipped=2)
```
A dot represents a passing test. 's' means a skipped test. 'E' means a test
that failed.
Tests can also be executed in verbose mode:
```
$ python3 /usr/share/doc/rdma-core-26.0/tests/run_tests.py -v
test_create_ah (test_addr.AHTest) ... ok
test_create_ah_roce (test_addr.AHTest) ... ok
test_destroy_ah (test_addr.AHTest) ... ok
test_create_comp_channel (test_cq.CCTest) ... ok
< many more lines here>
test_odp_rc_traffic (test_odp.OdpTestCase) ... skipped 'No port is up, can't run traffic'
test_odp_ud_traffic (test_odp.OdpTestCase) ... skipped 'No port is up, can't run traffic'
<more lines>
----------------------------------------------------------------------
Ran 59 tests in 12.857s
OK (skipped=2)
```
Verbose mode provides the reason for skipping the test (if one was provided by
the test developer).
### Customized Execution
tests/\_\_init\_\_.py defines a `_load_tests` function that returns an array with
the tests that will be executed.
The default implementation collects all test_* methods from all the classes that
inherit from `unittest.TestCase` (or `RDMATestCase`) and located in files under
tests directory which names starts with test_.
Users can execute part of the tests by adding `-k` to the run_tests.py command.
The following example executes only tests cases in files starting with
`test_device` and not `test_`.
```
$ build/bin/run_tests.py -v -k test_device
test_create_dm (tests.test_device.DMTest) ... ok
test_create_dm_bad_flow (tests.test_device.DMTest) ... ok
test_destroy_dm (tests.test_device.DMTest) ... ok
test_destroy_dm_bad_flow (tests.test_device.DMTest) ... ok
test_dm_read (tests.test_device.DMTest) ... ok
test_dm_write (tests.test_device.DMTest) ... ok
test_dm_write_bad_flow (tests.test_device.DMTest) ... ok
test_dev_list (tests.test_device.DeviceTest) ... ok
test_open_dev (tests.test_device.DeviceTest) ... ok
test_query_device (tests.test_device.DeviceTest) ... ok
test_query_device_ex (tests.test_device.DeviceTest) ... ok
test_query_gid (tests.test_device.DeviceTest) ... ok
test_query_port (tests.test_device.DeviceTest) ... ok
test_query_port_bad_flow (tests.test_device.DeviceTest) ... ok
----------------------------------------------------------------------
Ran 14 tests in 0.152s
OK
```
### GPUDirect RDMA
Part of the available tests includes GPUDirect RDMA.
Those tests run RDMA traffic over CUDA-allocated memory.
In order to run them successfully it's required to have a supported NVIDIA GPU,
CUDA 11.7 and above with "Open flavor" ("-m kernel-open") Driver 515 or later
and cuda-python 12.0 and above.
Running the tests is similar to other tests, with the option to choose which GPU
unit to use in case there are multiple GPUs on the setup.
```
$ build/bin/run_tests.py -v --gpu 0 -k cuda
test_cuda_dmabuf_rdma_write_traffic (tests.test_cuda_dmabuf.DmabufCudaTest)
Runs RDMA Write traffic over CUDA allocated memory using DMA BUF and ... ok
test_mlx_devx_cuda_send_imm_traffic (tests.test_mlx5_cuda_umem.Mlx5GpuDevxRcTrafficTest)
Creates two DevX RC QPs and runs SEND_IMM traffic over CUDA allocated ... ok
----------------------------------------------------------------------
Ran 2 tests in 3.033s
OK
```
## Writing Tests
The following section explains how to add a new test, using tests/test_odp.py
as an example. It's a simple test that runs ping-pong over a few different
traffic types.
ODP requires capability check, so a decorator was added to tests/utils.py.
The first change for ODP execution is when registering a memory region (need to
set the ON_DEMAND access flag), so we do as follows:
1. Create the players by inheriting from `RCResources` (for RC traffic).
2. In the player, override create_mr() and add the decorator to it. It will run
before the actual call to ibv_reg_mr and if ODP caps are off, the test will
be skipped.
3. Create the `OdpTestCase` by inheriting from `RDMATestCase`.
4. In the test case, add a method starting with test_, to let the unittest
infrastructure that this is a test.
5. In the test method, create the players (which already check the ODP caps)
and call the traffic() function, providing it the two players.
|
