File: drvnodedev.html.in

package info (click to toggle)
libvirt 5.6.0-2
  • links: PTS, VCS
  • area: main
  • in suites: bullseye
  • size: 240,844 kB
  • sloc: ansic: 584,521; xml: 176,725; sh: 9,912; python: 4,731; perl: 4,343; makefile: 3,321; ml: 465
file content (374 lines) | stat: -rw-r--r-- 14,021 bytes parent folder | download | duplicates (4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
  <body>
    <h1>Host device management</h1>

    <p>
      Libvirt provides management of both physical and virtual host devices
      (historically also referred to as node devices) like USB, PCI, SCSI, and
      network devices. This also includes various virtualization capabilities
      which the aforementioned devices provide for utilization, for example
      SR-IOV, NPIV, MDEV, DRM, etc.
    </p>

    <p>
      The node device driver provides means to list and show details about host
      devices (<code>virsh nodedev-list</code>,
      <code>virsh nodedev-dumpxml</code>), which are generic and can be used
      with all devices. It also provides means to create and destroy devices
      (<code>virsh nodedev-create</code>, <code>virsh nodedev-destroy</code>)
      which are meant to be used to create virtual devices, currently only
      supported by NPIV
      (<a href="http://wiki.libvirt.org/page/NPIV_in_libvirt">more info about NPIV)</a>).
      Devices on the host system are arranged in a tree-like hierarchy, with
      the root node being called <code>computer</code>. The node device driver
      supports two backends to manage the devices, HAL and udev, with the former
      being deprecated in favour of the latter.
    </p>

    <p>
      The generic format of a host device XML can be seen below.
      To identify a device both within the host and the device tree hierarchy,
      the following elements are used:
    </p>
      <dl>
        <dt><code>name</code></dt>
        <dd>
          The device's name will be generated by libvirt using the subsystem,
          like pci and the device's sysfs basename.
        </dd>
        <dt><code>path</code></dt>
        <dd>
          Fully qualified sysfs path to the device.
        </dd>
        <dt><code>parent</code></dt>
        <dd>
          This element identifies the parent node in the device hierarchy. The
          value of the element will correspond with the device parent's
          <code>name</code> element or <code>computer</code> if the device does
          not have any parent.
        </dd>
        <dt><code>driver</code></dt>
        <dd>
          This elements reports the driver in use for this device. The presence
          of this element in the output XML depends on whether the underlying
          device manager (most likely udev) exposes information about the
          driver.
        </dd>
        <dt><code>capability</code></dt>
        <dd>
          Describes the device in terms of feature support. The element has one
          mandatory attribute <code>type</code> the value of which determines
          the type of the device. Currently recognized values for the attribute
          are:
          <code>system</code>,
          <code>pci</code>,
          <code>usb</code>,
          <code>usb_device</code>,
          <code>net</code>,
          <code>scsi</code>,
          <code>scsi_host</code> (<span class="since">Since 0.4.7</span>),
          <code>fc_host</code>,
          <code>vports</code>,
          <code>scsi_target</code> (<span class="since">Since 0.7.3</span>),
          <code>storage</code> (<span class="since">Since 1.0.4</span>),
          <code>scsi_generic</code> (<span class="since">Since 1.0.7</span>),
          <code>drm</code> (<span class="since">Since 3.1.0</span>), and
          <code>mdev</code> (<span class="since">Since 3.4.0</span>).
          This element can be nested in which case it further specifies a
          device's capability. Refer to specific device types to see more values
          for the <code>type</code> attribute which are exclusive.
        </dd>
      </dl>

    <h2>Basic structure of a node device</h2>
    <pre>
&lt;device&gt;
  &lt;name&gt;pci_0000_00_17_0&lt;/name&gt;
  &lt;path&gt;/sys/devices/pci0000:00/0000:00:17.0&lt;/path&gt;
  &lt;parent&gt;computer&lt;/parent&gt;
  &lt;driver&gt;
    &lt;name&gt;ahci&lt;/name&gt;
  &lt;/driver&gt;
  &lt;capability type='pci'&gt;
...
  &lt;/capability&gt;
&lt;/device&gt;</pre>

    <ul id="toc"/>

    <h2><a id="PCI">PCI host devices</a></h2>
    <dl>
      <dt><code>capability</code></dt>
      <dd>
        When used as top level element, the supported values for the
        <code>type</code> attribute are <code>pci</code> and
        <code>phys_function</code> (see <a href="#SRIOVCap">SR-IOV below</a>).
      </dd>
    </dl>
    <pre>
&lt;device&gt;
  &lt;name&gt;pci_0000_04_00_1&lt;/name&gt;
  &lt;path&gt;/sys/devices/pci0000:00/0000:00:06.0/0000:04:00.1&lt;/path&gt;
  &lt;parent&gt;pci_0000_00_06_0&lt;/parent&gt;
  &lt;driver&gt;
    &lt;name&gt;igb&lt;/name&gt;
  &lt;/driver&gt;
  &lt;capability type='pci'&gt;
    &lt;domain&gt;0&lt;/domain&gt;
    &lt;bus&gt;4&lt;/bus&gt;
    &lt;slot&gt;0&lt;/slot&gt;
    &lt;function&gt;1&lt;/function&gt;
    &lt;product id='0x10c9'&gt;82576 Gigabit Network Connection&lt;/product&gt;
    &lt;vendor id='0x8086'&gt;Intel Corporation&lt;/vendor&gt;
    &lt;iommuGroup number='15'&gt;
      &lt;address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/&gt;
    &lt;/iommuGroup&gt;
    &lt;numa node='0'/&gt;
    &lt;pci-express&gt;
      &lt;link validity='cap' port='1' speed='2.5' width='2'/&gt;
      &lt;link validity='sta' speed='2.5' width='2'/&gt;
    &lt;/pci-express&gt;
  &lt;/capability&gt;
&lt;/device&gt;</pre>

    <p>
      The XML format for a PCI device stays the same for any further
      capabilities it supports, a single nested <code>&lt;capability&gt;</code>
      element will be included for each capability the device supports.
    </p>

    <h3><a id="SRIOVCap">SR-IOV capability</a></h3>
    <p>
      Single root input/output virtualization (SR-IOV) allows sharing of the
      PCIe resources by multiple virtual environments. That is achieved by
      slicing up a single full-featured physical resource called physical
      function (PF) into multiple devices called virtual functions (VFs) sharing
      their configuration with the underlying PF. Despite the SR-IOV
      specification, the amount of VFs that can be created on a PF varies among
      manufacturers.
    </p>

    <p>
      Suppose the NIC <a href="#PCI">above</a> was also SR-IOV capable, it would
      also include a nested
      <code>&lt;capability&gt;</code> element enumerating all virtual
      functions available on the physical device (physical port) like in the
      example below.
    </p>

    <pre>
&lt;capability type='pci'&gt;
...
  &lt;capability type='virt_functions' maxCount='7'&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x10' function='0x1'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x10' function='0x3'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x10' function='0x5'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x10' function='0x7'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x11' function='0x1'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x11' function='0x3'/&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x11' function='0x5'/&gt;
  &lt;/capability&gt;
...
&lt;/capability&gt;</pre>
    <p>
      A SR-IOV child device on the other hand, would then report its top level
      capability type as a <code>phys_function</code> instead:
    </p>

    <pre>
&lt;device&gt;
...
  &lt;capability type='phys_function'&gt;
    &lt;address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/&gt;
  &lt;/capability&gt;
...
&lt;device&gt;</pre>

    <h3><a id="MDEVCap">MDEV capability</a></h3>
    <p>
      A PCI device capable of creating mediated devices will include a nested
      capability <code>mdev_types</code> which enumerates all supported mdev
      types on the physical device, along with the type attributes available
      through sysfs:
    </p>

    <dl>
      <dt><code>type</code></dt>
      <dd>
        This element describes a mediated device type which acts as an
        abstract template defining a resource allocation for instances of this
        device type. The element has one attribute <code>id</code> which holds
        an official vendor-supplied identifier for the type.
        <span class="since">Since 3.4.0</span>
      </dd>

      <dt><code>name</code></dt>
      <dd>
        The <code>name</code> element holds a vendor-supplied code name for
        the given mediated device type. This is an optional element.
        <span class="since">Since 3.4.0</span>
      </dd>

      <dt><code>deviceAPI</code></dt>
      <dd>
        The value of this element describes how an instance of the given type
        will be presented to the guest by the VFIO framework.
        <span class="since">Since 3.4.0</span>
      </dd>

      <dt><code>availableInstances</code></dt>
      <dd>
        This element reports the current state of resource allocation. In other
        words, how many instances of the given type can still be successfully
        created on the physical device.
        <span class="since">Since 3.4.0</span>
      </dd>
    </dl>

    <p>
      For a more info about mediated devices, refer to the
      <a href="#MDEV">paragraph below</a>.
    </p>

<pre>
&lt;device&gt;
...
  &lt;driver&gt;
    &lt;name&gt;nvidia&lt;/name&gt;
  &lt;/driver&gt;
  &lt;capability type='pci'&gt;
...
    &lt;capability type='mdev_types'&gt;
      &lt;type id='nvidia-11'&gt;
        &lt;name&gt;GRID M60-0B&lt;/name&gt;
        &lt;deviceAPI&gt;vfio-pci&lt;/deviceAPI&gt;
        &lt;availableInstances&gt;16&lt;/availableInstances&gt;
      &lt;/type&gt;
      &lt;!-- Here would come the rest of the available mdev types --&gt;
    &lt;/capability&gt;
...
  &lt;/capability&gt;
&lt;/device&gt;</pre>

    <h2><a id="MDEV">Mediated devices (MDEVs)</a></h2>
    <p>
      Mediated devices (<span class="since">Since 3.2.0</span>) are software
      devices defining resource allocation on the backing physical device which
      in turn allows the parent physical device's resources to be divided into
      several mediated devices, thus sharing the physical device's performance
      among multiple guests. Unlike SR-IOV however, where a PCIe device appears
      as multiple separate PCIe devices on the host's PCI bus, mediated devices
      only appear on the mdev virtual bus. Therefore, no detach/reattach
      procedure from/to the host driver procedure is involved even though
      mediated devices are used in a direct device assignment manner.
    </p>

    <p>
      The following sub-elements and attributes are exposed within the
      <code>capability</code> element:
    </p>

    <dl>
      <dt><code>type</code></dt>
      <dd>
        This element describes a mediated device type which acts as an
        abstract template defining a resource allocation for instances of this
        device type. The element has one attribute <code>id</code> which holds
        an official vendor-supplied identifier for the type.
        <span class="since">Since 3.4.0</span>
      </dd>

      <dt><code>iommuGroup</code></dt>
      <dd>
        This element supports a single attribute <code>number</code> which holds
        the IOMMU group number the mediated device belongs to.
        <span class="since">Since 3.4.0</span>
      </dd>
    </dl>

    <h3>Example of a mediated device</h3>
    <pre>
&lt;device&gt;
  &lt;name&gt;mdev_4b20d080_1b54_4048_85b3_a6a62d165c01&lt;/name&gt;
  &lt;path&gt;/sys/devices/pci0000:00/0000:00:02.0/4b20d080-1b54-4048-85b3-a6a62d165c01&lt;/path&gt;
  &lt;parent&gt;pci_0000_06_00_0&lt;/parent&gt;
  &lt;driver&gt;
    &lt;name&gt;vfio_mdev&lt;/name&gt;
  &lt;/driver&gt;
  &lt;capability type='mdev'&gt;
    &lt;type id='nvidia-11'/&gt;
    &lt;iommuGroup number='12'/&gt;
  &lt;capability/&gt;
&lt;device/&gt;</pre>

    <p>
      The support of mediated device's framework in libvirt's node device driver
      covers the following features:
    </p>

    <ul>
      <li>
        list available mediated devices on the host
        (<span class="since">Since 3.4.0</span>)
      </li>
      <li>
        display device details
        (<span class="since">Since 3.4.0</span>)
      </li>
    </ul>

    <p>
      Because mediated devices are instantiated from vendor specific templates,
      simply called 'types', information describing these types is contained
      within the parent device's capabilities
      (see the example in <a href="#PCI">PCI host devices</a>).
    </p>

    <p>
      To see the supported mediated device types on a specific physical device
      use the following:
    </p>

    <pre>
$ ls /sys/class/mdev_bus/&lt;device&gt;/mdev_supported_types</pre>

    <p>
      Before creating a mediated device, unbind the device from the respective
      device driver, eg. subchannel I/O driver for a CCW device. Then bind the
      device to the respective VFIO driver. For a CCW device, also unbind the
      corresponding subchannel of the CCW device from the subchannel I/O driver
      and then bind the subchannel (instead of the CCW device) to the vfio_ccw
      driver. The below example shows the unbinding and binding steps for a CCW
      device.
    </p>

    <pre>
device="0.0.1234"
subchannel="0.0.0123"
echo $device &gt; /sys/bus/ccw/devices/$device/driver/unbind
echo $subchannel &gt; /sys/bus/css/devices/$subchannel/driver/unbind
echo $subchannel &gt; /sys/bus/css/drivers/vfio_ccw/bind
    </pre>

    <p>
      To manually instantiate a mediated device, use one of the following as a
      reference. For a CCW device, use the subchannel ID instead of the device
      ID.
    </p>

    <pre>
$ uuidgen &gt; /sys/class/mdev_bus/&lt;device&gt;/mdev_supported_types/&lt;type&gt;/create
...
$ echo &lt;UUID&gt; &gt; /sys/class/mdev_bus/&lt;device&gt;/mdev_supported_types/&lt;type&gt;/create</pre>

    <p>
      Manual removal of a mediated device is then performed as follows:
    </p>

    <pre>
$ echo 1 &gt; /sys/bus/mdev/devices/&lt;uuid&gt;/remove</pre>

  </body>
</html>