1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.8"/>
<title>Hardware Locality (hwloc): I/O Devices</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td style="padding-left: 0.5em;">
<div id="projectname">Hardware Locality (hwloc)
 <span id="projectnumber">1.10.0</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.8 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="annotated.html"><span>Data Structures</span></a></li>
</ul>
</div>
</div><!-- top -->
<div class="header">
<div class="headertitle">
<div class="title">I/O Devices </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>hwloc usually manipulates processing units and memory but it can also discover I/O devices and report their locality as well. This is useful for placing I/O intensive applications on cores near the I/O devices they use.</p>
<h1><a class="anchor" id="iodevices_enabling"></a>
Enabling and requirements</h1>
<p>I/O discovery is disabled by default (except in lstopo) so as not to break legacy application by adding unexpected I/O objects to the topology. It can be enabled by passing flags such as <code><a class="el" href="a00077.html#ggada025d3ec20b4b420f8038d23d6e7bdea46ae25e8896278840b1800ae9ce4de41" title="Detect PCI devices. ">HWLOC_TOPOLOGY_FLAG_IO_DEVICES</a></code> to <a class="el" href="a00077.html#gaaeed4df656979e5f16befea9d29b814b" title="Set OR'ed flags to non-yet-loaded topology. ">hwloc_topology_set_flags()</a> before loading the topology.</p>
<p>Note that I/O discovery requires significant help from the operating system. The pciaccess library (the development package is usually <code>libpciaccess-devel</code> or <code>libpciaccess-dev</code>) is needed to fully detect PCI devices and bridges, and the actual locality of these devices is only currently detected on Linux. Also, some operating systems require privileges for probing PCI devices, see <a class="el" href="a00028.html#faq_privileged">Does hwloc require privileged access?</a> for details.</p>
<p>On Linux, PCI discovery may still be performed even if <code>libpciaccess</code> cannot be used. But it misses PCI device names.</p>
<h1><a class="anchor" id="iodevices_hierarchy"></a>
I/O object hierarchy</h1>
<p>When I/O discovery is enabled and supported, some additional objects (types <code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a6825f10895fea60aca7a6ba9fe273db0" title="Bridge. Any bridge that connects the host or an I/O bus, to another I/O bus. Bridge objects have neit...">HWLOC_OBJ_BRIDGE</a></code>, <code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a5d8117a54df1fbd3606ab19e42cb0ea9" title="PCI device. These objects have neither CPU sets nor node sets. They are not added to the topology unl...">HWLOC_OBJ_PCI_DEVICE</a></code> and <code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a51e7280240fd9f25589cbbe538bdb075" title="Operating system device. These objects have neither CPU sets nor node sets. They are not added to the...">HWLOC_OBJ_OS_DEVICE</a></code>) are added to the topology as a child of the object they are close to. For instance, if a I/O Hub is connected to a socket, the corresponding hwloc bridge object (and its PCI bridges and devices children) is inserted as a child of the corresponding hwloc socket object.</p>
<p>These new objects have neither CPU sets nor node sets (NULL pointers) because they are not directly usable by the user applications. Moreover I/O hierarchies may be highly complex (asymmetric trees of bridges). So I/O objects are placed in specific levels with custom depths. Their lists may still be traversed with regular helpers such as <a class="el" href="a00078.html#ga759e88eaf5a230ad283e9d4c42486735" title="Returns the next object of type type. ">hwloc_get_next_obj_by_type()</a>. However, hwloc offers some dedicated helpers such as <a class="el" href="a00095.html#ga66470dabce9db19a57c5940a909d0baa" title="Get the next PCI device in the system. ">hwloc_get_next_pcidev()</a> and <a class="el" href="a00095.html#ga8b4584c8949e2c5f1c97ba7fe92b8145" title="Get the next OS device in the system. ">hwloc_get_next_osdev()</a> for convenience (see <a class="el" href="a00095.html">Finding I/O objects</a>).</p>
<p>An I/O hierarchy is organized as follows: A hostbridge object ( <code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a6825f10895fea60aca7a6ba9fe273db0" title="Bridge. Any bridge that connects the host or an I/O bus, to another I/O bus. Bridge objects have neit...">HWLOC_OBJ_BRIDGE</a></code> object with upstream type <em>Host</em> and downstream type <em>PCI</em>) is attached below a regular object (usually the entire machine or a NUMA node). There may be multiple hostbridges in the machine, attached to different places, but all I/O devices are below one of them. Each hostbridge contains one or several children, either other bridges (usually PCI to PCI) or PCI devices (<code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a5d8117a54df1fbd3606ab19e42cb0ea9" title="PCI device. These objects have neither CPU sets nor node sets. They are not added to the topology unl...">HWLOC_OBJ_PCI_DEVICE</a></code>). The number of bridges between the hostbridge and a PCI device depends on the machine and on the topology flags.</p>
<h1><a class="anchor" id="iodevices_osdev"></a>
Software devices</h1>
<p>Although each PCI device is uniquely identified by its bus ID (e.g. 0000:01:02.3), the application can hardly find out which PCI device is actually used when manipulating software handle (such as the <em>eth0</em> network interface, the <em>sda</em> hard drive, or the <em>mlx4_0</em> OpenFabrics HCA). Therefore hwloc tries to add software devices (<code><a class="el" href="a00074.html#ggacd37bb612667dc437d66bfb175a8dc55a51e7280240fd9f25589cbbe538bdb075" title="Operating system device. These objects have neither CPU sets nor node sets. They are not added to the...">HWLOC_OBJ_OS_DEVICE</a></code>, also known as OS devices) below their PCI objects.</p>
<p>hwloc first tries to discover the corresponding names, e.g. <em>eth0</em>, <em>sda</em> or <em>mlx4_0</em>, from the operating system. However, this ability is currently only available on Linux for some classes of devices.</p>
<p>hwloc then tries to discover software devices through additional I/O components using external libraries. For instance proprietary graphics drivers do not offer any OS name, but hwloc may still create one OS object per software handle when supported. For instance the <code>opencl</code> and <code>cuda</code> components may add some <em>opencl0d0</em> and <em>cuda0</em> OS device objects.</p>
<p>Here is a list of OS device objects commonly created by hwloc components when I/O discovery is enabled and supported.</p>
<ul>
<li>
Hard disks (HWLOC_OBJ_OSDEV_BLOCK) <ul>
<li>
<em>sda</em> (Linux component) </li>
</ul>
</li>
<li>
Network interfaces (HWLOC_OBJ_OSDEV_NETWORK) <ul>
<li>
<em>eth0</em>, <em>wlan0</em>, <em>ib0</em> (Linux component) </li>
</ul>
</li>
<li>
OpenFabrics HCAs (HWLOC_OBJ_OSDEV_OPENFABRICS) <ul>
<li>
<em>mlx4_0</em>, <em>qib0</em> (Linux component) </li>
</ul>
</li>
<li>
GPUs (HWLOC_OBJ_OSDEV_GPU) <ul>
<li>
<em>nvml0</em> for the first NVML device (NVML component, using the NVIDIA Management Library) </li>
<li>
<em>:0.0</em> for the first display (GL component, using the NV-CONTROL X extension library, NVCtrl) </li>
</ul>
</li>
<li>
Co-Processors (HWLOC_OBJ_OSDEV_COPROC) <ul>
<li>
<em>opencl0d0</em> for the first device of the first OpenCL platform, <em>opencl1d3</em> for the fourth device of the second OpenCL platform (OpenCL component) </li>
<li>
<em>cuda0</em> for the first NVIDIA CUDA device (CUDA component, using the NVIDIA CUDA Library) </li>
<li>
<em>mic0</em> for the first Intel Xeon Phi (MIC) coprocessor (Linux component) </li>
</ul>
</li>
<li>
DMA engine channel (HWLOC_OBJ_OSDEV_DMA) <ul>
<li>
<em>dma0chan0</em> (Linux component) </li>
</ul>
</li>
</ul>
<p>When none of the above strategies is supported and enabled, hwloc cannot place any OS object inside PCI objects. Note that some PCI devices may contain multiple software devices (see the example below).</p>
<p>See also <a class="el" href="a00020.html">Interoperability With Other Software</a> for managing these devices without considering them as hwloc objects.</p>
<h1><a class="anchor" id="iodevices_consult"></a>
Consulting I/O devices and binding</h1>
<p>I/O devices may be consulted by traversing the topology manually (with usual routines such as <a class="el" href="a00078.html#ga6f414dd80a2b943967a0ac92da3181a2" title="Returns the topology object at logical index idx with type type. ">hwloc_get_obj_by_type()</a>) or by using dedicated helpers (such as <a class="el" href="a00095.html#gacdbaf0db98872e224b7883a84bfb0455" title="Find the PCI device object matching the PCI bus id given domain, bus device and function PCI bus id...">hwloc_get_pcidev_by_busid()</a>, see <a class="el" href="a00095.html">Finding I/O objects</a>).</p>
<p>I/O objects do not actually contain any locality information because their CPU sets and node sets are NULL. Their locality must be retrieved by walking up the object tree (through the <code>parent</code> link) until an non-I/O object is found (see <a class="el" href="a00095.html#gaf139bb61375178e90cc3f1835b452ab6" title="Get the first non-I/O ancestor object. ">hwloc_get_non_io_ancestor_obj()</a>). This regular object should have non-NULL CPU sets and node sets which describe the processing units and memory that are immediately close to the I/O device. For instance the path from a OS device to its locality may go across a PCI device parent, one or several bridges, up to a a NUMA node with the same locality.</p>
<p>Command-line tools are also aware of I/O devices. lstopo displays the interesting ones by default (passing <code>–no-io</code> disables it).</p>
<p>hwloc-calc and hwloc-bind may manipulate I/O devices specified by PCI bus ID or by OS device name. </p><ul>
<li>
<code>pci=0000:02:03.0</code> is replaced by the set of CPUs that are close to the PCI device whose bus ID is given. </li>
<li>
<code>os=eth0</code> is replaced by CPUs that are close to the I/O device whose software handle is called <code>eth0</code>. </li>
</ul>
<p>This enables easy binding of I/O-intensive applications near the device they use.</p>
<h1><a class="anchor" id="iodevices_examples"></a>
Examples</h1>
<p>The following picture shows a dual-socket dual-core host whose PCI bus is connected to the first socket and NUMA node.</p>
<div class="image">
<img src="devel09-pci.png" alt="devel09-pci.png"/>
</div>
<p>Six interesting PCI devices were discovered. However hwloc found some corresponding software devices (<em>eth0</em>, <em>eth1</em>, <em>sda</em>, <em>mlx4_0</em>, <em>ib0</em>, and <em>ib1</em>) for only four of these physical devices. The other ones (<em>PCI 102b:0532</em> and <em>PCI 8086:3a20</em>) are an unused IDE controller (no disk attached) and a graphic card (no corresponding software device reported to the user by the operating system).</p>
<p>On the contrary, it should be noted three different software devices were found for the last PCI device (<em>PCI 15b3:634a</em>). Indeed this OpenFabrics HCA PCI device object contains one one OpenFabrics software device (<em>mlx4_0</em>) and two virtual network interface software devices (<em>ib0</em> and <em>ib1</em>).</p>
<p>PCI link speed is also reported for some bridges and devices because lstopo was privileged when it discovered the topology.</p>
<p>Here is the corresponding textual output:</p>
<pre class="fragment">Machine (24GB)
NUMANode L#0 (P#0 12GB)
Socket L#0 + L3 L#0 (8192KB)
L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#2)
HostBridge
PCIBridge
PCI 14e4:163b
Net "eth0"
PCI 14e4:163b
Net "eth1"
PCIBridge
PCI 1000:0060
Block "sda"
PCIBridge
PCI 102b:0532
PCI 8086:3a20
PCI 15b3:634a
Net "ib0"
Net "ib1"
Net "mlx4_0"
NUMANode L#1 (P#1 12GB) + Socket L#1 + L3 L#1 (8192KB)
L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#1)
L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3)
</pre> </div></div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Tue Oct 7 2014 11:56:52 for Hardware Locality (hwloc) by  <a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.8.8
</small></address>
</body>
</html>
|
