1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
|
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>11.2.6. CUDA — Open MPI 5.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
<script src="../../_static/jquery.js"></script>
<script src="../../_static/underscore.js"></script>
<script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../../_static/doctools.js"></script>
<script src="../../_static/sphinx_highlight.js"></script>
<script src="../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
<link rel="next" title="11.2.7. ROCm" href="rocm.html" />
<link rel="prev" title="11.2.5. iWARP Support" href="iwarp.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../index.html" class="icon icon-home">
Open MPI
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../getting-help.html">2. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../release-notes/index.html">3. Release notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../installing-open-mpi/index.html">4. Building and installing Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../features/index.html">5. Open MPI-specific features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../validate.html">6. Validating your installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../version-numbering.html">7. Version numbers and compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mca.html">8. The Modular Component Architecture (MCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../building-apps/index.html">9. Building MPI applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../launching-apps/index.html">10. Launching MPI applications</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../index.html">11. Run-time operation and tuning MPI applications</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../environment-var.html">11.1. Environment variables set for MPI applications</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="index.html">11.2. Networking support</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="ofi.html">11.2.1. OpenFabrics Interfaces (OFI) / Libfabric support</a></li>
<li class="toctree-l3"><a class="reference internal" href="tcp.html">11.2.2. TCP</a></li>
<li class="toctree-l3"><a class="reference internal" href="shared-memory.html">11.2.3. Shared Memory</a></li>
<li class="toctree-l3"><a class="reference internal" href="ib-and-roce.html">11.2.4. InifiniBand / RoCE support</a></li>
<li class="toctree-l3"><a class="reference internal" href="iwarp.html">11.2.5. iWARP Support</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">11.2.6. CUDA</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-build-open-mpi-with-cuda-aware-support">11.2.6.1. How do I build Open MPI with CUDA-aware support?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-verify-that-open-mpi-has-been-built-with-cuda-support">11.2.6.2. How do I verify that Open MPI has been built with CUDA support?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-run-open-mpi-with-applications-that-pass-cuda-buffers-to-mpi">11.2.6.3. How do I run Open MPI with applications that pass CUDA buffers to MPI?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-build-open-mpi-with-cuda-aware-support-using-pgi">11.2.6.4. How do I build Open MPI with CUDA-aware support using PGI?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#what-kind-of-cuda-support-exists-in-open-mpi">11.2.6.5. What kind of CUDA support exists in Open MPI?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#psm2-support-for-cuda">11.2.6.6. PSM2 support for CUDA</a></li>
<li class="toctree-l4"><a class="reference internal" href="#ofi-support-for-cuda">11.2.6.7. OFI support for CUDA</a></li>
<li class="toctree-l4"><a class="reference internal" href="#can-i-get-additional-cuda-debug-level-information-at-run-time">11.2.6.8. Can I get additional CUDA debug-level information at run-time?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#numa-node-issues">11.2.6.9. NUMA Node Issues</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-develop-cuda-aware-open-mpi-applications">11.2.6.10. How do I develop CUDA-aware Open MPI applications?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#which-mpi-apis-work-with-cuda-aware">11.2.6.11. Which MPI APIs work with CUDA-aware?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#which-mpi-apis-do-not-work-with-cuda-aware">11.2.6.12. Which MPI APIs do NOT work with CUDA-aware?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-use-cuda-aware-ucx-for-open-mpi">11.2.6.13. How do I use CUDA-aware UCX for Open MPI?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#which-mpi-apis-work-with-cuda-aware-ucx">11.2.6.14. Which MPI APIs work with CUDA-aware UCX?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#which-mpi-apis-do-not-work-with-cuda-aware-ucx">11.2.6.15. Which MPI APIs do NOT work with CUDA-aware UCX?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#can-i-tell-at-compile-time-or-runtime-whether-i-have-cuda-aware-support">11.2.6.16. Can I tell at compile time or runtime whether I have CUDA-aware support?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-limit-how-much-cuda-ipc-memory-is-held-in-the-registration-cache">11.2.6.17. How do I limit how much CUDA IPC memory is held in the registration cache?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#what-are-some-guidelines-for-using-cuda-and-open-mpi-with-omni-path">11.2.6.18. What are some guidelines for using CUDA and Open MPI with Omni-Path?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#when-do-i-need-to-select-a-cuda-device">11.2.6.19. When do I need to select a CUDA device?</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-do-i-enable-cuda-support-in-hcoll-collective-component">11.2.6.20. How do I enable CUDA support in HCOLL collective component</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="rocm.html">11.2.7. ROCm</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../multithreaded.html">11.3. Running multi-threaded MPI applications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dynamic-loading.html">11.4. Dynamically loading <code class="docutils literal notranslate"><span class="pre">libmpi</span></code> at runtime</a></li>
<li class="toctree-l2"><a class="reference internal" href="../fork-system-popen.html">11.5. Calling fork(), system(), or popen() in MPI processes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../fault-tolerance/index.html">11.6. Fault tolerance</a></li>
<li class="toctree-l2"><a class="reference internal" href="../large-clusters/index.html">11.7. Large Clusters</a></li>
<li class="toctree-l2"><a class="reference internal" href="../affinity.html">11.8. Processor and memory affinity</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mpi-io/index.html">11.9. MPI-IO tuning options</a></li>
<li class="toctree-l2"><a class="reference internal" href="../coll-tuned.html">11.10. Tuning Collectives</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking.html">11.11. Benchmarking Open MPI applications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../heterogeneity.html">11.12. Building heterogeneous MPI applications</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../app-debug/index.html">12. Debugging Open MPI Parallel Applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../developers/index.html">13. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributing.html">14. Contributing to Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../license/index.html">15. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../history.html">16. History of Open MPI</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../man-openmpi/index.html">17. Open MPI manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../man-openshmem/index.html">18. OpenSHMEM manual pages</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../index.html">Open MPI</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../index.html"><span class="section-number">11. </span>Run-time operation and tuning MPI applications</a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">11.2. </span>Networking support</a></li>
<li class="breadcrumb-item active"><span class="section-number">11.2.6. </span>CUDA</li>
<li class="wy-breadcrumbs-aside">
<a href="../../_sources/tuning-apps/networking/cuda.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="cuda">
<h1><span class="section-number">11.2.6. </span>CUDA<a class="headerlink" href="#cuda" title="Permalink to this heading"></a></h1>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>TODO This section needs to be converted from FAQ Q&A style
to regular documentation style.</p>
</div>
<div class="section" id="how-do-i-build-open-mpi-with-cuda-aware-support">
<h2><span class="section-number">11.2.6.1. </span>How do I build Open MPI with CUDA-aware support?<a class="headerlink" href="#how-do-i-build-open-mpi-with-cuda-aware-support" title="Permalink to this heading"></a></h2>
<p>CUDA-aware support means that the MPI library can send and receive GPU
buffers directly. CUDA support is being continuously updated so
different levels of support exist in different versions. We recommend
you use the latest version of Open MPI for best support.</p>
<p>Open MPI offers two flavors of CUDA support:</p>
<ol class="arabic">
<li><p>Via <a class="reference external" href="https://openucx.org/">UCX</a>.</p>
<p>This is the preferred mechanism. Since UCX will be providing the
CUDA support, it is important to ensure that UCX itself is built
with CUDA support.</p>
<p>To see if your ucx was built with CUDA support run the following
command:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Check if ucx was built with CUDA support</span>
shell$<span class="w"> </span>ucx_info<span class="w"> </span>-v
<span class="c1"># configured with: --build=powerpc64le-redhat-linux-gnu --host=powerpc64le-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --enable-mt --disable-params-check --enable-cma --without-cuda --without-gdrcopy --with-verbs --with-cm --with-knem --with-rdmacm --without-rocm --without-xpmem --without-ugni --without-java</span>
</pre></div>
</div>
<p>If you need to build ucx yourself to include CUDA support, please
see the UCX documentation for <a class="reference external" href="https://openucx.readthedocs.io/en/master/running.html#openmpi-with-ucx">building ucx with Open MPI:</a></p>
<p>It should look something like:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Configure UCX this way</span>
shell$<span class="w"> </span>./configure<span class="w"> </span>--prefix<span class="o">=</span>/path/to/ucx-cuda-install<span class="w"> </span>--with-cuda<span class="o">=</span>/usr/local/cuda<span class="w"> </span>--with-gdrcopy<span class="o">=</span>/usr
<span class="c1"># Configure Open MPI this way</span>
shell$<span class="w"> </span>./configure<span class="w"> </span>--with-cuda<span class="o">=</span>/usr/local/cuda<span class="w"> </span>--with-ucx<span class="o">=</span>/path/to/ucx-cuda-install<span class="w"> </span><other<span class="w"> </span>configure<span class="w"> </span>params>
</pre></div>
</div>
</li>
<li><p>Via internal Open MPI CUDA support</p></li>
</ol>
<p>Regardless of which flavor of CUDA support (or both) you plan to use,
Open MPI should be configured using the <code class="docutils literal notranslate"><span class="pre">--with-cuda=<path-to-cuda></span></code>
configure option to build CUDA support into Open MPI. The configure
script will automatically search the path given for <code class="docutils literal notranslate"><span class="pre">libcuda.so</span></code>. If it cannot
be found, please also pass <code class="docutils literal notranslate"><span class="pre">--with-cuda-libdir</span></code>. For example:
<code class="docutils literal notranslate"><span class="pre">--with-cuda=<path-to-cuda></span> <span class="pre">--with-cuda-libdir=/usr/local/cuda/lib64/stubs</span></code>.</p>
<p>Open MPI supports building with CUDA libraries and running on systems
without CUDA libraries or hardware.</p>
<p>For releases v5.0.2 and newer no special steps are required to get this behavior.</p>
<p>In order to realize this behavior for the v5.0.0 and v5.0.1 releases,
when configuring Open MPI, you have to specify the CUDA dependent components to be built as DSOs using the
<code class="docutils literal notranslate"><span class="pre">--enable-mca-dso=<comma-delimited-list-of-cuda-components.</span></code>
configure option.</p>
<p>This affects the <code class="docutils literal notranslate"><span class="pre">smcuda</span></code> shared memory and <code class="docutils literal notranslate"><span class="pre">uct</span></code> BTLs, as well
as the <code class="docutils literal notranslate"><span class="pre">rgpusm</span></code> and <code class="docutils literal notranslate"><span class="pre">gpusm</span></code> rcache components.</p>
<p>An example configure command would look like the following:</p>
<blockquote>
<div><div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Configure Open MPI this way</span>
shell$<span class="w"> </span>./configure<span class="w"> </span>--with-cuda<span class="o">=</span>/usr/local/cuda<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--enable-mca-dso<span class="o">=</span>btl-smcuda,rcache-rgpusm,rcache-gpusm,accelerator-cuda<span class="w"> </span><other<span class="w"> </span>configure<span class="w"> </span>params>
</pre></div>
</div>
</div></blockquote>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-verify-that-open-mpi-has-been-built-with-cuda-support">
<h2><span class="section-number">11.2.6.2. </span>How do I verify that Open MPI has been built with CUDA support?<a class="headerlink" href="#how-do-i-verify-that-open-mpi-has-been-built-with-cuda-support" title="Permalink to this heading"></a></h2>
<p>Verify that Open MPI has been built with cuda using <code class="docutils literal notranslate"><span class="pre">ompi_info</span></code>
with one of the following commands.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Use ompi_info to verify cuda support in Open MPI</span>
shell$<span class="w"> </span>ompi_info<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"MPI extensions"</span>
<span class="w"> </span>MPI<span class="w"> </span>extensions:<span class="w"> </span>affinity,<span class="w"> </span>cuda,<span class="w"> </span>pcollreq
shell$<span class="w"> </span>ompi_info<span class="w"> </span>--parsable<span class="w"> </span>--all<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>mpi_built_with_cuda_support:value
<span class="w"> </span>mca:mpi:base:param:mpi_built_with_cuda_support:value:true
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-run-open-mpi-with-applications-that-pass-cuda-buffers-to-mpi">
<h2><span class="section-number">11.2.6.3. </span>How do I run Open MPI with applications that pass CUDA buffers to MPI?<a class="headerlink" href="#how-do-i-run-open-mpi-with-applications-that-pass-cuda-buffers-to-mpi" title="Permalink to this heading"></a></h2>
<p>Open MPI will detect and enable CUDA enabled components at runtime with
no additional mpirun parameters.</p>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-build-open-mpi-with-cuda-aware-support-using-pgi">
<h2><span class="section-number">11.2.6.4. </span>How do I build Open MPI with CUDA-aware support using PGI?<a class="headerlink" href="#how-do-i-build-open-mpi-with-cuda-aware-support-using-pgi" title="Permalink to this heading"></a></h2>
<p>With CUDA 6.5, you can build all versions of CUDA-aware Open MPI
without doing anything special. However, with CUDA 7.0 and CUDA 7.5,
you need to pass in some specific compiler flags for things to work
correctly. Add the following to your configure line.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># For PGI 15.9 and later (Also called NVCC):</span>
shell$<span class="w"> </span>./configure<span class="w"> </span>--with-wrapper-cflags<span class="o">=</span>-ta:tesla
<span class="c1"># For earlier versions of PGI:</span>
shell$<span class="w"> </span>./configure<span class="w"> </span><span class="nv">CFLAGS</span><span class="o">=</span>-D__LP64__<span class="w"> </span>--with-wrapper-cflags<span class="o">=</span><span class="s2">"-D__LP64__ -ta:tesla"</span>
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="what-kind-of-cuda-support-exists-in-open-mpi">
<h2><span class="section-number">11.2.6.5. </span>What kind of CUDA support exists in Open MPI?<a class="headerlink" href="#what-kind-of-cuda-support-exists-in-open-mpi" title="Permalink to this heading"></a></h2>
<p>CUDA-aware support is defined as Open MPI automatically detecting that
the argument pointer being passed to an MPI routine is a CUDA device
memory pointer.</p>
<p>See <a class="reference internal" href="#faq-cuda-mpi-apis-cuda-label"><span class="std std-ref">this FAQ entry</span></a>
for more details on which APIs are CUDA-aware.</p>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>CUDA 4.0 is SUPER OLD! End users dont care about the
differences between cuda-aware, cuda-ipc, gpu-direct, and gpu-direct-rdma</p>
</div>
<p>Open MPI depends on various features of CUDA 4.0, so one needs to have
at least the CUDA 4.0 driver and toolkit. The new features of
interest are the Unified Virtual Addressing (UVA) so that all pointers
within a program have unique addresses. In addition, there is a new
API that allows one to determine if a pointer is a CUDA device pointer
or host memory pointer. This API is used by the library to decide
what needs to be done with each buffer. In addition, CUDA 4.1 also
provides the ability to register host memory with the CUDA driver,
which can improve performance. CUDA 4.1 also added CUDA IPC support
for fast communication between GPUs on the same node.</p>
<p>Note that derived datatypes — both contiguous and non-contiguous
— are supported. However, the non-contiguous datatypes
currently have high overhead because of the many calls to the CUDA
function <code class="docutils literal notranslate"><span class="pre">cuMemcpy()</span></code> to copy all the pieces of the buffer into the
intermediate buffer.</p>
<p>CUDA-aware support is available in:</p>
<ul class="simple">
<li><p>The UCX (<code class="docutils literal notranslate"><span class="pre">ucx</span></code>) PML</p></li>
<li><p>The PSM2 (<code class="docutils literal notranslate"><span class="pre">psm2</span></code>) MTL with the CM (<code class="docutils literal notranslate"><span class="pre">cm</span></code>) PML.</p></li>
<li><p>The OFI (<code class="docutils literal notranslate"><span class="pre">ofi</span></code>) MTL with the CM (<code class="docutils literal notranslate"><span class="pre">cm</span></code>) PML.</p></li>
<li><p>Both CUDA-ized shared memory (<code class="docutils literal notranslate"><span class="pre">smcuda</span></code>) and TCP (<code class="docutils literal notranslate"><span class="pre">tcp</span></code>) BTLs
with the OB1 (<code class="docutils literal notranslate"><span class="pre">ob1</span></code>) PML.</p></li>
<li><p>The HCOLL (<code class="docutils literal notranslate"><span class="pre">hcoll</span></code>) COLL</p></li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="psm2-support-for-cuda">
<h2><span class="section-number">11.2.6.6. </span>PSM2 support for CUDA<a class="headerlink" href="#psm2-support-for-cuda" title="Permalink to this heading"></a></h2>
<p>CUDA-aware support is present in PSM2 MTL. When running CUDA-aware
Open MPI on Cornelis Networks Omni-Path, the PSM2 MTL will automatically set
<code class="docutils literal notranslate"><span class="pre">PSM2_CUDA</span></code> environment variable which enables PSM2 to handle GPU
buffers. If the user wants to use host buffers with a CUDA-aware Open
MPI, it is recommended to set <code class="docutils literal notranslate"><span class="pre">PSM2_CUDA</span></code> to <code class="docutils literal notranslate"><span class="pre">0</span></code> in the execution
environment. PSM2 also has support for the NVIDIA GPUDirect support
feature. To enable this, users will need to set <code class="docutils literal notranslate"><span class="pre">PSM2_GPUDIRECT</span></code>
to <code class="docutils literal notranslate"><span class="pre">1</span></code> in the execution environment.</p>
<p>Note: The PSM2 library and <code class="docutils literal notranslate"><span class="pre">hfi1</span></code> driver with CUDA support are requirements
to use GPUDirect support on Cornelis Networks Omni-Path. The minimum
PSM2 build version required is <a class="reference external" href="https://github.com/01org/opa-psm2/releases/tag/PSM2_10.2-175">PSM2 10.2.175</a>.</p>
<p>For more information refer to the <a class="reference external" href="https://customercenter.cornelisnetworks.com/">Cornelis Networks Customer Center</a>.</p>
</div>
<hr class="docutils" />
<div class="section" id="ofi-support-for-cuda">
<h2><span class="section-number">11.2.6.7. </span>OFI support for CUDA<a class="headerlink" href="#ofi-support-for-cuda" title="Permalink to this heading"></a></h2>
<p>CUDA-aware support is present in OFI MTL. When running CUDA-aware
Open MPI over Libfabric, the OFI MTL will check if there are any
providers capable of handling GPU (or other accelerator) memory
through the <code class="docutils literal notranslate"><span class="pre">hmem</span></code>-related flags. If a CUDA-capable provider is
available, the OFI MTL will directly send GPU buffers through
Libfabric’s API after registering the memory. If there are no
CUDA-capable providers available, the buffers will automatically
be copied to host buffers before being transferred through
Libfabric’s API.</p>
</div>
<hr class="docutils" />
<div class="section" id="can-i-get-additional-cuda-debug-level-information-at-run-time">
<h2><span class="section-number">11.2.6.8. </span>Can I get additional CUDA debug-level information at run-time?<a class="headerlink" href="#can-i-get-additional-cuda-debug-level-information-at-run-time" title="Permalink to this heading"></a></h2>
<p>Yes, by enabling some vebosity flags.</p>
<ul>
<li><p>The <code class="docutils literal notranslate"><span class="pre">opal_cuda_verbose</span></code> parameter has only one level of verbosity:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun --mca opal_cuda_verbose 10 ...
</pre></div>
</div>
</li>
<li><p>The <code class="docutils literal notranslate"><span class="pre">mpi_common_cuda_verbose</span></code> parameter provides additional
information about CUDA-aware related activities. This can be set to
a variety of different values. There is really no need to use these
unless you have strange problems:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># A bunch of CUDA debug information</span>
shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>mpi_common_cuda_verbose<span class="w"> </span><span class="m">10</span><span class="w"> </span>...
<span class="c1"># Even more CUDA debug information</span>
shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>mpi_common_cuda_verbose<span class="w"> </span><span class="m">20</span><span class="w"> </span>...
<span class="c1"># Yet more CUDA debug information</span>
shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>mpi_common_cuda_verbose<span class="w"> </span><span class="m">100</span><span class="w"> </span>...
</pre></div>
</div>
</li>
<li><p>The <code class="docutils literal notranslate"><span class="pre">smcuda</span></code> BTL has three MCA parameters related to the use of
CUDA IPC. By default, CUDA IPC is used where possible. But the
user can now turn it off if they prefer.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>btl_smcuda_use_cuda_ipc<span class="w"> </span><span class="m">0</span><span class="w"> </span>...
</pre></div>
</div>
<p>In addition, it is assumed that CUDA IPC is possible when running on
the same GPU, and this is typically true. However, there is the
ability to turn it off.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>btl_smcuda_use_cuda_ipc_same_gpu<span class="w"> </span><span class="m">0</span><span class="w"> </span>...
</pre></div>
</div>
<p>Last, to get some insight into whether CUDA IPC is being used, you
can turn on some verbosity that shows whether CUDA IPC gets enabled
between two GPUs.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>mpirun<span class="w"> </span>--mca<span class="w"> </span>btl_smcuda_cuda_ipc_verbose<span class="w"> </span><span class="m">100</span><span class="w"> </span>...
</pre></div>
</div>
</li>
</ul>
<hr class="docutils" />
</div>
<div class="section" id="numa-node-issues">
<span id="faq-cuda-mpi-cuda-numa-issues-label"></span><h2><span class="section-number">11.2.6.9. </span>NUMA Node Issues<a class="headerlink" href="#numa-node-issues" title="Permalink to this heading"></a></h2>
<p>When running on a node that has multiple GPUs, you may want to select
the GPU that is closest to the NUMA node on which your process is
running. One way to do this is to make use of the <code class="docutils literal notranslate"><span class="pre">hwloc</span></code> library.
The following is a C code snippet that can be used in your application
to select a GPU that is close. It will determine on which CPU it is
running and then look for the closest GPU. There could be multiple
GPUs that are the same distance away. This is dependent on having
<code class="docutils literal notranslate"><span class="pre">hwloc</span></code> somewhere on your system.</p>
<div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cm">/**</span>
<span class="cm"> * Test program to show the use of hwloc to select the GPU closest to the CPU</span>
<span class="cm"> * that the MPI program is running on. Note that this works even without</span>
<span class="cm"> * any libpciaccess or libpci support as it keys off the NVIDIA vendor ID.</span>
<span class="cm"> * There may be other ways to implement this but this is one way.</span>
<span class="cm"> * January 10, 2014</span>
<span class="cm"> */</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><assert.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"cuda.h"</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"mpi.h"</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"hwloc.h"</span>
<span class="cp">#define ABORT_ON_ERROR(func) \</span>
<span class="cp"> { CUresult res; \</span>
<span class="cp"> res = func; \</span>
<span class="cp"> if (CUDA_SUCCESS != res) { \</span>
<span class="cp"> printf("%s returned error=%d\n", #func, res); \</span>
<span class="cp"> abort(); \</span>
<span class="cp"> } \</span>
<span class="cp"> }</span>
<span class="k">static</span><span class="w"> </span><span class="n">hwloc_topology_t</span><span class="w"> </span><span class="n">topology</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">NULL</span><span class="p">;</span>
<span class="k">static</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">gpuIndex</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="k">static</span><span class="w"> </span><span class="n">hwloc_obj_t</span><span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="mi">16</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="mi">0</span><span class="p">};</span>
<span class="cm">/**</span>
<span class="cm"> * This function searches for all the GPUs that are hanging off a NUMA</span>
<span class="cm"> * node. It walks through each of the PCI devices and looks for ones</span>
<span class="cm"> * with the NVIDIA vendor ID. It then stores them into an array.</span>
<span class="cm"> * Note that there can be more than one GPU on the NUMA node.</span>
<span class="cm"> */</span>
<span class="k">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">find_gpus</span><span class="p">(</span><span class="n">hwloc_topology_t</span><span class="w"> </span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">hwloc_obj_t</span><span class="w"> </span><span class="n">parent</span><span class="p">,</span><span class="w"> </span><span class="n">hwloc_obj_t</span><span class="w"> </span><span class="n">child</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">hwloc_obj_t</span><span class="w"> </span><span class="n">pcidev</span><span class="p">;</span>
<span class="w"> </span><span class="n">pcidev</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_get_next_child</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">parent</span><span class="p">,</span><span class="w"> </span><span class="n">child</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">NULL</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">pcidev</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="mi">0</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">pcidev</span><span class="o">-></span><span class="n">arity</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="cm">/* This device has children so need to look recursively at them */</span>
<span class="w"> </span><span class="n">find_gpus</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">pcidev</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">);</span>
<span class="w"> </span><span class="n">find_gpus</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">parent</span><span class="p">,</span><span class="w"> </span><span class="n">pcidev</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pcidev</span><span class="o">-></span><span class="n">attr</span><span class="o">-></span><span class="n">pcidev</span><span class="p">.</span><span class="n">vendor_id</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mh">0x10de</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="n">gpuIndex</span><span class="o">++</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">pcidev</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">find_gpus</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">parent</span><span class="p">,</span><span class="w"> </span><span class="n">pcidev</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">rank</span><span class="p">,</span><span class="w"> </span><span class="n">retval</span><span class="p">,</span><span class="w"> </span><span class="n">length</span><span class="p">;</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">procname</span><span class="p">[</span><span class="n">MPI_MAX_PROCESSOR_NAME</span><span class="o">+</span><span class="mi">1</span><span class="p">];</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">flags</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">HWLOC_TOPOLOGY_FLAG_IO_DEVICES</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">HWLOC_TOPOLOGY_FLAG_IO_BRIDGES</span><span class="p">;</span>
<span class="w"> </span><span class="n">hwloc_cpuset_t</span><span class="w"> </span><span class="n">newset</span><span class="p">;</span>
<span class="w"> </span><span class="n">hwloc_obj_t</span><span class="w"> </span><span class="n">node</span><span class="p">,</span><span class="w"> </span><span class="n">bridge</span><span class="p">;</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">pciBusId</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span>
<span class="w"> </span><span class="n">CUdevice</span><span class="w"> </span><span class="n">dev</span><span class="p">;</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">devName</span><span class="p">[</span><span class="mi">256</span><span class="p">];</span>
<span class="w"> </span><span class="n">MPI_Init</span><span class="p">(</span><span class="o">&</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">argv</span><span class="p">);</span>
<span class="w"> </span><span class="n">MPI_Comm_rank</span><span class="p">(</span><span class="n">MPI_COMM_WORLD</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">rank</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">MPI_SUCCESS</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">MPI_Get_processor_name</span><span class="p">(</span><span class="n">procname</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">length</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">strcpy</span><span class="p">(</span><span class="n">procname</span><span class="p">,</span><span class="w"> </span><span class="s">"unknown"</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="cm">/* Now decide which GPU to pick. This requires hwloc to work properly.</span>
<span class="cm"> * We first see which CPU we are bound to, then try and find a GPU nearby.</span>
<span class="cm"> */</span>
<span class="w"> </span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_topology_init</span><span class="p">(</span><span class="o">&</span><span class="n">topology</span><span class="p">);</span>
<span class="w"> </span><span class="n">assert</span><span class="p">(</span><span class="n">retval</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_topology_set_flags</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">flags</span><span class="p">);</span>
<span class="w"> </span><span class="n">assert</span><span class="p">(</span><span class="n">retval</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_topology_load</span><span class="p">(</span><span class="n">topology</span><span class="p">);</span>
<span class="w"> </span><span class="n">assert</span><span class="p">(</span><span class="n">retval</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="n">newset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_bitmap_alloc</span><span class="p">();</span>
<span class="w"> </span><span class="n">retval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_get_last_cpu_location</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">newset</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="n">assert</span><span class="p">(</span><span class="n">retval</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="cm">/* Get the object that contains the cpuset */</span>
<span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_get_first_largest_obj_inside_cpuset</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">newset</span><span class="p">);</span>
<span class="w"> </span><span class="cm">/* Climb up from that object until we find the HWLOC_OBJ_NODE */</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="o">-></span><span class="n">type</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">HWLOC_OBJ_NODE</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">node</span><span class="o">-></span><span class="n">parent</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="cm">/* Now look for the HWLOC_OBJ_BRIDGE. All PCI busses hanging off the</span>
<span class="cm"> * node will have one of these */</span>
<span class="w"> </span><span class="n">bridge</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_get_next_child</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">node</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">);</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">bridge</span><span class="o">-></span><span class="n">type</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">HWLOC_OBJ_BRIDGE</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">bridge</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hwloc_get_next_child</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">node</span><span class="p">,</span><span class="w"> </span><span class="n">bridge</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="cm">/* Now find all the GPUs on this NUMA node and put them into an array */</span>
<span class="w"> </span><span class="n">find_gpus</span><span class="p">(</span><span class="n">topology</span><span class="p">,</span><span class="w"> </span><span class="n">bridge</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">);</span>
<span class="w"> </span><span class="n">ABORT_ON_ERROR</span><span class="p">(</span><span class="n">cuInit</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span>
<span class="w"> </span><span class="cm">/* Now select the first GPU that we find */</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">gpus</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"No GPU found</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sprintf</span><span class="p">(</span><span class="n">pciBusId</span><span class="p">,</span><span class="w"> </span><span class="s">"%.2x:%.2x:%.2x.%x"</span><span class="p">,</span><span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-></span><span class="n">attr</span><span class="o">-></span><span class="n">pcidev</span><span class="p">.</span><span class="n">domain</span><span class="p">,</span><span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-></span><span class="n">attr</span><span class="o">-></span><span class="n">pcidev</span><span class="p">.</span><span class="n">bus</span><span class="p">,</span>
<span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-></span><span class="n">attr</span><span class="o">-></span><span class="n">pcidev</span><span class="p">.</span><span class="n">dev</span><span class="p">,</span><span class="w"> </span><span class="n">gpus</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-></span><span class="n">attr</span><span class="o">-></span><span class="n">pcidev</span><span class="p">.</span><span class="n">func</span><span class="p">);</span>
<span class="w"> </span><span class="n">ABORT_ON_ERROR</span><span class="p">(</span><span class="n">cuDeviceGetByPCIBusId</span><span class="p">(</span><span class="o">&</span><span class="n">dev</span><span class="p">,</span><span class="w"> </span><span class="n">pciBusId</span><span class="p">));</span>
<span class="w"> </span><span class="n">ABORT_ON_ERROR</span><span class="p">(</span><span class="n">cuDeviceGetName</span><span class="p">(</span><span class="n">devName</span><span class="p">,</span><span class="w"> </span><span class="mi">256</span><span class="p">,</span><span class="w"> </span><span class="n">dev</span><span class="p">));</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"rank=%d (%s): Selected GPU=%s, name=%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">rank</span><span class="p">,</span><span class="w"> </span><span class="n">procname</span><span class="p">,</span><span class="w"> </span><span class="n">pciBusId</span><span class="p">,</span><span class="w"> </span><span class="n">devName</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">MPI_Finalize</span><span class="p">();</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-develop-cuda-aware-open-mpi-applications">
<h2><span class="section-number">11.2.6.10. </span>How do I develop CUDA-aware Open MPI applications?<a class="headerlink" href="#how-do-i-develop-cuda-aware-open-mpi-applications" title="Permalink to this heading"></a></h2>
<p>Developing CUDA-aware applications is a complex topic, and beyond the
scope of this document. CUDA-aware applications often have to take
machine-specific considerations into account, including the number of
GPUs installed on each node and how the GPUs are connected to the CPUs
and to each other. Often, when using a particular transport layer
(such as OPA/PSM2) there will be run-time decisions to make about
which CPU cores will be used with which GPUs.</p>
<p>A good place to start is the <a class="reference external" href="https://docs.nvidia.com/cuda/">NVIDIA CUDA Toolkit Documentation</a> including the <a class="reference external" href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/">Programming Guide</a> and the
<a class="reference external" href="https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/">Best Practices Guide</a>. For
examples of how to write CUDA-aware MPI applications, the <a class="reference external" href="https://github.com/NVIDIA-developer-blog/code-samples/tree/master/posts/cuda-aware-mpi-example">NVIDIA
developers blog</a>
offers examples and the <a class="reference external" href="https://mvapich.cse.ohio-state.edu/benchmarks/">OSU Micro-Benchmarks</a> offer an excellent
example of how to write CUDA-aware MPI applications.</p>
<hr class="docutils" />
</div>
<div class="section" id="which-mpi-apis-work-with-cuda-aware">
<span id="faq-cuda-mpi-apis-cuda-label"></span><h2><span class="section-number">11.2.6.11. </span>Which MPI APIs work with CUDA-aware?<a class="headerlink" href="#which-mpi-apis-work-with-cuda-aware" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p>MPI_Allgather</p></li>
<li><p>MPI_Allgatherv</p></li>
<li><p>MPI_Allreduce</p></li>
<li><p>MPI_Alltoall</p></li>
<li><p>MPI_Alltoallv</p></li>
<li><p>MPI_Alltoallw</p></li>
<li><p>MPI_Bcast</p></li>
<li><p>MPI_Bsend</p></li>
<li><p>MPI_Bsend_init</p></li>
<li><p>MPI_Exscan</p></li>
<li><p>MPI_Ibsend</p></li>
<li><p>MPI_Irecv</p></li>
<li><p>MPI_Isend</p></li>
<li><p>MPI_Irsend</p></li>
<li><p>MPI_Issend</p></li>
<li><p>MPI_Gather</p></li>
<li><p>MPI_Gatherv</p></li>
<li><p>MPI_Get</p></li>
<li><p>MPI_Put</p></li>
<li><p>MPI_Rsend</p></li>
<li><p>MPI_Rsend_init</p></li>
<li><p>MPI_Recv</p></li>
<li><p>MPI_Recv_init</p></li>
<li><p>MPI_Reduce</p></li>
<li><p>MPI_Reduce_scatter</p></li>
<li><p>MPI_Reduce_scatter_block</p></li>
<li><p>MPI_Scan</p></li>
<li><p>MPI_Scatter</p></li>
<li><p>MPI_Scatterv</p></li>
<li><p>MPI_Send</p></li>
<li><p>MPI_Send_init</p></li>
<li><p>MPI_Sendrecv</p></li>
<li><p>MPI_Ssend</p></li>
<li><p>MPI_Ssend_init</p></li>
<li><p>MPI_Win_create</p></li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="which-mpi-apis-do-not-work-with-cuda-aware">
<h2><span class="section-number">11.2.6.12. </span>Which MPI APIs do NOT work with CUDA-aware?<a class="headerlink" href="#which-mpi-apis-do-not-work-with-cuda-aware" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p>MPI_Accumulate</p></li>
<li><p>MPI_Compare_and_swap</p></li>
<li><p>MPI_Fetch_and_op</p></li>
<li><p>MPI_Get_Accumulate</p></li>
<li><p>MPI_Iallgather</p></li>
<li><p>MPI_Iallgatherv</p></li>
<li><p>MPI_Iallreduce</p></li>
<li><p>MPI_Ialltoall</p></li>
<li><p>MPI_Ialltoallv</p></li>
<li><p>MPI_Ialltoallw</p></li>
<li><p>MPI_Ibcast</p></li>
<li><p>MPI_Iexscan</p></li>
<li><p>MPI_Rget</p></li>
<li><p>MPI_Rput</p></li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-use-cuda-aware-ucx-for-open-mpi">
<h2><span class="section-number">11.2.6.13. </span>How do I use CUDA-aware UCX for Open MPI?<a class="headerlink" href="#how-do-i-use-cuda-aware-ucx-for-open-mpi" title="Permalink to this heading"></a></h2>
<p>Example of running <code class="docutils literal notranslate"><span class="pre">osu_latency</span></code> from the <a class="reference external" href="https://mvapich.cse.ohio-state.edu/benchmarks">OSU benchmarks</a> with CUDA buffers
using Open MPI and UCX CUDA support:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun -n 2 --mca pml ucx \
-x UCX_TLS=rc,sm,cuda_copy,gdr_copy,cuda_ipc ./osu_latency D D
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="which-mpi-apis-work-with-cuda-aware-ucx">
<h2><span class="section-number">11.2.6.14. </span>Which MPI APIs work with CUDA-aware UCX?<a class="headerlink" href="#which-mpi-apis-work-with-cuda-aware-ucx" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p>MPI_Send</p></li>
<li><p>MPI_Bsend</p></li>
<li><p>MPI_Ssend</p></li>
<li><p>MPI_Rsend</p></li>
<li><p>MPI_Isend</p></li>
<li><p>MPI_Ibsend</p></li>
<li><p>MPI_Issend</p></li>
<li><p>MPI_Irsend</p></li>
<li><p>MPI_Send_init</p></li>
<li><p>MPI_Bsend_init</p></li>
<li><p>MPI_Ssend_init</p></li>
<li><p>MPI_Rsend_init</p></li>
<li><p>MPI_Recv</p></li>
<li><p>MPI_Irecv</p></li>
<li><p>MPI_Recv_init</p></li>
<li><p>MPI_Sendrecv</p></li>
<li><p>MPI_Bcast</p></li>
<li><p>MPI_Gather</p></li>
<li><p>MPI_Gatherv</p></li>
<li><p>MPI_Allgather</p></li>
<li><p>MPI_Reduce</p></li>
<li><p>MPI_Reduce_scatter</p></li>
<li><p>MPI_Reduce_scatter_block</p></li>
<li><p>MPI_Allreduce</p></li>
<li><p>MPI_Scan</p></li>
<li><p>MPI_Exscan</p></li>
<li><p>MPI_Allgatherv</p></li>
<li><p>MPI_Alltoall</p></li>
<li><p>MPI_Alltoallv</p></li>
<li><p>MPI_Alltoallw</p></li>
<li><p>MPI_Scatter</p></li>
<li><p>MPI_Scatterv</p></li>
<li><p>MPI_Iallgather</p></li>
<li><p>MPI_Iallgatherv</p></li>
<li><p>MPI_Ialltoall</p></li>
<li><p>MPI_Iialltoallv</p></li>
<li><p>MPI_Ialltoallw</p></li>
<li><p>MPI_Ibcast</p></li>
<li><p>MPI_Iexscan</p></li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="which-mpi-apis-do-not-work-with-cuda-aware-ucx">
<h2><span class="section-number">11.2.6.15. </span>Which MPI APIs do NOT work with CUDA-aware UCX?<a class="headerlink" href="#which-mpi-apis-do-not-work-with-cuda-aware-ucx" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p>All one-sided operations such as MPI_Put, MPI_Get, MPI_Accumulate,
MPI_Rget, MPI_Rput, MPI_Get_Accumulate, MPI_Fetch_and_op,
MPI_Compare_and_swap, etc</p></li>
<li><p>All window creation calls such as MPI_Win_create</p></li>
<li><p>All non-blocking reduction collectives like MPI_Ireduce,
MPI_Iallreduce, etc</p></li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="can-i-tell-at-compile-time-or-runtime-whether-i-have-cuda-aware-support">
<h2><span class="section-number">11.2.6.16. </span>Can I tell at compile time or runtime whether I have CUDA-aware support?<a class="headerlink" href="#can-i-tell-at-compile-time-or-runtime-whether-i-have-cuda-aware-support" title="Permalink to this heading"></a></h2>
<p>There is both a compile time check and a run-time check available.
You can use whichever is the most convenient for your program. To
access them, you need to include <code class="docutils literal notranslate"><span class="pre">mpi-ext.h</span></code>. Note that
<code class="docutils literal notranslate"><span class="pre">mpi-ext.h</span></code> is specific to Open MPI. The following program shows an
example of using the CUDA-aware macro and run-time check.</p>
<div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cm">/*</span>
<span class="cm"> * Program that shows the use of CUDA-aware macro and runtime check.</span>
<span class="cm"> */</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"mpi.h"</span>
<span class="cp">#if !defined(OPEN_MPI) || !OPEN_MPI</span>
<span class="cp">#error This source code uses an Open MPI-specific extension</span>
<span class="cp">#endif</span>
<span class="cm">/* Needed for MPIX_Query_cuda_support(), below */</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"mpi-ext.h"</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">MPI_Init</span><span class="p">(</span><span class="o">&</span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">argv</span><span class="p">);</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Compile time check:</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="cp">#if defined(MPIX_CUDA_AWARE_SUPPORT) && MPIX_CUDA_AWARE_SUPPORT</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library has CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">MPIX_CUDA_AWARE_SUPPORT</span><span class="p">);</span>
<span class="cp">#elif defined(MPIX_CUDA_AWARE_SUPPORT) && !MPIX_CUDA_AWARE_SUPPORT</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library does not have CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="cp">#else</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library cannot determine if there is CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="cp">#endif </span><span class="cm">/* MPIX_CUDA_AWARE_SUPPORT */</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Run time check:</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="cp">#if defined(MPIX_CUDA_AWARE_SUPPORT)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">MPIX_Query_cuda_support</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library has CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library does not have CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="cp">#else </span><span class="cm">/* !defined(MPIX_CUDA_AWARE_SUPPORT) */</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"This MPI library cannot determine if there is CUDA-aware support.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="cp">#endif </span><span class="cm">/* MPIX_CUDA_AWARE_SUPPORT */</span>
<span class="w"> </span><span class="n">MPI_Finalize</span><span class="p">();</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-limit-how-much-cuda-ipc-memory-is-held-in-the-registration-cache">
<h2><span class="section-number">11.2.6.17. </span>How do I limit how much CUDA IPC memory is held in the registration cache?<a class="headerlink" href="#how-do-i-limit-how-much-cuda-ipc-memory-is-held-in-the-registration-cache" title="Permalink to this heading"></a></h2>
<p>As mentioned earlier, the Open MPI library will make use of CUDA IPC support where
possible to move the GPU data quickly between GPUs that are on the same node and
same PCI root complex. The library holds on to registrations even after the data
transfer is complete as it is expensive to make some of the CUDA IPC registration
calls. If you want to limit how much memory is registered, you can use the
<code class="docutils literal notranslate"><span class="pre">mpool_rgpusm_rcache_size_limit</span></code> MCA parameter. For example, this sets the limit
to 1000000 bytes:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun --mca mpool_rgpusm_rcache_size_limit 1000000 ...
</pre></div>
</div>
<p>When the cache reaches this size, it will kick out the least recently used until
it can fit the new registration in.</p>
<p>There also is the ability to have the cache empty itself out when the
limit is reached:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun --mca mpool_rgpusm_rcache_empty_cache 1 ...
</pre></div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="what-are-some-guidelines-for-using-cuda-and-open-mpi-with-omni-path">
<h2><span class="section-number">11.2.6.18. </span>What are some guidelines for using CUDA and Open MPI with Omni-Path?<a class="headerlink" href="#what-are-some-guidelines-for-using-cuda-and-open-mpi-with-omni-path" title="Permalink to this heading"></a></h2>
<p>When developing CUDA-aware Open MPI applications for OPA-based fabrics, the
PSM2 transport is preferred and a CUDA-aware version of PSM2 is provided with
all versions of the Cornelis Networks Omni-Path OPXS software suite.</p>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>TODO Are Intel/OPA references still correct?</p>
</div>
<p>The PSM2 library provides a number of settings that will govern how it
will interact with CUDA, including <code class="docutils literal notranslate"><span class="pre">PSM2_CUDA</span></code> and <code class="docutils literal notranslate"><span class="pre">PSM2_GPUDIRECT</span></code>,
which should be set in the environment before <code class="docutils literal notranslate"><span class="pre">MPI_Init()</span></code> is called. For
example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun -x PSM2_CUDA=1 -x PSM2_GPUDIRECT=1 --mca mtl psm2 mpi_hello
</pre></div>
</div>
<p>In addition, each process of the application should select a specific
GPU card to use before calling <code class="docutils literal notranslate"><span class="pre">MPI_Init()</span></code>, by using
<code class="docutils literal notranslate"><span class="pre">cudaChooseDevice()</span></code>, <code class="docutils literal notranslate"><span class="pre">cudaSetDevice()</span></code> and similar. The chosen
GPU should be within the same NUMA node as the CPU the MPI process is
running on. You will also want to use the <code class="docutils literal notranslate"><span class="pre">mpirun</span></code>
<code class="docutils literal notranslate"><span class="pre">--bind-to-core</span></code> or <code class="docutils literal notranslate"><span class="pre">--bind-to-socket</span></code> option to ensure that MPI
processes do not move between NUMA nodes. See the section on
<a class="reference internal" href="#faq-cuda-mpi-cuda-numa-issues-label"><span class="std std-ref">NUMA Node Issues</span></a>, for
more information.</p>
<p>For more information see the <em>Cornelis Networks Performance Scaled Messaging 2
(PSM2) Programmer’s Guide</em> and the <em>Cornelis Networks Omni-Path Performance
Tuning Guide</em>, which can be found in the <a class="reference external" href="https://customercenter.cornelisnetworks.com/">Cornelis Networks Customer Center</a>.</p>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>TODO Are Intel/OPA references still correct?</p>
</div>
</div>
<hr class="docutils" />
<div class="section" id="when-do-i-need-to-select-a-cuda-device">
<h2><span class="section-number">11.2.6.19. </span>When do I need to select a CUDA device?<a class="headerlink" href="#when-do-i-need-to-select-a-cuda-device" title="Permalink to this heading"></a></h2>
<p>“mpi-cuda-dev-selection”</p>
<p>OpenMPI requires CUDA resources allocated for internal use. These
are allocated lazily when they are first needed, e.g. CUDA IPC mem handles
are created when a communication routine first requires them during a
transfer. So, the CUDA device needs to be selected before the first MPI
call requiring a CUDA resource. MPI_Init and most communicator related
operations do not create any CUDA resources (guaranteed for MPI_Init,
MPI_Comm_rank, MPI_Comm_size, MPI_Comm_split_type and MPI_Comm_free). It
is thus possible to use those routines to query rank information and use
those to select a GPU, e.g. using</p>
<div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="n">local_rank</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">-1</span><span class="p">;</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">MPI_Comm</span><span class="w"> </span><span class="n">local_comm</span><span class="p">;</span>
<span class="w"> </span><span class="n">MPI_Comm_split_type</span><span class="p">(</span><span class="n">MPI_COMM_WORLD</span><span class="p">,</span><span class="w"> </span><span class="n">MPI_COMM_TYPE_SHARED</span><span class="p">,</span><span class="w"> </span><span class="n">rank</span><span class="p">,</span><span class="w"> </span><span class="n">MPI_INFO_NULL</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">local_comm</span><span class="p">);</span>
<span class="w"> </span><span class="n">MPI_Comm_rank</span><span class="p">(</span><span class="n">local_comm</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">local_rank</span><span class="p">);</span>
<span class="w"> </span><span class="n">MPI_Comm_free</span><span class="p">(</span><span class="o">&</span><span class="n">local_comm</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">int</span><span class="w"> </span><span class="n">num_devices</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="n">cudaGetDeviceCount</span><span class="p">(</span><span class="o">&</span><span class="n">num_devices</span><span class="p">);</span>
<span class="n">cudaSetDevice</span><span class="p">(</span><span class="n">local_rank</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">num_devices</span><span class="p">);</span>
</pre></div>
</div>
<p>MPI internal CUDA resources are released during MPI_Finalize. Thus it is an
application error to call cudaDeviceReset before MPI_Finalize is called.</p>
</div>
<hr class="docutils" />
<div class="section" id="how-do-i-enable-cuda-support-in-hcoll-collective-component">
<h2><span class="section-number">11.2.6.20. </span>How do I enable CUDA support in HCOLL collective component<a class="headerlink" href="#how-do-i-enable-cuda-support-in-hcoll-collective-component" title="Permalink to this heading"></a></h2>
<p>HCOLL component supports CUDA GPU buffers for the following
collectives:</p>
<p>MPI_Allreduce
MPI_Bcast
MPI_Allgather
MPI_Ibarrier
MPI_Ibcast
MPI_Iallgather
MPI_Iallreduce</p>
<p>To enable CUDA GPU buffer support in these collectives pass the
following environment variables via mpirun:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>shell$ mpirun -x HCOLL_GPU_ENABLE=1 -x HCOLL_ENABLE_NBC=1 ..
</pre></div>
</div>
<p>See <a class="reference external" href="https://docs.nvidia.com/networking/display/HPCXv29/HCOLL">nVidia HCOLL documentation</a>
for more information.</p>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="iwarp.html" class="btn btn-neutral float-left" title="11.2.5. iWARP Support" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="rocm.html" class="btn btn-neutral float-right" title="11.2.7. ROCm" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2003-2025, The Open MPI Community.
<span class="lastupdated">Last updated on 2025-05-30 16:41:43 UTC.
</span></p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
