1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
|
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>6.1. Session Directories — OpenPMIx 5.0.8a1 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="6.2. Resolving Peers and Nodes" href="resolve.html" />
<link rel="prev" title="6. How Things Work" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
OpenPMIx
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">2. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes/index.html">3. Release notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../exceptions.html">4. Exceptions to the PMIx Standard</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installing-pmix/index.html">5. Building and installing PMIx</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">6. How Things Work</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">6.1. Session Directories</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#client-session-directories">6.1.1. Client Session Directories</a></li>
<li class="toctree-l3"><a class="reference internal" href="#tool-and-server-session-directories">6.1.2. Tool and Server Session Directories</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="resolve.html">6.2. Resolving Peers and Nodes</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">7. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../history.html">8. History</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">9. Version Numbers and Binary Compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mca.html">10. The Modular Component Architecture (MCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../building-apps/index.html">11. Building PMIx applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">12. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">13. Contributing to OpenPMIx</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">14. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">15. OpenPMIx Security Policy</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">17. OpenPMIx manual pages</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">OpenPMIx</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">6. </span>How Things Work</a></li>
<li class="breadcrumb-item active"><span class="section-number">6.1. </span>Session Directories</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/how-things-work/session_dirs.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="session-directories">
<h1><span class="section-number">6.1. </span>Session Directories<a class="headerlink" href="#session-directories" title="Permalink to this heading"></a></h1>
<p>In general, servers, tools, and application processes all have access to their own <code class="docutils literal notranslate"><span class="pre">session</span> <span class="pre">directory</span></code> - a location where scratch files can be safely placed with a reasonable guarantee of automatic cleanup upon termination. Session directories provide a safe location (i.e., in a temporary file system and guaranteed not to conflict with other sessions/jobs/applications) for executables to use when creating scratch files such as shared memory backing files and rendezvous files. PMIx and PRRTE also provide a reasonable guarantee that any files and/or subdirectories created under the specified location will be automatically cleaned up at finalize and/or termination. In this case, <code class="docutils literal notranslate"><span class="pre">reasonable</span></code> means that we will do our best to remove all files and subdirectories, but cannot fully guarantee removal in situations outside of our control (e.g., being forcibly terminated via <cite>SIGKILL</cite>).</p>
<blockquote>
<div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>In general, the host (e.g., PRRTE) is responsible for creating session
directories. In some cases, PMIx creates a limited set of session
directories if the host does not provide them - e.g., in the case of a
self-launched tool - for storing contact information. This is outlined
below.</p>
</div>
</div></blockquote>
<p>The following attributes can be used to pass session directory information to the PMIx library:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_SYSTEM_TMPDIR</span></code>: temporary directory for this system (typically <code class="docutils literal notranslate"><span class="pre">/tmp</span></code> for Linux systems).
The PMIx server will place tool rendezvous points and contact info in this location. In the
absence of this attribute during <code class="docutils literal notranslate"><span class="pre">PMIx_Init</span></code> (or the server/tool version), the PMIx library
will first look for the <code class="docutils literal notranslate"><span class="pre">TMPDIR</span></code> envar, then <code class="docutils literal notranslate"><span class="pre">TEMP</span></code>, and finally <code class="docutils literal notranslate"><span class="pre">TMP</span></code> - if none of
those are found, the library will default to the <code class="docutils literal notranslate"><span class="pre">/tmp</span></code> location.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code>: session directory where the PMIx server will place client rendezvous
points and contact info. If not provided, the library will first look for the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code>
envar, then <code class="docutils literal notranslate"><span class="pre">TMPDIR</span></code>, <code class="docutils literal notranslate"><span class="pre">TEMP</span></code>, and finally <code class="docutils literal notranslate"><span class="pre">TMP</span></code> - if none of those are found, the
library will default to the <code class="docutils literal notranslate"><span class="pre">/tmp</span></code> location.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_TMPDIR</span></code>: top-level temporary directory assigned to a session. Often equated to
<code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code> by host environments.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_NSDIR</span></code>: session directory assigned to a namespace. Usually placed underneath the
<code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code> and given a name based on the namespace itself.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_PROCDIR</span></code>: session directory assigned to an individual process. Usually placed underneath
the <code class="docutils literal notranslate"><span class="pre">PMIX_NSDIR</span></code> assigned to the namespace of the process, and given a string name equivalent
to the rank of the process.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_LAUNCHER_RENDEZVOUS_FILE</span></code>: the full path name of a file wherein a tool shall output its
connection information - e.g., a launcher to store contact information so that a debugger
can attach to it.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">PMIX_TDIR_RMCLEAN</span></code>: the host environment (often known as the “resource manager” or “RM”) will
cleanup the session directories. In the absence of this attribute, the PMIx library will
remove all files in any session directory it created and then remove the directory itself.</p></li>
</ul>
<p>These same attributes can be used in calls to <code class="docutils literal notranslate"><span class="pre">PMIx_Get</span></code> by application processes to retrieve
the session directory locations for their own use.</p>
<div class="section" id="client-session-directories">
<h2><span class="section-number">6.1.1. </span>Client Session Directories<a class="headerlink" href="#client-session-directories" title="Permalink to this heading"></a></h2>
<p>The PMIx client library does not create its own session directories as it does not publish
contact information. Host daemons often create one or more session directory levels for use
by client application processes (e.g., for storing shared memory backing files) - the location
of those directories is passed to clients using the above attributes.</p>
<p>As the client library never creates session directories, it does not perform any cleanup of
the session directory tree.</p>
</div>
<div class="section" id="tool-and-server-session-directories">
<h2><span class="section-number">6.1.2. </span>Tool and Server Session Directories<a class="headerlink" href="#tool-and-server-session-directories" title="Permalink to this heading"></a></h2>
<p>The PMIx library utilizes appropriate session directory locations to store one or more
“rendezvous files” - i.e., files containing connection information. Note that the library does not
always create all levels of the session directory tree, although the process
itself can of course create directories as it sees fit. Only the directories that (a) are required
for generating the particular rendezvous file, and (b) do not already exist are created. Only
directories actually created by the library are cleaned up and removed upon finalization.</p>
<p>The following rendezvous files are provided:</p>
<ul>
<li><p>if <code class="docutils literal notranslate"><span class="pre">PMIX_LAUNCHER_RENDEZVOUS_FILE</span></code> is given (either via an attribute to an “init” function
or as an envar), then the specified file (including any required path elements) will be created.</p></li>
<li><p>if the server is designated as a “system” server (i.e., the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_SYSTEM_SUPPORT</span></code> attribute
was provided to <code class="docutils literal notranslate"><span class="pre">PMIx_server_init</span></code>), then a rendezvous file named “pmix.sys.<hostname>” will be
created in the <code class="docutils literal notranslate"><span class="pre">PMIX_SYSTEM_TMPDIR</span></code> location.</p></li>
<li><p>if the server is designated as a “session” server (i.e., the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_SESSION_SUPPORT</span></code> attribute
was provided to <code class="docutils literal notranslate"><span class="pre">PMIx_server_init</span></code>), then a rendezvous file named “pmix.sys.<hostname>” will be
created in the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code> location.</p></li>
<li><p>if the server declares that it will support tool connections (i.e., the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TOOL_SUPPORT</span></code>
attribute was provided to <code class="docutils literal notranslate"><span class="pre">PMIx_server_init</span></code>), then the following rendezvous files will be created
under the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_TMPDIR</span></code> location:</p>
<blockquote>
<div><ul class="simple">
<li><p>a PID file: “pmix.<hostname>.<pid>”</p></li>
<li><p>a namespace file using the nspace of the server: “pmix.<hostname>.<nspace>”</p></li>
</ul>
</div></blockquote>
</li>
<li><p>if the server is designated as a “scheduler” (i.e., the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_SCHEDULER</span></code> attribute
was provided to <code class="docutils literal notranslate"><span class="pre">PMIx_server_init</span></code>), then a rendezvous file named “pmix.sched.<hostname>” will be
created in the <code class="docutils literal notranslate"><span class="pre">PMIX_SYSTEM_TMPDIR</span></code> location.</p></li>
<li><p>if the server is designated as a “system controller” (i.e., the <code class="docutils literal notranslate"><span class="pre">PMIX_SERVER_SYS_CONTROLLER</span></code> attribute
was provided to <code class="docutils literal notranslate"><span class="pre">PMIx_server_init</span></code>), then a rendezvous file named “pmix.sysctrlr.<hostname>” will be
created in the <code class="docutils literal notranslate"><span class="pre">PMIX_SYSTEM_TMPDIR</span></code> location.</p>
<blockquote>
<div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>The above rendezvous files are additive - i.e., generating any one of the files has
no bearing on whether another file will be output. Thus, a single server could
generate anywhere from one to five (or more) rendezvous files spanning several
directory levels.</p>
</div>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>There is a potential conflict in rendezvous file names - e.g., if multiple processes
declare themselves to be a “session” server on the same node. The format of the
names used by PMIx are intended to support tool connection in the absence of specific
connection directives - i.e., they provide a means by which PMIx can search for and
find the rendezvous file for a particular type of process without requiring the user
to manually identify it. Thus, a tool can request connection to the “system controller”
without necessarily knowing the PID of that process and PMIx can facilitate the
connection. As a result, only one system server can be operating on a node at a time.
This is also (independently) true for schedulers and system controllers.</p>
</div>
</div></blockquote>
</li>
</ul>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="6. How Things Work" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="resolve.html" class="btn btn-neutral float-right" title="6.2. Resolving Peers and Nodes" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2014-2025, OpenPMIx Community.
<span class="lastupdated">Last updated on 2025-05-30 16:40:24 UTC.
</span></p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
|
