1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
|
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>How To Validate a New Release — LLVM 13 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/llvm-theme.css" type="text/css" />
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="How To Add Your Build Configuration To LLVM Buildbot Infrastructure" href="HowToAddABuilder.html" />
<link rel="prev" title="Advice on Packaging LLVM" href="Packaging.html" />
<style type="text/css">
table.right { float: right; margin-left: 20px; }
table.right td { border: 1px solid #ccc; }
</style>
</head><body>
<div class="logo">
<a href="index.html">
<img src="_static/logo.png"
alt="LLVM Logo" width="250" height="88"/></a>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="HowToAddABuilder.html" title="How To Add Your Build Configuration To LLVM Buildbot Infrastructure"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="Packaging.html" title="Advice on Packaging LLVM"
accesskey="P">previous</a> |</li>
<li><a href="https://llvm.org/">LLVM Home</a> | </li>
<li><a href="index.html">Documentation</a>»</li>
<li class="nav-item nav-item-1"><a href="GettingInvolved.html" accesskey="U">Getting Involved</a> »</li>
<li class="nav-item nav-item-this"><a href="">How To Validate a New Release</a></li>
</ul>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3>Documentation</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/GettingStartedTutorials.html">Getting Started/Tutorials</a></li>
<li><a href="https://llvm.org/docs/UserGuides.html">User Guides</a></li>
<li><a href="https://llvm.org/docs/Reference.html">Reference</a></li>
</ul>
<h3>Getting Involved</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/Contributing.html">Contributing to LLVM</a></li>
<li><a href="https://llvm.org/docs/HowToSubmitABug.html">Submitting Bug Reports</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#mailing-lists">Mailing Lists</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#irc">IRC</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#meetups-and-social-events">Meetups and Social Events</a></li>
</ul>
<h3>Additional Links</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/FAQ.html">FAQ</a></li>
<li><a href="https://llvm.org/docs/Lexicon.html">Glossary</a></li>
<li><a href="https://llvm.org/pubs">Publications</a></li>
<li><a href="https://github.com/llvm/llvm-project//">Github Repository</a></li>
</ul>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/ReleaseProcess.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="how-to-validate-a-new-release">
<h1>How To Validate a New Release<a class="headerlink" href="#how-to-validate-a-new-release" title="Permalink to this headline">¶</a></h1>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#introduction" id="id9">Introduction</a></p></li>
<li><p><a class="reference internal" href="#scripts" id="id10">Scripts</a></p></li>
<li><p><a class="reference internal" href="#test-suite" id="id11">Test Suite</a></p></li>
<li><p><a class="reference internal" href="#pre-release-process" id="id12">Pre-Release Process</a></p></li>
<li><p><a class="reference internal" href="#release-process" id="id13">Release Process</a></p></li>
<li><p><a class="reference internal" href="#bug-reporting-process" id="id14">Bug Reporting Process</a></p></li>
</ul>
</div>
<div class="section" id="introduction">
<h2><a class="toc-backref" href="#id9">Introduction</a><a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>This document contains information about testing the release candidates that
will ultimately be the next LLVM release. For more information on how to
manage the actual release, please refer to <a class="reference internal" href="HowToReleaseLLVM.html"><span class="doc">How To Release LLVM To The Public</span></a>.</p>
<div class="section" id="overview-of-the-release-process">
<h3>Overview of the Release Process<a class="headerlink" href="#overview-of-the-release-process" title="Permalink to this headline">¶</a></h3>
<p>Once the release process starts, the Release Manager will ask for volunteers,
and it’ll be the role of each volunteer to:</p>
<ul class="simple">
<li><p>Test and benchmark the previous release</p></li>
<li><p>Test and benchmark each release candidate, comparing to the previous release
and candidates</p></li>
<li><p>Identify, reduce and report every regression found during tests and benchmarks</p></li>
<li><p>Make sure the critical bugs get fixed and merged to the next release candidate</p></li>
</ul>
<p>Not all bugs or regressions are show-stoppers and it’s a bit of a grey area what
should be fixed before the next candidate and what can wait until the next
release.</p>
<p>It’ll depend on:</p>
<ul class="simple">
<li><p>The severity of the bug, how many people it affects and if it’s a regression
or a known bug. Known bugs are “unsupported features” and some bugs can be
disabled if they have been implemented recently.</p></li>
<li><p>The stage in the release. Less critical bugs should be considered to be
fixed between RC1 and RC2, but not so much at the end of it.</p></li>
<li><p>If it’s a correctness or a performance regression. Performance regression
tends to be taken more lightly than correctness.</p></li>
</ul>
</div>
</div>
<div class="section" id="scripts">
<span id="id1"></span><h2><a class="toc-backref" href="#id10">Scripts</a><a class="headerlink" href="#scripts" title="Permalink to this headline">¶</a></h2>
<p>The scripts are in the <code class="docutils literal notranslate"><span class="pre">utils/release</span></code> directory.</p>
<div class="section" id="test-release-sh">
<h3>test-release.sh<a class="headerlink" href="#test-release-sh" title="Permalink to this headline">¶</a></h3>
<p>This script will check-out, configure and compile LLVM+Clang (+ most add-ons,
like <code class="docutils literal notranslate"><span class="pre">compiler-rt</span></code>, <code class="docutils literal notranslate"><span class="pre">libcxx</span></code>, <code class="docutils literal notranslate"><span class="pre">libomp</span></code> and <code class="docutils literal notranslate"><span class="pre">clang-extra-tools</span></code>) in
three stages, and will test the final stage.
It’ll have installed the final binaries on the Phase3/Releasei(+Asserts)
directory, and that’s the one you should use for the test-suite and other
external tests.</p>
<p>To run the script on a specific release candidate run:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">test</span><span class="o">-</span><span class="n">release</span><span class="o">.</span><span class="n">sh</span> \
<span class="o">-</span><span class="n">release</span> <span class="mf">3.3</span> \
<span class="o">-</span><span class="n">rc</span> <span class="mi">1</span> \
<span class="o">-</span><span class="n">no</span><span class="o">-</span><span class="mi">64</span><span class="n">bit</span> \
<span class="o">-</span><span class="n">test</span><span class="o">-</span><span class="n">asserts</span> \
<span class="o">-</span><span class="n">no</span><span class="o">-</span><span class="n">compare</span><span class="o">-</span><span class="n">files</span>
</pre></div>
</div>
<p>Each system will require different options. For instance, x86_64 will
obviously not need <code class="docutils literal notranslate"><span class="pre">-no-64bit</span></code> while 32-bit systems will, or the script will
fail.</p>
<p>The important flags to get right are:</p>
<ul class="simple">
<li><p>On the pre-release, you should change <code class="docutils literal notranslate"><span class="pre">-rc</span> <span class="pre">1</span></code> to <code class="docutils literal notranslate"><span class="pre">-final</span></code>. On RC2,
change it to <code class="docutils literal notranslate"><span class="pre">-rc</span> <span class="pre">2</span></code> and so on.</p></li>
<li><p>On non-release testing, you can use <code class="docutils literal notranslate"><span class="pre">-final</span></code> in conjunction with
<code class="docutils literal notranslate"><span class="pre">-no-checkout</span></code>, but you’ll have to create the <code class="docutils literal notranslate"><span class="pre">final</span></code> directory by hand
and link the correct source dir to <code class="docutils literal notranslate"><span class="pre">final/llvm.src</span></code>.</p></li>
<li><p>For release candidates, you need <code class="docutils literal notranslate"><span class="pre">-test-asserts</span></code>, or it won’t create a
“Release+Asserts” directory, which is needed for release testing and
benchmarking. This will take twice as long.</p></li>
<li><p>On the final candidate you just need Release builds, and that’s the binary
directory you’ll have to pack.</p></li>
<li><p>On macOS, you must export <code class="docutils literal notranslate"><span class="pre">MACOSX_DEPLOYMENT_TARGET=10.9</span></code> before running
the script.</p></li>
</ul>
<p>This script builds three phases of Clang+LLVM twice each (Release and
Release+Asserts), so use screen or nohup to avoid headaches, since it’ll take
a long time.</p>
<p>Use the <code class="docutils literal notranslate"><span class="pre">--help</span></code> option to see all the options and chose it according to
your needs.</p>
</div>
<div class="section" id="findregressions-nightly-py">
<h3>findRegressions-nightly.py<a class="headerlink" href="#findregressions-nightly-py" title="Permalink to this headline">¶</a></h3>
<p>TODO</p>
</div>
</div>
<div class="section" id="test-suite">
<span id="id2"></span><h2><a class="toc-backref" href="#id11">Test Suite</a><a class="headerlink" href="#test-suite" title="Permalink to this headline">¶</a></h2>
<p>Follow the <a class="reference external" href="https://llvm.org/docs/lnt/quickstart.html">LNT Quick Start Guide</a> link on how to set-up the
test-suite</p>
<p>The binary location you’ll have to use for testing is inside the
<code class="docutils literal notranslate"><span class="pre">rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install</span></code>.
Link that directory to an easier location and run the test-suite.</p>
<p>An example on the run command line, assuming you created a link from the correct
install directory to <code class="docutils literal notranslate"><span class="pre">~/devel/llvm/install</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">sandbox</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">python</span> <span class="n">sandbox</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">lnt</span> <span class="n">runtest</span> \
<span class="n">nt</span> \
<span class="o">-</span><span class="n">j4</span> \
<span class="o">--</span><span class="n">sandbox</span> <span class="n">sandbox</span> \
<span class="o">--</span><span class="n">test</span><span class="o">-</span><span class="n">suite</span> <span class="o">~/</span><span class="n">devel</span><span class="o">/</span><span class="n">llvm</span><span class="o">/</span><span class="n">test</span><span class="o">/</span><span class="n">test</span><span class="o">-</span><span class="n">suite</span> \
<span class="o">--</span><span class="n">cc</span> <span class="o">~/</span><span class="n">devel</span><span class="o">/</span><span class="n">llvm</span><span class="o">/</span><span class="n">install</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">clang</span> \
<span class="o">--</span><span class="n">cxx</span> <span class="o">~/</span><span class="n">devel</span><span class="o">/</span><span class="n">llvm</span><span class="o">/</span><span class="n">install</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">clang</span><span class="o">++</span>
</pre></div>
</div>
<p>It should have no new regressions, compared to the previous release or release
candidate. You don’t need to fix all the bugs in the test-suite, since they’re
not necessarily meant to pass on all architectures all the time. This is
due to the nature of the result checking, which relies on direct comparison,
and most of the time, the failures are related to bad output checking, rather
than bad code generation.</p>
<p>If the errors are in LLVM itself, please report every single regression found
as blocker, and all the other bugs as important, but not necessarily blocking
the release to proceed. They can be set as “known failures” and to be
fix on a future date.</p>
</div>
<div class="section" id="pre-release-process">
<span id="id4"></span><h2><a class="toc-backref" href="#id12">Pre-Release Process</a><a class="headerlink" href="#pre-release-process" title="Permalink to this headline">¶</a></h2>
<p>When the release process is announced on the mailing list, you should prepare
for the testing, by applying the same testing you’ll do on the release
candidates, on the previous release.</p>
<p>You should:</p>
<ul class="simple">
<li><p>Download the previous release sources from
<a class="reference external" href="https://llvm.org/releases/download.html">https://llvm.org/releases/download.html</a>.</p></li>
<li><p>Run the test-release.sh script on <code class="docutils literal notranslate"><span class="pre">final</span></code> mode (change <code class="docutils literal notranslate"><span class="pre">-rc</span> <span class="pre">1</span></code> to
<code class="docutils literal notranslate"><span class="pre">-final</span></code>).</p></li>
<li><p>Once all three stages are done, it’ll test the final stage.</p></li>
<li><p>Using the <code class="docutils literal notranslate"><span class="pre">Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install</span></code> base,
run the test-suite.</p></li>
</ul>
<p>If the final phase’s <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">check-all</span></code> failed, it’s a good idea to also test
the intermediate stages by going on the obj directory and running
<code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">check-all</span></code> to find if there’s at least one stage that passes (helps
when reducing the error for bug report purposes).</p>
</div>
<div class="section" id="release-process">
<span id="id6"></span><h2><a class="toc-backref" href="#id13">Release Process</a><a class="headerlink" href="#release-process" title="Permalink to this headline">¶</a></h2>
<p>When the Release Manager sends you the release candidate, download all sources,
unzip on the same directory (there will be sym-links from the appropriate places
to them), and run the release test as above.</p>
<p>You should:</p>
<ul class="simple">
<li><p>Download the current candidate sources from where the release manager points
you (ex. <a class="reference external" href="https://llvm.org/pre-releases/3.3/rc1/">https://llvm.org/pre-releases/3.3/rc1/</a>).</p></li>
<li><p>Repeat the steps above with <code class="docutils literal notranslate"><span class="pre">-rc</span> <span class="pre">1</span></code>, <code class="docutils literal notranslate"><span class="pre">-rc</span> <span class="pre">2</span></code> etc modes and run the
test-suite the same way.</p></li>
<li><p>Compare the results, report all errors on Bugzilla and publish the binary blob
where the release manager can grab it.</p></li>
</ul>
<p>Once the release manages announces that the latest candidate is the good one,
you have to pack the <code class="docutils literal notranslate"><span class="pre">Release</span></code> (no Asserts) install directory on <code class="docutils literal notranslate"><span class="pre">Phase3</span></code>
and that will be the official binary.</p>
<ul class="simple">
<li><p>Rename (or link) <code class="docutils literal notranslate"><span class="pre">clang+llvm-REL-ARCH-ENV</span></code> to the .install directory</p></li>
<li><p>Tar that into the same name with <code class="docutils literal notranslate"><span class="pre">.tar.gz</span></code> extension from outside the
directory</p></li>
<li><p>Make it available for the release manager to download</p></li>
</ul>
</div>
<div class="section" id="bug-reporting-process">
<span id="bug-reporting"></span><h2><a class="toc-backref" href="#id14">Bug Reporting Process</a><a class="headerlink" href="#bug-reporting-process" title="Permalink to this headline">¶</a></h2>
<p>If you found regressions or failures when comparing a release candidate with the
previous release, follow the rules below:</p>
<ul class="simple">
<li><p>Critical bugs on compilation should be fixed as soon as possible, possibly
before releasing the binary blobs.</p></li>
<li><p>Check-all tests should be fixed before the next release candidate, but can
wait until the test-suite run is finished.</p></li>
<li><p>Bugs in the test suite or unimportant check-all tests can be fixed in between
release candidates.</p></li>
<li><p>New features or recent big changes, when close to the release, should have
done in a way that it’s easy to disable. If they misbehave, prefer disabling
them than releasing an unstable (but untested) binary package.</p></li>
</ul>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="HowToAddABuilder.html" title="How To Add Your Build Configuration To LLVM Buildbot Infrastructure"
>next</a> |</li>
<li class="right" >
<a href="Packaging.html" title="Advice on Packaging LLVM"
>previous</a> |</li>
<li><a href="https://llvm.org/">LLVM Home</a> | </li>
<li><a href="index.html">Documentation</a>»</li>
<li class="nav-item nav-item-1"><a href="GettingInvolved.html" >Getting Involved</a> »</li>
<li class="nav-item nav-item-this"><a href="">How To Validate a New Release</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2003-2021, LLVM Project.
Last updated on 2021-09-18.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.5.4.
</div>
</body>
</html>
|
