1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
|
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>lit - LLVM Integrated Tester — LLVM 13 documentation</title>
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/llvm-theme.css" type="text/css" />
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="llvm-exegesis - LLVM Machine Instruction Benchmark" href="llvm-exegesis.html" />
<link rel="prev" title="mlir-tblgen - Description to C++ Code for MLIR" href="mlir-tblgen.html" />
<style type="text/css">
table.right { float: right; margin-left: 20px; }
table.right td { border: 1px solid #ccc; }
</style>
</head><body>
<div class="logo">
<a href="../index.html">
<img src="../_static/logo.png"
alt="LLVM Logo" width="250" height="88"/></a>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="llvm-exegesis.html" title="llvm-exegesis - LLVM Machine Instruction Benchmark"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="mlir-tblgen.html" title="mlir-tblgen - Description to C++ Code for MLIR"
accesskey="P">previous</a> |</li>
<li><a href="https://llvm.org/">LLVM Home</a> | </li>
<li><a href="../index.html">Documentation</a>»</li>
<li class="nav-item nav-item-1"><a href="../Reference.html" >Reference</a> »</li>
<li class="nav-item nav-item-2"><a href="index.html" accesskey="U">LLVM Command Guide</a> »</li>
<li class="nav-item nav-item-this"><a href="">lit - LLVM Integrated Tester</a></li>
</ul>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3>Documentation</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/GettingStartedTutorials.html">Getting Started/Tutorials</a></li>
<li><a href="https://llvm.org/docs/UserGuides.html">User Guides</a></li>
<li><a href="https://llvm.org/docs/Reference.html">Reference</a></li>
</ul>
<h3>Getting Involved</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/Contributing.html">Contributing to LLVM</a></li>
<li><a href="https://llvm.org/docs/HowToSubmitABug.html">Submitting Bug Reports</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#mailing-lists">Mailing Lists</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#irc">IRC</a></li>
<li><a href="https://llvm.org/docs/GettingInvolved.html#meetups-and-social-events">Meetups and Social Events</a></li>
</ul>
<h3>Additional Links</h3>
<ul class="want-points">
<li><a href="https://llvm.org/docs/FAQ.html">FAQ</a></li>
<li><a href="https://llvm.org/docs/Lexicon.html">Glossary</a></li>
<li><a href="https://llvm.org/pubs">Publications</a></li>
<li><a href="https://github.com/llvm/llvm-project//">Github Repository</a></li>
</ul>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/CommandGuide/lit.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="lit-llvm-integrated-tester">
<h1>lit - LLVM Integrated Tester<a class="headerlink" href="#lit-llvm-integrated-tester" title="Permalink to this headline">¶</a></h1>
<div class="section" id="synopsis">
<h2>SYNOPSIS<a class="headerlink" href="#synopsis" title="Permalink to this headline">¶</a></h2>
<p><strong class="program">lit</strong> [<em>options</em>] [<em>tests</em>]</p>
</div>
<div class="section" id="description">
<h2>DESCRIPTION<a class="headerlink" href="#description" title="Permalink to this headline">¶</a></h2>
<p><strong class="program">lit</strong> is a portable tool for executing LLVM and Clang style test
suites, summarizing their results, and providing indication of failures.
<strong class="program">lit</strong> is designed to be a lightweight testing tool with as simple a
user interface as possible.</p>
<p><strong class="program">lit</strong> should be run with one or more <em>tests</em> to run specified on the
command line. Tests can be either individual test files or directories to
search for tests (see <a class="reference internal" href="#test-discovery"><span class="std std-ref">TEST DISCOVERY</span></a>).</p>
<p>Each specified test will be executed (potentially concurrently) and once all
tests have been run <strong class="program">lit</strong> will print summary information on the number
of tests which passed or failed (see <a class="reference internal" href="#test-status-results"><span class="std std-ref">TEST STATUS RESULTS</span></a>). The
<strong class="program">lit</strong> program will execute with a non-zero exit code if any tests
fail.</p>
<p>By default <strong class="program">lit</strong> will use a succinct progress display and will only
print summary information for test failures. See <a class="reference internal" href="#output-options"><span class="std std-ref">OUTPUT OPTIONS</span></a> for
options controlling the <strong class="program">lit</strong> progress display and output.</p>
<p><strong class="program">lit</strong> also includes a number of options for controlling how tests are
executed (specific features may depend on the particular test format). See
<a class="reference internal" href="#execution-options"><span class="std std-ref">EXECUTION OPTIONS</span></a> for more information.</p>
<p>Finally, <strong class="program">lit</strong> also supports additional options for only running a
subset of the options specified on the command line, see
<a class="reference internal" href="#selection-options"><span class="std std-ref">SELECTION OPTIONS</span></a> for more information.</p>
<p><strong class="program">lit</strong> parses options from the environment variable <code class="docutils literal notranslate"><span class="pre">LIT_OPTS</span></code> after
parsing options from the command line. <code class="docutils literal notranslate"><span class="pre">LIT_OPTS</span></code> is primarily useful for
supplementing or overriding the command-line options supplied to <strong class="program">lit</strong>
by <code class="docutils literal notranslate"><span class="pre">check</span></code> targets defined by a project’s build system.</p>
<p>Users interested in the <strong class="program">lit</strong> architecture or designing a
<strong class="program">lit</strong> testing implementation should see <a class="reference internal" href="#lit-infrastructure"><span class="std std-ref">LIT INFRASTRUCTURE</span></a>.</p>
</div>
<div class="section" id="general-options">
<h2>GENERAL OPTIONS<a class="headerlink" href="#general-options" title="Permalink to this headline">¶</a></h2>
<dl class="std option">
<dt id="cmdoption-lit-h">
<span id="cmdoption-lit-help"></span><code class="sig-name descname"><span class="pre">-h</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--help</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-h" title="Permalink to this definition">¶</a></dt>
<dd><p>Show the <strong class="program">lit</strong> help message.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-j">
<span id="cmdoption-lit-workers"></span><code class="sig-name descname"><span class="pre">-j</span></code><code class="sig-prename descclassname"> <span class="pre">N</span></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--workers</span></code><code class="sig-prename descclassname"><span class="pre">=N</span></code><a class="headerlink" href="#cmdoption-lit-j" title="Permalink to this definition">¶</a></dt>
<dd><p>Run <code class="docutils literal notranslate"><span class="pre">N</span></code> tests in parallel. By default, this is automatically chosen to
match the number of detected available CPUs.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-config-prefix">
<code class="sig-name descname"><span class="pre">--config-prefix</span></code><code class="sig-prename descclassname"><span class="pre">=NAME</span></code><a class="headerlink" href="#cmdoption-lit-config-prefix" title="Permalink to this definition">¶</a></dt>
<dd><p>Search for <code class="file docutils literal notranslate"><em><span class="pre">NAME</span></em><span class="pre">.cfg</span></code> and <code class="file docutils literal notranslate"><em><span class="pre">NAME</span></em><span class="pre">.site.cfg</span></code> when searching for
test suites, instead of <code class="file docutils literal notranslate"><span class="pre">lit.cfg</span></code> and <code class="file docutils literal notranslate"><span class="pre">lit.site.cfg</span></code>.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-D">
<span id="cmdoption-lit-d"></span><span id="cmdoption-lit-param"></span><code class="sig-name descname"><span class="pre">-D</span></code><code class="sig-prename descclassname"> <span class="pre">NAME[=VALUE]</span></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--param</span></code><code class="sig-prename descclassname"> <span class="pre">NAME[=VALUE]</span></code><a class="headerlink" href="#cmdoption-lit-D" title="Permalink to this definition">¶</a></dt>
<dd><p>Add a user defined parameter <code class="docutils literal notranslate"><span class="pre">NAME</span></code> with the given <code class="docutils literal notranslate"><span class="pre">VALUE</span></code> (or the empty
string if not given). The meaning and use of these parameters is test suite
dependent.</p>
</dd></dl>
</div>
<div class="section" id="output-options">
<span id="id1"></span><h2>OUTPUT OPTIONS<a class="headerlink" href="#output-options" title="Permalink to this headline">¶</a></h2>
<dl class="std option">
<dt id="cmdoption-lit-q">
<span id="cmdoption-lit-quiet"></span><code class="sig-name descname"><span class="pre">-q</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--quiet</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-q" title="Permalink to this definition">¶</a></dt>
<dd><p>Suppress any output except for test failures.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-s">
<span id="cmdoption-lit-succinct"></span><code class="sig-name descname"><span class="pre">-s</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--succinct</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-s" title="Permalink to this definition">¶</a></dt>
<dd><p>Show less output, for example don’t show information on tests that pass.
Also show a progress bar, unless <code class="docutils literal notranslate"><span class="pre">--no-progress-bar</span></code> is specified.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-v">
<span id="cmdoption-lit-verbose"></span><code class="sig-name descname"><span class="pre">-v</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--verbose</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-v" title="Permalink to this definition">¶</a></dt>
<dd><p>Show more information on test failures, for example the entire test output
instead of just the test result.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-vv">
<span id="cmdoption-lit-echo-all-commands"></span><code class="sig-name descname"><span class="pre">-vv</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--echo-all-commands</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-vv" title="Permalink to this definition">¶</a></dt>
<dd><p>Echo all commands to stdout, as they are being executed.
This can be valuable for debugging test failures, as the last echoed command
will be the one which has failed.
<strong class="program">lit</strong> normally inserts a no-op command (<code class="docutils literal notranslate"><span class="pre">:</span></code> in the case of bash)
with argument <code class="docutils literal notranslate"><span class="pre">'RUN:</span> <span class="pre">at</span> <span class="pre">line</span> <span class="pre">N'</span></code> before each command pipeline, and this
option also causes those no-op commands to be echoed to stdout to help you
locate the source line of the failed command.
This option implies <code class="docutils literal notranslate"><span class="pre">--verbose</span></code>.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-a">
<span id="cmdoption-lit-show-all"></span><code class="sig-name descname"><span class="pre">-a</span></code><code class="sig-prename descclassname"></code><code class="sig-prename descclassname"><span class="pre">,</span> </code><code class="sig-name descname"><span class="pre">--show-all</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-a" title="Permalink to this definition">¶</a></dt>
<dd><p>Show more information about all tests, for example the entire test
commandline and output.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-no-progress-bar">
<code class="sig-name descname"><span class="pre">--no-progress-bar</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-no-progress-bar" title="Permalink to this definition">¶</a></dt>
<dd><p>Do not use curses based progress bar.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-show-unsupported">
<code class="sig-name descname"><span class="pre">--show-unsupported</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-show-unsupported" title="Permalink to this definition">¶</a></dt>
<dd><p>Show the names of unsupported tests.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-show-xfail">
<code class="sig-name descname"><span class="pre">--show-xfail</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-show-xfail" title="Permalink to this definition">¶</a></dt>
<dd><p>Show the names of tests that were expected to fail.</p>
</dd></dl>
</div>
<div class="section" id="execution-options">
<span id="id2"></span><h2>EXECUTION OPTIONS<a class="headerlink" href="#execution-options" title="Permalink to this headline">¶</a></h2>
<dl class="std option">
<dt id="cmdoption-lit-path">
<code class="sig-name descname"><span class="pre">--path</span></code><code class="sig-prename descclassname"><span class="pre">=PATH</span></code><a class="headerlink" href="#cmdoption-lit-path" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify an additional <code class="docutils literal notranslate"><span class="pre">PATH</span></code> to use when searching for executables in tests.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-vg">
<code class="sig-name descname"><span class="pre">--vg</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-vg" title="Permalink to this definition">¶</a></dt>
<dd><p>Run individual tests under valgrind (using the memcheck tool). The
<code class="docutils literal notranslate"><span class="pre">--error-exitcode</span></code> argument for valgrind is used so that valgrind failures
will cause the program to exit with a non-zero status.</p>
<p>When this option is enabled, <strong class="program">lit</strong> will also automatically provide a
“<code class="docutils literal notranslate"><span class="pre">valgrind</span></code>” feature that can be used to conditionally disable (or expect
failure in) certain tests.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-vg-arg">
<code class="sig-name descname"><span class="pre">--vg-arg</span></code><code class="sig-prename descclassname"><span class="pre">=ARG</span></code><a class="headerlink" href="#cmdoption-lit-vg-arg" title="Permalink to this definition">¶</a></dt>
<dd><p>When <a class="reference internal" href="#cmdoption-lit-vg"><code class="xref std std-option docutils literal notranslate"><span class="pre">--vg</span></code></a> is used, specify an additional argument to pass to
<strong class="program">valgrind</strong> itself.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-vg-leak">
<code class="sig-name descname"><span class="pre">--vg-leak</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-vg-leak" title="Permalink to this definition">¶</a></dt>
<dd><p>When <a class="reference internal" href="#cmdoption-lit-vg"><code class="xref std std-option docutils literal notranslate"><span class="pre">--vg</span></code></a> is used, enable memory leak checks. When this option is
enabled, <strong class="program">lit</strong> will also automatically provide a “<code class="docutils literal notranslate"><span class="pre">vg_leak</span></code>”
feature that can be used to conditionally disable (or expect failure in)
certain tests.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-time-tests">
<code class="sig-name descname"><span class="pre">--time-tests</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-time-tests" title="Permalink to this definition">¶</a></dt>
<dd><p>Track the wall time individual tests take to execute and includes the results
in the summary output. This is useful for determining which tests in a test
suite take the most time to execute.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-ignore-fail">
<code class="sig-name descname"><span class="pre">--ignore-fail</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-ignore-fail" title="Permalink to this definition">¶</a></dt>
<dd><p>Exit with status zero even if some tests fail.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-no-indirectly-run-check">
<code class="sig-name descname"><span class="pre">--no-indirectly-run-check</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-no-indirectly-run-check" title="Permalink to this definition">¶</a></dt>
<dd><p>Do not error if a test would not be run if the user had specified the
containing directory instead of naming the test directly.</p>
</dd></dl>
</div>
<div class="section" id="selection-options">
<span id="id3"></span><h2>SELECTION OPTIONS<a class="headerlink" href="#selection-options" title="Permalink to this headline">¶</a></h2>
<p>By default, <cite>lit</cite> will run failing tests first, then run tests in descending
execution time order to optimize concurrency. The execution order can be
changed using the <a class="reference internal" href="#cmdoption-lit-order"><code class="xref std std-option docutils literal notranslate"><span class="pre">--order</span></code></a> option.</p>
<p>The timing data is stored in the <cite>test_exec_root</cite> in a file named
<cite>.lit_test_times.txt</cite>. If this file does not exist, then <cite>lit</cite> checks the
<cite>test_source_root</cite> for the file to optionally accelerate clean builds.</p>
<dl class="std option">
<dt id="cmdoption-lit-shuffle">
<code class="sig-name descname"><span class="pre">--shuffle</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-shuffle" title="Permalink to this definition">¶</a></dt>
<dd><p>Run the tests in a random order, not failing/slowest first. Deprecated,
use <a class="reference internal" href="#cmdoption-lit-order"><code class="xref std std-option docutils literal notranslate"><span class="pre">--order</span></code></a> instead.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-max-failures">
<code class="sig-name descname"><span class="pre">--max-failures</span></code><code class="sig-prename descclassname"> <span class="pre">N</span></code><a class="headerlink" href="#cmdoption-lit-max-failures" title="Permalink to this definition">¶</a></dt>
<dd><p>Stop execution after the given number <code class="docutils literal notranslate"><span class="pre">N</span></code> of failures.
An integer argument should be passed on the command line
prior to execution.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-max-tests">
<code class="sig-name descname"><span class="pre">--max-tests</span></code><code class="sig-prename descclassname"><span class="pre">=N</span></code><a class="headerlink" href="#cmdoption-lit-max-tests" title="Permalink to this definition">¶</a></dt>
<dd><p>Run at most <code class="docutils literal notranslate"><span class="pre">N</span></code> tests and then terminate.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-max-time">
<code class="sig-name descname"><span class="pre">--max-time</span></code><code class="sig-prename descclassname"><span class="pre">=N</span></code><a class="headerlink" href="#cmdoption-lit-max-time" title="Permalink to this definition">¶</a></dt>
<dd><p>Spend at most <code class="docutils literal notranslate"><span class="pre">N</span></code> seconds (approximately) running tests and then terminate.
Note that this is not an alias for <a class="reference internal" href="#cmdoption-lit-timeout"><code class="xref std std-option docutils literal notranslate"><span class="pre">--timeout</span></code></a>; the two are
different kinds of maximums.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-num-shards">
<code class="sig-name descname"><span class="pre">--num-shards</span></code><code class="sig-prename descclassname"><span class="pre">=M</span></code><a class="headerlink" href="#cmdoption-lit-num-shards" title="Permalink to this definition">¶</a></dt>
<dd><p>Divide the set of selected tests into <code class="docutils literal notranslate"><span class="pre">M</span></code> equal-sized subsets or
“shards”, and run only one of them. Must be used with the
<code class="docutils literal notranslate"><span class="pre">--run-shard=N</span></code> option, which selects the shard to run. The environment
variable <code class="docutils literal notranslate"><span class="pre">LIT_NUM_SHARDS</span></code> can also be used in place of this
option. These two options provide a coarse mechanism for partitioning large
testsuites, for parallel execution on separate machines (say in a large
testing farm).</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-order">
<code class="sig-name descname"><span class="pre">--order</span></code><code class="sig-prename descclassname"><span class="pre">={lexical,random,smart}</span></code><a class="headerlink" href="#cmdoption-lit-order" title="Permalink to this definition">¶</a></dt>
<dd><p>Define the order in which tests are run. The supported values are:</p>
<ul class="simple">
<li><p>lexical - tests will be run in lexical order according to the test file
path. This option is useful when predictable test order is desired.</p></li>
<li><p>random - tests will be run in random order.</p></li>
<li><p>smart - tests that failed previously will be run first, then the remaining
tests, all in descending execution time order. This is the default as it
optimizes concurrency.</p></li>
</ul>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-run-shard">
<code class="sig-name descname"><span class="pre">--run-shard</span></code><code class="sig-prename descclassname"><span class="pre">=N</span></code><a class="headerlink" href="#cmdoption-lit-run-shard" title="Permalink to this definition">¶</a></dt>
<dd><p>Select which shard to run, assuming the <code class="docutils literal notranslate"><span class="pre">--num-shards=M</span></code> option was
provided. The two options must be used together, and the value of <code class="docutils literal notranslate"><span class="pre">N</span></code>
must be in the range <code class="docutils literal notranslate"><span class="pre">1..M</span></code>. The environment variable
<code class="docutils literal notranslate"><span class="pre">LIT_RUN_SHARD</span></code> can also be used in place of this option.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-timeout">
<code class="sig-name descname"><span class="pre">--timeout</span></code><code class="sig-prename descclassname"><span class="pre">=N</span></code><a class="headerlink" href="#cmdoption-lit-timeout" title="Permalink to this definition">¶</a></dt>
<dd><p>Spend at most <code class="docutils literal notranslate"><span class="pre">N</span></code> seconds (approximately) running each individual test.
<code class="docutils literal notranslate"><span class="pre">0</span></code> means no time limit, and <code class="docutils literal notranslate"><span class="pre">0</span></code> is the default. Note that this is not an
alias for <a class="reference internal" href="#cmdoption-lit-max-time"><code class="xref std std-option docutils literal notranslate"><span class="pre">--max-time</span></code></a>; the two are different kinds of maximums.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-filter">
<code class="sig-name descname"><span class="pre">--filter</span></code><code class="sig-prename descclassname"><span class="pre">=REGEXP</span></code><a class="headerlink" href="#cmdoption-lit-filter" title="Permalink to this definition">¶</a></dt>
<dd><p>Run only those tests whose name matches the regular expression specified in
<code class="docutils literal notranslate"><span class="pre">REGEXP</span></code>. The environment variable <code class="docutils literal notranslate"><span class="pre">LIT_FILTER</span></code> can be also used in place
of this option, which is especially useful in environments where the call
to <code class="docutils literal notranslate"><span class="pre">lit</span></code> is issued indirectly.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-filter-out">
<code class="sig-name descname"><span class="pre">--filter-out</span></code><code class="sig-prename descclassname"><span class="pre">=REGEXP</span></code><a class="headerlink" href="#cmdoption-lit-filter-out" title="Permalink to this definition">¶</a></dt>
<dd><p>Filter out those tests whose name matches the regular expression specified in
<code class="docutils literal notranslate"><span class="pre">REGEXP</span></code>. The environment variable <code class="docutils literal notranslate"><span class="pre">LIT_FILTER_OUT</span></code> can be also used in
place of this option, which is especially useful in environments where the
call to <code class="docutils literal notranslate"><span class="pre">lit</span></code> is issued indirectly.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-xfail">
<code class="sig-name descname"><span class="pre">--xfail</span></code><code class="sig-prename descclassname"><span class="pre">=LIST</span></code><a class="headerlink" href="#cmdoption-lit-xfail" title="Permalink to this definition">¶</a></dt>
<dd><p>Treat those tests whose name is in the semicolon separated list <code class="docutils literal notranslate"><span class="pre">LIST</span></code> as
<code class="docutils literal notranslate"><span class="pre">XFAIL</span></code>. This can be helpful when one does not want to modify the test
suite. The environment variable <code class="docutils literal notranslate"><span class="pre">LIT_XFAIL</span></code> can be also used in place of
this option, which is especially useful in environments where the call to
<code class="docutils literal notranslate"><span class="pre">lit</span></code> is issued indirectly.</p>
<p>A test name can specified as a file name relative to the test suite directory.
For example:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
</pre></div>
</div>
<p>In this case, all of the following tests are treated as <code class="docutils literal notranslate"><span class="pre">XFAIL</span></code>:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>libomp :: affinity/kmp-hw-subset.c
libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
</pre></div>
</div>
<p>Alternatively, a test name can be specified as the full test name
reported in LIT output. For example, we can adjust the previous
example not to treat the <code class="docutils literal notranslate"><span class="pre">nvptx64-nvidia-cuda</span></code> version of
<code class="docutils literal notranslate"><span class="pre">offloading/memory_manager.cpp</span></code> as XFAIL:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
</pre></div>
</div>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-xfail-not">
<code class="sig-name descname"><span class="pre">--xfail-not</span></code><code class="sig-prename descclassname"><span class="pre">=LIST</span></code><a class="headerlink" href="#cmdoption-lit-xfail-not" title="Permalink to this definition">¶</a></dt>
<dd><p>Do not treat the specified tests as <code class="docutils literal notranslate"><span class="pre">XFAIL</span></code>. The environment variable
<code class="docutils literal notranslate"><span class="pre">LIT_XFAIL_NOT</span></code> can also be used in place of this option. The syntax is the
same as for <a class="reference internal" href="#cmdoption-lit-xfail"><code class="xref std std-option docutils literal notranslate"><span class="pre">--xfail</span></code></a> and <code class="docutils literal notranslate"><span class="pre">LIT_XFAIL</span></code>. <a class="reference internal" href="#cmdoption-lit-xfail-not"><code class="xref std std-option docutils literal notranslate"><span class="pre">--xfail-not</span></code></a> and
<code class="docutils literal notranslate"><span class="pre">LIT_XFAIL_NOT</span></code> always override all other <code class="docutils literal notranslate"><span class="pre">XFAIL</span></code> specifications,
including an <a class="reference internal" href="#cmdoption-lit-xfail"><code class="xref std std-option docutils literal notranslate"><span class="pre">--xfail</span></code></a> appearing later on the command line. The
primary purpose is to suppress an <code class="docutils literal notranslate"><span class="pre">XPASS</span></code> result without modifying a test
case that uses the <code class="docutils literal notranslate"><span class="pre">XFAIL</span></code> directive.</p>
</dd></dl>
</div>
<div class="section" id="additional-options">
<h2>ADDITIONAL OPTIONS<a class="headerlink" href="#additional-options" title="Permalink to this headline">¶</a></h2>
<dl class="std option">
<dt id="cmdoption-lit-debug">
<code class="sig-name descname"><span class="pre">--debug</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-debug" title="Permalink to this definition">¶</a></dt>
<dd><p>Run <strong class="program">lit</strong> in debug mode, for debugging configuration issues and
<strong class="program">lit</strong> itself.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-show-suites">
<code class="sig-name descname"><span class="pre">--show-suites</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-show-suites" title="Permalink to this definition">¶</a></dt>
<dd><p>List the discovered test suites and exit.</p>
</dd></dl>
<dl class="std option">
<dt id="cmdoption-lit-show-tests">
<code class="sig-name descname"><span class="pre">--show-tests</span></code><code class="sig-prename descclassname"></code><a class="headerlink" href="#cmdoption-lit-show-tests" title="Permalink to this definition">¶</a></dt>
<dd><p>List all of the discovered tests and exit.</p>
</dd></dl>
</div>
<div class="section" id="exit-status">
<h2>EXIT STATUS<a class="headerlink" href="#exit-status" title="Permalink to this headline">¶</a></h2>
<p><strong class="program">lit</strong> will exit with an exit code of 1 if there are any FAIL or XPASS
results. Otherwise, it will exit with the status 0. Other exit codes are used
for non-test related failures (for example a user error or an internal program
error).</p>
</div>
<div class="section" id="test-discovery">
<span id="id4"></span><h2>TEST DISCOVERY<a class="headerlink" href="#test-discovery" title="Permalink to this headline">¶</a></h2>
<p>The inputs passed to <strong class="program">lit</strong> can be either individual tests, or entire
directories or hierarchies of tests to run. When <strong class="program">lit</strong> starts up, the
first thing it does is convert the inputs into a complete list of tests to run
as part of <em>test discovery</em>.</p>
<p>In the <strong class="program">lit</strong> model, every test must exist inside some <em>test suite</em>.
<strong class="program">lit</strong> resolves the inputs specified on the command line to test suites
by searching upwards from the input path until it finds a <code class="file docutils literal notranslate"><span class="pre">lit.cfg</span></code> or
<code class="file docutils literal notranslate"><span class="pre">lit.site.cfg</span></code> file. These files serve as both a marker of test suites
and as configuration files which <strong class="program">lit</strong> loads in order to understand
how to find and run the tests inside the test suite.</p>
<p>Once <strong class="program">lit</strong> has mapped the inputs into test suites it traverses the
list of inputs adding tests for individual files and recursively searching for
tests in directories.</p>
<p>This behavior makes it easy to specify a subset of tests to run, while still
allowing the test suite configuration to control exactly how tests are
interpreted. In addition, <strong class="program">lit</strong> always identifies tests by the test
suite they are in, and their relative path inside the test suite. For
appropriately configured projects, this allows <strong class="program">lit</strong> to provide
convenient and flexible support for out-of-tree builds.</p>
</div>
<div class="section" id="test-status-results">
<span id="id5"></span><h2>TEST STATUS RESULTS<a class="headerlink" href="#test-status-results" title="Permalink to this headline">¶</a></h2>
<p>Each test ultimately produces one of the following eight results:</p>
<p><strong>PASS</strong></p>
<blockquote>
<div><p>The test succeeded.</p>
</div></blockquote>
<p><strong>FLAKYPASS</strong></p>
<blockquote>
<div><p>The test succeeded after being re-run more than once. This only applies to
tests containing an <code class="docutils literal notranslate"><span class="pre">ALLOW_RETRIES:</span></code> annotation.</p>
</div></blockquote>
<p><strong>XFAIL</strong></p>
<blockquote>
<div><p>The test failed, but that is expected. This is used for test formats which allow
specifying that a test does not currently work, but wish to leave it in the test
suite.</p>
</div></blockquote>
<p><strong>XPASS</strong></p>
<blockquote>
<div><p>The test succeeded, but it was expected to fail. This is used for tests which
were specified as expected to fail, but are now succeeding (generally because
the feature they test was broken and has been fixed).</p>
</div></blockquote>
<p><strong>FAIL</strong></p>
<blockquote>
<div><p>The test failed.</p>
</div></blockquote>
<p><strong>UNRESOLVED</strong></p>
<blockquote>
<div><p>The test result could not be determined. For example, this occurs when the test
could not be run, the test itself is invalid, or the test was interrupted.</p>
</div></blockquote>
<p><strong>UNSUPPORTED</strong></p>
<blockquote>
<div><p>The test is not supported in this environment. This is used by test formats
which can report unsupported tests.</p>
</div></blockquote>
<p><strong>TIMEOUT</strong></p>
<blockquote>
<div><p>The test was run, but it timed out before it was able to complete. This is
considered a failure.</p>
</div></blockquote>
<p>Depending on the test format tests may produce additional information about
their status (generally only for failures). See the <a class="reference internal" href="#output-options"><span class="std std-ref">OUTPUT OPTIONS</span></a>
section for more information.</p>
</div>
<div class="section" id="lit-infrastructure">
<span id="id6"></span><h2>LIT INFRASTRUCTURE<a class="headerlink" href="#lit-infrastructure" title="Permalink to this headline">¶</a></h2>
<p>This section describes the <strong class="program">lit</strong> testing architecture for users interested in
creating a new <strong class="program">lit</strong> testing implementation, or extending an existing one.</p>
<p><strong class="program">lit</strong> proper is primarily an infrastructure for discovering and running
arbitrary tests, and to expose a single convenient interface to these
tests. <strong class="program">lit</strong> itself doesn’t know how to run tests, rather this logic is
defined by <em>test suites</em>.</p>
<div class="section" id="test-suites">
<h3>TEST SUITES<a class="headerlink" href="#test-suites" title="Permalink to this headline">¶</a></h3>
<p>As described in <a class="reference internal" href="#test-discovery"><span class="std std-ref">TEST DISCOVERY</span></a>, tests are always located inside a <em>test
suite</em>. Test suites serve to define the format of the tests they contain, the
logic for finding those tests, and any additional information to run the tests.</p>
<p><strong class="program">lit</strong> identifies test suites as directories containing <code class="docutils literal notranslate"><span class="pre">lit.cfg</span></code> or
<code class="docutils literal notranslate"><span class="pre">lit.site.cfg</span></code> files (see also <a class="reference internal" href="#cmdoption-lit-config-prefix"><code class="xref std std-option docutils literal notranslate"><span class="pre">--config-prefix</span></code></a>). Test suites are
initially discovered by recursively searching up the directory hierarchy for
all the input files passed on the command line. You can use
<a class="reference internal" href="#cmdoption-lit-show-suites"><code class="xref std std-option docutils literal notranslate"><span class="pre">--show-suites</span></code></a> to display the discovered test suites at startup.</p>
<p>Once a test suite is discovered, its config file is loaded. Config files
themselves are Python modules which will be executed. When the config file is
executed, two important global variables are predefined:</p>
<p><strong>lit_config</strong></p>
<blockquote>
<div><p>The global <strong>lit</strong> configuration object (a <em>LitConfig</em> instance), which defines
the builtin test formats, global configuration parameters, and other helper
routines for implementing test configurations.</p>
</div></blockquote>
<p><strong>config</strong></p>
<blockquote>
<div><p>This is the config object (a <em>TestingConfig</em> instance) for the test suite,
which the config file is expected to populate. The following variables are also
available on the <em>config</em> object, some of which must be set by the config and
others are optional or predefined:</p>
<p><strong>name</strong> <em>[required]</em> The name of the test suite, for use in reports and
diagnostics.</p>
<p><strong>test_format</strong> <em>[required]</em> The test format object which will be used to
discover and run tests in the test suite. Generally this will be a builtin test
format available from the <em>lit.formats</em> module.</p>
<p><strong>test_source_root</strong> The filesystem path to the test suite root. For out-of-dir
builds this is the directory that will be scanned for tests.</p>
<p><strong>test_exec_root</strong> For out-of-dir builds, the path to the test suite root inside
the object directory. This is where tests will be run and temporary output files
placed.</p>
<p><strong>environment</strong> A dictionary representing the environment to use when executing
tests in the suite.</p>
<p><strong>standalone_tests</strong> When true, mark a directory with tests expected to be run
standalone. Test discovery is disabled for that directory and
<em>–no-indirectly-run-check</em> is in effect. <em>lit.suffixes</em> and <em>lit.excludes</em>
must be empty when this variable is true.</p>
<p><strong>suffixes</strong> For <strong>lit</strong> test formats which scan directories for tests, this
variable is a list of suffixes to identify test files. Used by: <em>ShTest</em>.</p>
<p><strong>substitutions</strong> For <strong>lit</strong> test formats which substitute variables into a test
script, the list of substitutions to perform. Used by: <em>ShTest</em>.</p>
<p><strong>unsupported</strong> Mark an unsupported directory, all tests within it will be
reported as unsupported. Used by: <em>ShTest</em>.</p>
<p><strong>parent</strong> The parent configuration, this is the config object for the directory
containing the test suite, or None.</p>
<p><strong>root</strong> The root configuration. This is the top-most <strong class="program">lit</strong> configuration in
the project.</p>
<p><strong>pipefail</strong> Normally a test using a shell pipe fails if any of the commands
on the pipe fail. If this is not desired, setting this variable to false
makes the test fail only if the last command in the pipe fails.</p>
<p><strong>available_features</strong> A set of features that can be used in <cite>XFAIL</cite>,
<cite>REQUIRES</cite>, and <cite>UNSUPPORTED</cite> directives.</p>
</div></blockquote>
</div>
<div class="section" id="id7">
<h3>TEST DISCOVERY<a class="headerlink" href="#id7" title="Permalink to this headline">¶</a></h3>
<p>Once test suites are located, <strong class="program">lit</strong> recursively traverses the source
directory (following <em>test_source_root</em>) looking for tests. When <strong class="program">lit</strong>
enters a sub-directory, it first checks to see if a nested test suite is
defined in that directory. If so, it loads that test suite recursively,
otherwise it instantiates a local test config for the directory (see
<a class="reference internal" href="#local-configuration-files"><span class="std std-ref">LOCAL CONFIGURATION FILES</span></a>).</p>
<p>Tests are identified by the test suite they are contained within, and the
relative path inside that suite. Note that the relative path may not refer to
an actual file on disk; some test formats (such as <em>GoogleTest</em>) define
“virtual tests” which have a path that contains both the path to the actual
test file and a subpath to identify the virtual test.</p>
</div>
<div class="section" id="local-configuration-files">
<span id="id8"></span><h3>LOCAL CONFIGURATION FILES<a class="headerlink" href="#local-configuration-files" title="Permalink to this headline">¶</a></h3>
<p>When <strong class="program">lit</strong> loads a subdirectory in a test suite, it instantiates a
local test configuration by cloning the configuration for the parent directory
— the root of this configuration chain will always be a test suite. Once the
test configuration is cloned <strong class="program">lit</strong> checks for a <em>lit.local.cfg</em> file
in the subdirectory. If present, this file will be loaded and can be used to
specialize the configuration for each individual directory. This facility can
be used to define subdirectories of optional tests, or to change other
configuration parameters — for example, to change the test format, or the
suffixes which identify test files.</p>
</div>
<div class="section" id="substitutions">
<h3>SUBSTITUTIONS<a class="headerlink" href="#substitutions" title="Permalink to this headline">¶</a></h3>
<p><strong class="program">lit</strong> allows patterns to be substituted inside RUN commands. It also
provides the following base set of substitutions, which are defined in
TestRunner.py:</p>
<blockquote>
<div><table class="docutils align-default">
<colgroup>
<col style="width: 24%" />
<col style="width: 76%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Macro</p></th>
<th class="head"><p>Substitution</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>%s</p></td>
<td><p>source path (path to the file currently being run)</p></td>
</tr>
<tr class="row-odd"><td><p>%S</p></td>
<td><p>source dir (directory of the file currently being run)</p></td>
</tr>
<tr class="row-even"><td><p>%p</p></td>
<td><p>same as %S</p></td>
</tr>
<tr class="row-odd"><td><p>%{pathsep}</p></td>
<td><p>path separator</p></td>
</tr>
<tr class="row-even"><td><p>%t</p></td>
<td><p>temporary file name unique to the test</p></td>
</tr>
<tr class="row-odd"><td><p>%basename_t</p></td>
<td><p>The last path component of %t but without the <code class="docutils literal notranslate"><span class="pre">.tmp</span></code> extension</p></td>
</tr>
<tr class="row-even"><td><p>%T</p></td>
<td><p>parent directory of %t (not unique, deprecated, do not use)</p></td>
</tr>
<tr class="row-odd"><td><p>%%</p></td>
<td><p>%</p></td>
</tr>
<tr class="row-even"><td><p>%/s</p></td>
<td><p>%s but <code class="docutils literal notranslate"><span class="pre">\</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">/</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>%/S</p></td>
<td><p>%S but <code class="docutils literal notranslate"><span class="pre">\</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">/</span></code></p></td>
</tr>
<tr class="row-even"><td><p>%/p</p></td>
<td><p>%p but <code class="docutils literal notranslate"><span class="pre">\</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">/</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>%/t</p></td>
<td><p>%t but <code class="docutils literal notranslate"><span class="pre">\</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">/</span></code></p></td>
</tr>
<tr class="row-even"><td><p>%/T</p></td>
<td><p>%T but <code class="docutils literal notranslate"><span class="pre">\</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">/</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>%{/s:regex_replacement}</p></td>
<td><p>%/s but escaped for use in the replacement of a <code class="docutils literal notranslate"><span class="pre">s@@@</span></code> command in sed</p></td>
</tr>
<tr class="row-even"><td><p>%{/S:regex_replacement}</p></td>
<td><p>%/S but escaped for use in the replacement of a <code class="docutils literal notranslate"><span class="pre">s@@@</span></code> command in sed</p></td>
</tr>
<tr class="row-odd"><td><p>%{/p:regex_replacement}</p></td>
<td><p>%/p but escaped for use in the replacement of a <code class="docutils literal notranslate"><span class="pre">s@@@</span></code> command in sed</p></td>
</tr>
<tr class="row-even"><td><p>%{/t:regex_replacement}</p></td>
<td><p>%/t but escaped for use in the replacement of a <code class="docutils literal notranslate"><span class="pre">s@@@</span></code> command in sed</p></td>
</tr>
<tr class="row-odd"><td><p>%{/T:regex_replacement}</p></td>
<td><p>%/T but escaped for use in the replacement of a <code class="docutils literal notranslate"><span class="pre">s@@@</span></code> command in sed</p></td>
</tr>
<tr class="row-even"><td><p>%:s</p></td>
<td><p>On Windows, %/s but a <code class="docutils literal notranslate"><span class="pre">:</span></code> is removed if its the second character.
Otherwise, %s but with a single leading <code class="docutils literal notranslate"><span class="pre">/</span></code> removed.</p></td>
</tr>
<tr class="row-odd"><td><p>%:S</p></td>
<td><p>On Windows, %/S but a <code class="docutils literal notranslate"><span class="pre">:</span></code> is removed if its the second character.
Otherwise, %S but with a single leading <code class="docutils literal notranslate"><span class="pre">/</span></code> removed.</p></td>
</tr>
<tr class="row-even"><td><p>%:p</p></td>
<td><p>On Windows, %/p but a <code class="docutils literal notranslate"><span class="pre">:</span></code> is removed if its the second character.
Otherwise, %p but with a single leading <code class="docutils literal notranslate"><span class="pre">/</span></code> removed.</p></td>
</tr>
<tr class="row-odd"><td><p>%:t</p></td>
<td><p>On Windows, %/t but a <code class="docutils literal notranslate"><span class="pre">:</span></code> is removed if its the second character.
Otherwise, %t but with a single leading <code class="docutils literal notranslate"><span class="pre">/</span></code> removed.</p></td>
</tr>
<tr class="row-even"><td><p>%:T</p></td>
<td><p>On Windows, %/T but a <code class="docutils literal notranslate"><span class="pre">:</span></code> is removed if its the second character.
Otherwise, %T but with a single leading <code class="docutils literal notranslate"><span class="pre">/</span></code> removed.</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
<p>Other substitutions are provided that are variations on this base set and
further substitution patterns can be defined by each test module. See the
modules <a class="reference internal" href="#local-configuration-files"><span class="std std-ref">LOCAL CONFIGURATION FILES</span></a>.</p>
<p>By default, substitutions are expanded exactly once, so that if e.g. a
substitution <code class="docutils literal notranslate"><span class="pre">%build</span></code> is defined in top of another substitution <code class="docutils literal notranslate"><span class="pre">%cxx</span></code>,
<code class="docutils literal notranslate"><span class="pre">%build</span></code> will expand to <code class="docutils literal notranslate"><span class="pre">%cxx</span></code> textually, not to what <code class="docutils literal notranslate"><span class="pre">%cxx</span></code> expands to.
However, if the <code class="docutils literal notranslate"><span class="pre">recursiveExpansionLimit</span></code> property of the <code class="docutils literal notranslate"><span class="pre">TestingConfig</span></code>
is set to a non-negative integer, substitutions will be expanded recursively
until that limit is reached. It is an error if the limit is reached and
expanding substitutions again would yield a different result.</p>
<p>More detailed information on substitutions can be found in the
<a class="reference internal" href="../TestingGuide.html"><span class="doc">LLVM Testing Infrastructure Guide</span></a>.</p>
</div>
<div class="section" id="test-run-output-format">
<h3>TEST RUN OUTPUT FORMAT<a class="headerlink" href="#test-run-output-format" title="Permalink to this headline">¶</a></h3>
<p>The <strong class="program">lit</strong> output for a test run conforms to the following schema, in
both short and verbose modes (although in short mode no PASS lines will be
shown). This schema has been chosen to be relatively easy to reliably parse by
a machine (for example in buildbot log scraping), and for other tools to
generate.</p>
<p>Each test result is expected to appear on a line that matches:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span><result code>: <test name> (<progress info>)
</pre></div>
</div>
<p>where <code class="docutils literal notranslate"><span class="pre"><result-code></span></code> is a standard test result such as PASS, FAIL, XFAIL,
XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and
REGRESSED are also allowed.</p>
<p>The <code class="docutils literal notranslate"><span class="pre"><test</span> <span class="pre">name></span></code> field can consist of an arbitrary string containing no
newline.</p>
<p>The <code class="docutils literal notranslate"><span class="pre"><progress</span> <span class="pre">info></span></code> field can be used to report progress information such
as (1/300) or can be empty, but even when empty the parentheses are required.</p>
<p>Each test result may include additional (multiline) log information in the
following format:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span><log delineator> TEST '(<test name>)' <trailing delineator>
... log message ...
<log delineator>
</pre></div>
</div>
<p>where <code class="docutils literal notranslate"><span class="pre"><test</span> <span class="pre">name></span></code> should be the name of a preceding reported test, <code class="docutils literal notranslate"><span class="pre"><log</span>
<span class="pre">delineator></span></code> is a string of “*” characters <em>at least</em> four characters long
(the recommended length is 20), and <code class="docutils literal notranslate"><span class="pre"><trailing</span> <span class="pre">delineator></span></code> is an arbitrary
(unparsed) string.</p>
<p>The following is an example of a test run output which consists of four tests A,
B, C, and D, and a log message for the failing test C:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>PASS: A (1 of 4)
PASS: B (2 of 4)
FAIL: C (3 of 4)
******************** TEST 'C' FAILED ********************
Test 'C' failed as a result of exit code 1.
********************
PASS: D (4 of 4)
</pre></div>
</div>
</div>
<div class="section" id="lit-example-tests">
<h3>LIT EXAMPLE TESTS<a class="headerlink" href="#lit-example-tests" title="Permalink to this headline">¶</a></h3>
<p>The <strong class="program">lit</strong> distribution contains several example implementations of
test suites in the <em>ExampleTests</em> directory.</p>
</div>
</div>
<div class="section" id="see-also">
<h2>SEE ALSO<a class="headerlink" href="#see-also" title="Permalink to this headline">¶</a></h2>
<p>valgrind(1)</p>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="llvm-exegesis.html" title="llvm-exegesis - LLVM Machine Instruction Benchmark"
>next</a> |</li>
<li class="right" >
<a href="mlir-tblgen.html" title="mlir-tblgen - Description to C++ Code for MLIR"
>previous</a> |</li>
<li><a href="https://llvm.org/">LLVM Home</a> | </li>
<li><a href="../index.html">Documentation</a>»</li>
<li class="nav-item nav-item-1"><a href="../Reference.html" >Reference</a> »</li>
<li class="nav-item nav-item-2"><a href="index.html" >LLVM Command Guide</a> »</li>
<li class="nav-item nav-item-this"><a href="">lit - LLVM Integrated Tester</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2003-2021, LLVM Project.
Last updated on 2021-09-18.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.5.4.
</div>
</body>
</html>
|
