File: index.html

package info (click to toggle)
nvidia-cuda-toolkit 12.4.1-3
links: PTS, VCS
area: non-free
in suites: forky, sid
size: 18,505,836 kB
sloc: ansic: 203,477; cpp: 64,769; python: 34,699; javascript: 22,006; xml: 13,410; makefile: 3,085; sh: 2,343; perl: 352
file content (3709 lines) | stat: -rw-r--r-- 302,739 bytes
parent folder | download | duplicates (6)
<!DOCTYPE html
  PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-us" xml:lang="en-us">
   <head>
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
      <meta http-equiv="X-UA-Compatible" content="IE=edge"></meta>
      <meta name="copyright" content="(C) Copyright 2005"></meta>
      <meta name="DC.rights.owner" content="(C) Copyright 2005"></meta>
      <meta name="DC.Type" content="concept"></meta>
      <meta name="DC.Title" content="Compute Sanitizer"></meta>
      <meta name="abstract" content="The user manual for Compute Sanitizer."></meta>
      <meta name="description" content="The user manual for Compute Sanitizer."></meta>
      <meta name="DC.Coverage" content="Tools"></meta>
      <meta name="DC.subject" content="Compute Sanitizer, Compute Sanitizer features, Compute Sanitizer tools, Compute Sanitizer supported OS, Compute Sanitizer supported devices, Compute Sanitizer error, Compute Sanitizer memcheck, Compute Sanitizer racecheck, Compute Sanitizer synccheck, Compute Sanitizer initcheck, Compute Sanitizer backtrace, Compute Sanitizer hardware exception, Compute Sanitizer memory access"></meta>
      <meta name="keywords" content="Compute Sanitizer, Compute Sanitizer features, Compute Sanitizer tools, Compute Sanitizer supported OS, Compute Sanitizer supported devices, Compute Sanitizer error, Compute Sanitizer memcheck, Compute Sanitizer racecheck, Compute Sanitizer synccheck, Compute Sanitizer initcheck, Compute Sanitizer backtrace, Compute Sanitizer hardware exception, Compute Sanitizer memory access"></meta>
      <meta name="DC.Format" content="XHTML"></meta>
      <meta name="DC.Identifier" content="abstract"></meta>
      <link rel="stylesheet" type="text/css" href="../common/formatting/commonltr.css"></link>
      <link rel="stylesheet" type="text/css" href="../common/formatting/site.css"></link>
      <title>Compute Sanitizer User Manual :: Compute Sanitizer Documentation</title>
      <!--[if lt IE 9]>
      <script src="../common/formatting/html5shiv-printshiv.min.js"></script>
      <![endif]-->
      <script type="text/javascript" charset="utf-8" src="../common/scripts/tynt/tynt.js"></script>
      --&gt;
      
      <script src="https://assets.adobedtm.com/5d4962a43b79/c1061d2c5e7b/launch-191c2462b890.min.js"></script>
      <script type="text/javascript" charset="utf-8" src="../common/formatting/jquery.min.js"></script>
      <script type="text/javascript" charset="utf-8" src="../common/formatting/jquery.ba-hashchange.min.js"></script>
      <script type="text/javascript" charset="utf-8" src="../common/formatting/jquery.scrollintoview.min.js"></script>
      <script type="text/javascript" src="../search/htmlFileList.js"></script>
      <script type="text/javascript" src="../search/htmlFileInfoList.js"></script>
      <script type="text/javascript" src="../search/nwSearchFnt.min.js"></script>
      <script type="text/javascript" src="../search/stemmers/en_stemmer.min.js"></script>
      <script type="text/javascript" src="../search/index-1.js"></script>
      <script type="text/javascript" src="../search/index-2.js"></script>
      <script type="text/javascript" src="../search/index-3.js"></script>
      <link rel="canonical" href="https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html"></link>
      <link rel="stylesheet" type="text/css" href="../common/formatting/qwcode.highlight.css"></link>
   </head>
   <body>
      
      <header id="header"><span id="company">NVIDIA</span><span id="site-title">Compute Sanitizer Documentation</span><form id="search" method="get" action="search">
            <input type="text" name="search-text"></input><fieldset id="search-location">
               <legend>Search In:</legend>
               <label><input type="radio" name="search-type" value="site"></input>Entire Site</label>
               <label><input type="radio" name="search-type" value="document"></input>Just This Document</label></fieldset>
            <button type="reset">clear search</button>
            <button id="submit" type="submit">search</button></form>
      </header>
      <div id="site-content">
         <nav id="site-nav">
            <div class="category closed"><a href="../index.html" title="The root of the site.">Compute Sanitizer
                  v2024.1.1</a></div>
            <div class="category"><a href="index.html" title="Compute Sanitizer User Manual">Compute Sanitizer User Manual</a></div>
            <ul>
               <li>
                  <div class="section-link"><a href="#introduction">1.&nbsp;Introduction</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#about-compute-sanitizer">1.1.&nbsp;About Compute Sanitizer</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#why-compute-sanitizer">1.2.&nbsp;Why Compute Sanitizer</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#how-to-get-compute-sanitizer">1.3.&nbsp;How to Get Compute Sanitizer</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#compute-sanitizer-tools">1.4.&nbsp;Compute Sanitizer Tools</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#using-compute-sanitizer">2.&nbsp;Compute Sanitizer</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#command-line-options">2.1.&nbsp;Command Line Options</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#compilation-options">2.2.&nbsp;Compilation Options</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#environment-variables">2.3.&nbsp;Environment Variables</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#memcheck-tool">3.&nbsp;Memcheck Tool</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#what-is-memcheck">3.1.&nbsp;What is Memcheck?</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#supported-error-detection">3.2.&nbsp;Supported Error Detection</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#using-memcheck">3.3.&nbsp;Using Memcheck</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#understanding-memcheck-errors">3.4.&nbsp;Understanding Memcheck Errors</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#api-error-checking">3.5.&nbsp;CUDA API Error Checking</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#device-side-allocation-checking">3.6.&nbsp;Device Side Allocation Checking</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#leak-checking">3.7.&nbsp;Leak Checking</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#padding">3.8.&nbsp;Padding</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#stream-ordered-races">3.9.&nbsp;Stream-ordered race detection</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#racecheck-tool">4.&nbsp;Racecheck Tool</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#what-is-racecheck">4.1.&nbsp;What is Racecheck?</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#what-are-hazards">4.2.&nbsp;What are Hazards?</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#using-racecheck">4.3.&nbsp;Using Racecheck</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#racecheck-report-modes">4.4.&nbsp;Racecheck Report Modes</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#understanding-racecheck-analysis-reports">4.5.&nbsp;Understanding Racecheck Analysis Reports</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#understanding-racecheck-hazard-reports">4.6.&nbsp;Understanding Racecheck Hazard Reports</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#racecheck-severity-levels">4.7.&nbsp;Racecheck Severity Levels</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#racecheck-cuda-barrier">4.8.&nbsp;Racecheck support for cuda::barrier</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#racecheck-asynchronous-copy">4.9.&nbsp;Racecheck support for asynchronous copy</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#racecheck-cluster-races">4.10.&nbsp;Racecheck cluster entry and exit race detection</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#initcheck-tool">5.&nbsp;Initcheck Tool</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#what-is-initcheck">5.1.&nbsp;What is Initcheck? </a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#using-initcheck">5.2.&nbsp;Using Initcheck</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#unused-memory">5.3.&nbsp;Unused memory detection</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#synccheck-tool">6.&nbsp;Synccheck Tool</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#what-is-synccheck">6.1.&nbsp;What is Synccheck?</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#using-synccheck">6.2.&nbsp;Using Synccheck</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#understanding-synccheck-reports">6.3.&nbsp;Understanding Synccheck Reports</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#synccheck-cuda-barrier">6.4.&nbsp;Synccheck support for cuda::barrier</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#synccheck-wgmma">6.5.&nbsp;Synccheck support for wgmma</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#compute-sanitizer-features">7.&nbsp;Compute Sanitizer Features</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#nonblocking-mode">7.1.&nbsp;Nonblocking Mode</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#stack-backtraces">7.2.&nbsp;Stack Backtraces</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#name-demangling">7.3.&nbsp;Name Demangling</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#dynamic-parallelism">7.4.&nbsp;Dynamic Parallelism</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#error-actions">7.5.&nbsp;Error Actions</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#escape-sequences">7.6.&nbsp;Escape Sequences</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#specifying-filters">7.7.&nbsp;Specifying Filters</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#coredump">7.8.&nbsp;Coredump support</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#error-suppression">7.9.&nbsp;Error suppression</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#optix">7.10.&nbsp;OptiX support</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#usage-guide">8.&nbsp;Usage Guide</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#memory-footprint">8.1.&nbsp;Memory Footprint</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#os-specific-behavior">9.&nbsp;Operating System Specific Behavior</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#os-specific-windows">9.1.&nbsp;Windows Specific Behavior</a></div>
                     </li>
                     <li>
                        <div class="section-link"><a href="#tegra-setup">9.2.&nbsp;Using the Compute Sanitizer on Jetson and Tegra devices</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#cuda-fortran-support">10.&nbsp;CUDA Fortran Support</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#cuda-fortran-specific-behavior">10.1.&nbsp;CUDA Fortran Specific Behavior</a></div>
                     </li>
                  </ul>
               </li>
               <li>
                  <div class="section-link"><a href="#compute-sanitizer-tool-examples">11.&nbsp;Compute Sanitizer Tool Examples</a></div>
                  <ul>
                     <li>
                        <div class="section-link"><a href="#example-use-of-memcheck">11.1.&nbsp;Example Use of Memcheck</a></div>
                        <ul>
                           <li>
                              <div class="section-link"><a href="#memcheck-demo-output">11.1.1.&nbsp;memcheck_demo Output </a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#memcheck-demo-output-with-memcheck-release-build">11.1.2.&nbsp;memcheck_demo Output with Memcheck (Release Build)</a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#memcheck-demo-output-with-memcheck-debug-build">11.1.3.&nbsp;memcheck_demo Output with Memcheck (Debug Build)</a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#leak-checking-in-compute-sanitizer">11.1.4.&nbsp;Leak Checking in Compute Sanitizer</a></div>
                           </li>
                        </ul>
                     </li>
                     <li>
                        <div class="section-link"><a href="#example-use-of-racecheck">11.2.&nbsp;Example Use of Racecheck</a></div>
                        <ul>
                           <li>
                              <div class="section-link"><a href="#racecheck-demo-block-error">11.2.1.&nbsp;Block-level Hazards</a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#racecheck-demo-warp-error">11.2.2.&nbsp;Warp-level Hazards</a></div>
                           </li>
                        </ul>
                     </li>
                     <li>
                        <div class="section-link"><a href="#example-use-of-initcheck">11.3.&nbsp;Example Use of Initcheck</a></div>
                        <ul>
                           <li>
                              <div class="section-link"><a href="#initcheck-demo-memset-error">11.3.1.&nbsp;Memset Error</a></div>
                           </li>
                        </ul>
                     </li>
                     <li>
                        <div class="section-link"><a href="#example-use-of-synccheck">11.4.&nbsp;Example Use of Synccheck</a></div>
                        <ul>
                           <li>
                              <div class="section-link"><a href="#synccheck-demo-divergent-threads">11.4.1.&nbsp;Divergent Threads</a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#synccheck-demo-illegal-syncwarp">11.4.2.&nbsp;Illegal Syncwarp</a></div>
                           </li>
                        </ul>
                     </li>
                     <li>
                        <div class="section-link"><a href="#example-use-of-suppressions">11.5.&nbsp;Example Use of suppressions</a></div>
                        <ul>
                           <li>
                              <div class="section-link"><a href="#suppressions-demo-api">11.5.1.&nbsp;API error suppression</a></div>
                           </li>
                           <li>
                              <div class="section-link"><a href="#suppressions-demo-initcheck">11.5.2.&nbsp;Initcheck error suppression</a></div>
                           </li>
                        </ul>
                     </li>
                  </ul>
               </li>
            </ul>
         </nav>
         <div id="resize-nav"></div>
         <nav id="search-results">
            <h2>Search Results</h2>
            <ol></ol>
         </nav>
         
         <div id="contents-container">
            <div id="breadcrumbs-container">
               <div id="release-info">Compute Sanitizer User Manual
                  (<a href="../pdf/ComputeSanitizer.pdf">PDF</a>)
                  
                  -
                  
                  v2024.1.1
                  (<a href="https://developer.nvidia.com/cuda-toolkit-archive">older</a>)
                  -
                  Last updated April 3, 2024
                  -
                  <a href="mailto:devtools@nvidia.com?subject=Compute Sanitizer Documentation Feedback: Compute Sanitizer User Manual">Send Feedback</a></div>
            </div>
            <article id="contents">
               <div class="topic nested0" id="abstract"><a name="abstract" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#abstract" name="abstract" shape="rect">Compute Sanitizer</a></h2>
                  <div class="body conbody">
                     <p class="shortdesc">The user manual for Compute Sanitizer.</p>
                  </div>
               </div>
               <div class="topic concept nested0" id="introduction"><a name="introduction" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#introduction" name="introduction" shape="rect">1.&nbsp;Introduction</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="about-compute-sanitizer"><a name="about-compute-sanitizer" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#about-compute-sanitizer" name="about-compute-sanitizer" shape="rect">1.1.&nbsp;About Compute Sanitizer</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Compute Sanitizer is a functional correctness checking suite included in the CUDA toolkit.
                           This suite contains multiple tools that can perform different type of checks.
                           The <dfn class="term">memcheck</dfn> tool is capable of precisely detecting and attributing out of bounds
                           and misaligned memory access errors in CUDA applications.
                           The tool can also report hardware exceptions encountered by the GPU.
                           The <dfn class="term">racecheck</dfn> tool can report shared memory data access hazards that can cause data races.
                           The <dfn class="term">initcheck</dfn> tool can report cases where the GPU performs uninitialized accesses to global memory.
                           The <dfn class="term">synccheck</dfn> tool can report cases where the application is attempting invalid usages
                           of synchronization primitives.
                           This document describes the usage of these tools.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="why-compute-sanitizer"><a name="why-compute-sanitizer" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#why-compute-sanitizer" name="why-compute-sanitizer" shape="rect">1.2.&nbsp;Why Compute Sanitizer</a></h3>
                     <div class="body conbody">
                        <p class="p"> NVIDIA allows developers to easily harness the power of GPUs to solve problems in
                           parallel using CUDA. CUDA applications often run thousands of threads in parallel. Every
                           programmer invariably encounters memory access errors and thread ordering, hazards that
                           are hard to detect and time consuming to debug. The number of such errors increases
                           substantially when dealing with thousands of threads. The Compute Sanitizer suite is
                           designed to detect those problems in your CUDA application. 
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="how-to-get-compute-sanitizer"><a name="how-to-get-compute-sanitizer" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#how-to-get-compute-sanitizer" name="how-to-get-compute-sanitizer" shape="rect">1.3.&nbsp;How to Get Compute Sanitizer</a></h3>
                     <div class="body conbody">
                        <p class="p">Compute Sanitizer is installed as part of the CUDA toolkit.</p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="compute-sanitizer-tools"><a name="compute-sanitizer-tools" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#compute-sanitizer-tools" name="compute-sanitizer-tools" shape="rect">1.4.&nbsp;Compute Sanitizer Tools</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           Compute Sanitizer provides different checking mechanisms through different tools.
                           Currently the supported tools are:
                           
                           <ul class="ul">
                              <li class="li"><dfn class="term">Memcheck</dfn> – The memory access error and leak detection tool. See
                                 <a class="xref" href="index.html#memcheck-tool" shape="rect">Memcheck Tool</a></li>
                              <li class="li"><dfn class="term">Racecheck</dfn> – The shared memory data access hazard detection tool.
                                 See <a class="xref" href="index.html#racecheck-tool" shape="rect">Racecheck Tool</a></li>
                              <li class="li"><dfn class="term">Initcheck</dfn> – The uninitialized device global memory access detection
                                 tool. See <a class="xref" href="index.html#initcheck-tool" shape="rect">Initcheck Tool</a></li>
                              <li class="li"><dfn class="term">Synccheck</dfn> – The thread synchronization hazard detection tool. See
                                 <a class="xref" href="index.html#synccheck-tool" shape="rect">Synccheck Tool</a></li>
                           </ul>
                        </div>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="using-compute-sanitizer"><a name="using-compute-sanitizer" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#using-compute-sanitizer" name="using-compute-sanitizer" shape="rect">2.&nbsp;Compute Sanitizer</a></h2>
                  <div class="body conbody">
                     <div class="p">Compute Sanitizer tools can be invoked by running the <samp class="ph codeph">compute-sanitizer</samp>
                        executable as follows:
                        <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer [options] app_name [app_options]</strong>
</pre></div>
                     <p class="p">
                        For a full list of options that can be specified to compute-sanitizer and their default values,
                        see <a class="xref" href="index.html#command-line-options" shape="rect">Command Line Options</a></p>
                  </div>
                  <div class="topic concept nested1" id="command-line-options"><a name="command-line-options" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#command-line-options" name="command-line-options" shape="rect">2.1.&nbsp;Command Line Options</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Command line options can be specified to <samp class="ph codeph">compute-sanitizer</samp>.
                           With some exceptions, the options are usually of the form <samp class="ph codeph">--option value</samp>.
                           The option list can be terminated by specifying <samp class="ph codeph">--</samp>. All subsequent words
                           are treated as the application being run and its arguments.
                           
                        </p>
                        <p class="p"> The table below describes the supported options in detail. The first column is the
                           option name passed to <samp class="ph codeph">compute-sanitizer</samp>. Some options have a one
                           character short form, which is given in parentheses. These options can be invoked using
                           a single hyphen. For example, the help option can be invoked as <samp class="ph codeph">-h</samp>. The
                           options that have a short form do not take a value. 
                        </p>
                        <p class="p"> The second column contains the permissible values for the option. In case the value is
                           user defined, it is shown below in braces <samp class="ph codeph">{}</samp>. An option that can accept
                           any numerical value is represented as <dfn class="term"> {number}.</dfn></p>
                        <p class="p">
                           The third column contains the default value of the option. Some options have different default values
                           depending on the architecture they are being run on.
                           
                        </p>
                        <div class="tablenoborder"><a name="command-line-options__compute-sanitizer-command-line-options" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="command-line-options__compute-sanitizer-command-line-options" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 1. Compute Sanitizer command line options</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="20%" id="d54e266" rowspan="1" colspan="1">Option</th>
                                    <th class="entry" valign="top" width="20%" id="d54e269" rowspan="1" colspan="1">Values</th>
                                    <th class="entry" valign="top" width="20%" id="d54e272" rowspan="1" colspan="1">Default</th>
                                    <th class="entry" valign="top" width="40%" id="d54e275" rowspan="1" colspan="1">Description</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">check-device-heap</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">yes,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Enables checking of device heap allocations. This applies to both error checking and leak checking.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">check-exit-code</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">yes, no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Checks the application exit code and print an error if it is different than 0.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">check-optix-leaks</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Detects and reports OptiX resources that were created and were not destroyed at OptixDeviceContextDestroy time.
                                       For more information, see <a class="xref" href="index.html#optix" shape="rect">OptiX support</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">check-warpgroup-mma</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">yes,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Enables memcheck and synccheck support for PTX <samp class="ph codeph">wgmma</samp> instructions (requires sm_90a).
                                       For memcheck, the tool checks that the matrices loaded by <samp class="ph codeph">wgmma.mma_async</samp> are in shared memory range.
                                       For synccheck, see <a class="xref" href="index.html#synccheck-wgmma" shape="rect">Synccheck support for wgmma</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">coredump-behavior</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">full,exit</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">full</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Controls the behavior of the target application after generating a CUDA coredump.
                                       
                                       <ul class="ul">
                                          <li class="li"><strong class="ph b">full:</strong> Abort the target application and generate a CPU coredump.
                                          </li>
                                          <li class="li"><strong class="ph b">exit:</strong> Exit the target application without generating a CPU coredump.
                                          </li>
                                       </ul>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">coredump-name</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{filename}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Name to use for the generated coredump file.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">demangle</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">full, simple, no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">full</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Enables the demangling of device function names. For more information, see <a class="xref" href="index.html#name-demangling" shape="rect">Name Demangling</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">destroy-on-device-error</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">context,kernel</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">context</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       This controls how the application proceeds on hitting a memory access error.
                                       For more information, see <a class="xref" href="index.html#error-actions" shape="rect">Error Actions</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">error-exitcode</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       The exit code Compute Sanitizer will return if the original application succeeded but
                                       the tool detected that errors were present. This is meant to allow Compute Sanitizer to be integrated
                                       into automated test suites.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">force-blocking-launches</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       This forces all host kernel launches to be sequential. When enabled, the number and precision of
                                       reported errors will decrease.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">force-synchronization-limit</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       This forces a synchronization after a stream reaches the given number of launches without
                                       synchronizing. This is meant to reduce the memory usage of the Compute Sanitizer tools, but
                                       it can affect performances.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">generate-coredump</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       When this is set, a coredump will be generated for the first error encountered and program execution will be stopped.
                                       For more information, see <a class="xref" href="index.html#coredump" shape="rect">Coredump support</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">help (h)</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Displays the help message</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">injection-path</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Sets the path to injection libraries.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">injection-path32</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Sets the path to 32bit injection libraries.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">kernel-name</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{key1=val1}[{,key2=val2}]</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Controls which application kernels will be checked by the running the Compute Sanitizer tool.
                                       For more information, see <a class="xref" href="index.html#specifying-filters" shape="rect">Specifying Filters</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">kernel-name-exclude</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{key1=val1}[{,key2=val2}]</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Controls which application kernels will be checked by the running the Compute Sanitizer tool.
                                       For more information, see <a class="xref" href="index.html#specifying-filters" shape="rect">Specifying Filters</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">language</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">c,fortran</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">c</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1"> This controls the application source language
                                       specific behavior in Compute Sanitizer tools. For fortran specific
                                       behavior, see <a class="xref" href="index.html#cuda-fortran-specific-behavior" shape="rect">CUDA Fortran Specific Behavior</a>. 
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">c,launch-count</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Limit the number of kernel launches to check.
                                       The count is only incremented for launches that match the kernel filters.
                                       Use 0 for unlimited.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">s,launch-skip</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Set the number of kernel launches to skip before starting to check.
                                       The count is only incremented for launches that match the kernel filters.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">launch-timeout</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">10 for single process, 60 for multi-process</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Timeout in seconds for the connection to the target process.
                                       A value of zero forces compute-sanitizer to wait infinitely.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">log-file</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{filename}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       This is the file Compute Sanitizer will write all of its text output to. By default,
                                       Compute Sanitizer will print all output to stdout. For more information, see
                                       <a class="xref" href="index.html#escape-sequences" shape="rect">Escape Sequences</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">max-connections</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">10</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Maximum number of ports for connecting to the target application.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">kill</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Makes the compute-sanitizer kill the target application when a communication error is met.
                                       By default, the compute-sanitizer will instead await for the normal completion of the
                                       program without reporting potential errors.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">mode</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">launch-and-attach,launch,attach</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">launch-and-attach</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Select the mode of interaction with the target application
                                       
                                       <ul class="ul">
                                          <li class="li"><strong class="ph b">launch-and-attach:</strong> Launch the target application and immediately attach.
                                          </li>
                                          <li class="li"><strong class="ph b">launch:</strong> Launch the target application and suspend it, waiting for tool to attach.
                                          </li>
                                          <li class="li"><strong class="ph b">attach:</strong> Attach to a previously launched application to which no other tool is attached.
                                          </li>
                                       </ul>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">num-callers-device</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Set the number of callers to print in device stack traces.
                                       Use 0 for unlimited.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">num-callers-host</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Set the number of callers to print in host stack traces.
                                       Use 0 for unlimited.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">num-cuda-barriers</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Set the number of cuda::barriers that the target application will use per block.
                                       Use 0 for automatic detection.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">nvtx</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">true,false</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">true</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Enable NVTX support.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">port</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">49152</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Base port for connecting to the target application.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">prefix</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{string}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">========</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">The string prepended to Compute Sanitizer output lines.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">print-level</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">info,warn,error,fatal</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">warn</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">The minimum print level of messages from Compute Sanitizer.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">print-limit</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">100</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       When this option is set, Compute Sanitizer will stop printing errors after reaching the given number of errors.
                                       Use 0 for unlimited printing.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">print-session-details</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Print details about the sanitizer session for each target application such as process ID, command line,
                                       target system etc.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">quiet,q</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Controls whether to run silently and only print error messages.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">read</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{filename}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       The input Compute Sanitizer file to read data from. This can be used in conjunction with the --save option
                                       to allow processing records after a run.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">require-cuda-init</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">yes, no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Controls whether Compute Sanitizer should return an error if the target
                                       application does not use CUDA.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">save</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{filename}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Filename where Compute Sanitizer will save the output from the current run.
                                       For more information, see <a class="xref" href="index.html#escape-sequences" shape="rect">Escape Sequences</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">save-session-details</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Save details about the sanitizer session for each target application in the file specified by <samp class="ph codeph">--save</samp>.
                                       This option has no effect if the <samp class="ph codeph">--save</samp> option is not used.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">show-backtrace</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">yes,host,device,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Displays a backtrace for most types of errors. "no" disables all backtraces, "yes" enables all backtraces.
                                       "host" enables only host side backtraces. "device" enables only device side backtraces.
                                       For more information, see <a class="xref" href="index.html#stack-backtraces" shape="rect">Stack Backtraces</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">support-32bit</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       This option only exists on Linux x86_64. Enables the support for tracking application that includes
                                       32-bit processes. On Windows, the support is always enabled if the 32bit injection libraries are found.
                                       Note: Only the 64bit processes are supported for actual checking, the purpose of the option is to
                                       allow tracking of the children process of a 32bit process.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">suppressions</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{filename}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Input XML file containing a list of reports that should be filtered out by the tool if detected.
                                       For more information, see <a class="xref" href="index.html#error-suppression" shape="rect">Error suppression</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">target-processes</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">application-only,all</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">all</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Select which processes are to be tracked by compute-sanitizer: The root application process,
                                       or the root application and all its child processes.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">target-processes-filter</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">{string}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Set the comma separated expressions to filter which processes are tracked.
                                       
                                       <ul class="ul">
                                          <li class="li"><samp class="ph codeph">&lt;process name&gt;</samp> Set the process name to filter by. Only exactly matched processes are tracked.
                                             
                                          </li>
                                          <li class="li"><samp class="ph codeph">regex:&lt;expression&gt;</samp> Set the regex to filter matching process name profiling.
                                             On shells that recognize regular expression symbols as special characters (e.g. Linux bash),
                                             the expression needs to be escaped with quotes, e.g. <samp class="ph codeph">--target-processes-filter regex:".*Process"</samp>.
                                             
                                          </li>
                                       </ul>
                                       
                                       The executable name will be considered as process name to match.
                                       If the process name or the provided expression match, the process will be tracked.
                                       
                                       <p class="p"><strong class="ph b">Examples</strong></p>
                                       <p class="p"><samp class="ph codeph">--target-processes-filter MatrixMul</samp> Filter all processes having executable name exactly as "MatrixMul".
                                       </p>
                                       <p class="p"><samp class="ph codeph">--target-processes-filter regex:Matrix</samp>Filter all processes that include the string "Matrix" in their executable name, e.g. "MatrixMul" and "MatrixAdd".
                                       </p>
                                       <p class="p"><samp class="ph codeph">--target-processes-filter MatrixMul,MatrixAdd</samp>Filter all processes having executable name exactly as "MatrixMul" or "MatrixAdd".
                                       </p>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">tool</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">memcheck, racecheck, initcheck, synccheck</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">memcheck</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Controls which Compute Sanitizer tool is actively
                                       running.
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">version (V)</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">Prints the version of Compute Sanitizer.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e266" rowspan="1" colspan="1">xml</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e269" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e272" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e275" rowspan="1" colspan="1">
                                       Emit error output to file in XML format. When used, --save must also be set to specify
                                       the file to save to.
                                       
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <div class="tablenoborder"><a name="command-line-options__memcheck-tool-command-line-options" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="command-line-options__memcheck-tool-command-line-options" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 2. <dfn class="term">Memcheck</dfn> tool command line options</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="20%" id="d54e1136" rowspan="1" colspan="1">Option</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1139" rowspan="1" colspan="1">Values</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1142" rowspan="1" colspan="1">Default</th>
                                    <th class="entry" valign="top" width="40%" id="d54e1145" rowspan="1" colspan="1">Description</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">check-cache-control</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Check cache control memory accesses.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">detect-missing-module-unload</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Detect leaks caused by missing module unload calls.
                                       This option can report false positives if the application uses the CUDA runtime as it depends
                                       on the destruction order between runtime and driver when the application exits which is not
                                       guaranteed.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">ignore-getprocaddress-notfound</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Ignore CUDA_ERROR_NOT_FOUND API errors for cuGetProcAddress.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">leak-check</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">full,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">no</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Prints information about all allocations that have not been freed via cudaFree at the point
                                       when the context was destroyed. For more information, see <a class="xref" href="index.html#leak-checking" shape="rect">Leak Checking</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">padding</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Makes the compute-sanitizer allocate padding buffers after every CUDA allocation.
                                       <dfn class="term">number</dfn> is the size in bytes of a padding buffer.
                                       Fore more information, see <a class="xref" href="index.html#padding" shape="rect">Padding</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">report-api-errors</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">all, explicit, no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">explicit</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Reports errors if any CUDA API call fails. For more information, see <a class="xref" href="index.html#api-error-checking" shape="rect">CUDA API Error Checking</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1136" rowspan="1" colspan="1">track-stream-ordered-races arg</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1139" rowspan="1" colspan="1">all,use-before-alloc,use-after-free,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1142" rowspan="1" colspan="1">no</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1145" rowspan="1" colspan="1">
                                       Track CUDA stream-ordered allocations races. For more information, see <a class="xref" href="index.html#stream-ordered-races" shape="rect">Stream-ordered race detection</a>.
                                       
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <div class="tablenoborder"><a name="command-line-options__racecheck-tool-command-line-options" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="command-line-options__racecheck-tool-command-line-options" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 3. <dfn class="term">Racecheck</dfn> tool command line options</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="20%" id="d54e1298" rowspan="1" colspan="1">Option</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1301" rowspan="1" colspan="1">Values</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1304" rowspan="1" colspan="1">Default</th>
                                    <th class="entry" valign="top" width="40%" id="d54e1307" rowspan="1" colspan="1">Description</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1298" rowspan="1" colspan="1">racecheck-detect-level</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1301" rowspan="1" colspan="1">{info,warn,error}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1304" rowspan="1" colspan="1">warn</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1307" rowspan="1" colspan="1">
                                       Set the minimum level of race conditions to detect.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1298" rowspan="1" colspan="1">racecheck-indirect-barrier-dependency</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1301" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1304" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1307" rowspan="1" colspan="1">
                                       Enables tracking of indirect cuda::barrier dependency in racecheck. Avoids false positives when the target
                                       application is relying on chains of arrive-waits on multiple different barriers to synchronize shared memory
                                       accesses between threads that would not have participated in the same barrier.
                                       Using this option may have a performance impact.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1298" rowspan="1" colspan="1">racecheck-memcpy-async</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1301" rowspan="1" colspan="1">yes,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1304" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1307" rowspan="1" colspan="1">
                                       Enables check for asynchronous memory copy operations. For more information, see <a class="xref" href="index.html#racecheck-asynchronous-copy" shape="rect">Racecheck support for asynchronous copy</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1298" rowspan="1" colspan="1">racecheck-num-workers</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1301" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1304" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1307" rowspan="1" colspan="1">
                                       Number of CPU worker threads used by the tool. Use 0 for automatic.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1298" rowspan="1" colspan="1">racecheck-report</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1301" rowspan="1" colspan="1">hazard,analysis,all</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1304" rowspan="1" colspan="1">analysis</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1307" rowspan="1" colspan="1">
                                       Controls how racecheck reports information. For more information, see <a class="xref" href="index.html#racecheck-report-modes" shape="rect">Racecheck Report Modes</a>.
                                       
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <div class="tablenoborder"><a name="command-line-options__initcheck-tool-command-line-options" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="command-line-options__initcheck-tool-command-line-options" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 4. <dfn class="term">Initcheck</dfn> tool command line options</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="20%" id="d54e1420" rowspan="1" colspan="1">Option</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1423" rowspan="1" colspan="1">Values</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1426" rowspan="1" colspan="1">Default</th>
                                    <th class="entry" valign="top" width="40%" id="d54e1429" rowspan="1" colspan="1">Description</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1420" rowspan="1" colspan="1">check-api-memory-access</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1423" rowspan="1" colspan="1">yes,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1426" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1429" rowspan="1" colspan="1">Enables checking of cudaMemcpy/cudaMemset</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1420" rowspan="1" colspan="1">check-optix</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1423" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1426" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1429" rowspan="1" colspan="1">
                                       Check OptiX kernel launches with initcheck.
                                       For more information, see <a class="xref" href="index.html#optix" shape="rect">OptiX support</a>.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1420" rowspan="1" colspan="1">track-unused-memory</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1423" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1426" rowspan="1" colspan="1">disabled</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1429" rowspan="1" colspan="1">Check for unused memory allocations.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1420" rowspan="1" colspan="1">unused-memory-threshold</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1423" rowspan="1" colspan="1">{number}</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1426" rowspan="1" colspan="1">0</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1429" rowspan="1" colspan="1">Threshold in percentage under which unused memory reports are silenced. The value needs to be a number between 0 and 100.</td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <div class="tablenoborder"><a name="command-line-options__synccheck-tool-command-line-options" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="command-line-options__synccheck-tool-command-line-options" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 5. <dfn class="term">Synccheck</dfn> tool command line options</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="20%" id="d54e1524" rowspan="1" colspan="1">Option</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1527" rowspan="1" colspan="1">Values</th>
                                    <th class="entry" valign="top" width="20%" id="d54e1530" rowspan="1" colspan="1">Default</th>
                                    <th class="entry" valign="top" width="40%" id="d54e1533" rowspan="1" colspan="1">Description</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="20%" headers="d54e1524" rowspan="1" colspan="1">missing-barrier-init-is-fatal</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1527" rowspan="1" colspan="1">yes,no</td>
                                    <td class="entry" valign="top" width="20%" headers="d54e1530" rowspan="1" colspan="1">yes</td>
                                    <td class="entry" valign="top" width="40%" headers="d54e1533" rowspan="1" colspan="1">Controls whether a missing <samp class="ph codeph">cuda::barrier</samp> initialization will exit the warp.
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="compilation-options"><a name="compilation-options" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#compilation-options" name="compilation-options" shape="rect">2.2.&nbsp;Compilation Options</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The Compute Sanitizer tools do not need any special compilation flags to function.
                           
                        </p>
                        <p class="p">
                           The output displayed by the Compute Sanitizer tools is more useful with some extra
                           compiler flags. The <samp class="ph codeph">-G</samp> option to nvcc forces the compiler to
                           generate debug information for the CUDA application. To generate line number
                           information for applications without affecting the optimization level of
                           the output, the <samp class="ph codeph">-lineinfo</samp> nvcc option can be used.
                           The Compute Sanitizer tools fully support both of these options
                           and can display source attribution of errors for applications
                           compiled with line information.
                           
                        </p>
                        <p class="p">
                           The stack backtrace feature of the Compute Sanitizer tools is more useful
                           when the application contains function symbol names. For the host backtrace,
                           this varies based on the host OS. On Linux, the host compiler
                           must be given the <samp class="ph codeph">-rdynamic</samp> option to retain function
                           symbols. On Windows, the application must be compiled for debugging,
                           i.e. the <samp class="ph codeph">/Zi</samp> option. When using nvcc, flags to the host
                           compiler can be specified using the <samp class="ph codeph">-Xcompiler</samp> option.
                           For the device backtrace, the full frame information is only available
                           when the application is compiled with device debug information. The compiler
                           can skip generation of frame information when building with optimizations.
                           
                        </p>
                        <div class="p">
                           Sample command line to build with function symbols and device side line
                           information on Linux:
                           <pre class="pre screen" xml:space="preserve">
nvcc -Xcompiler -rdynamic -lineinfo  -o out in.cu
</pre></div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="environment-variables"><a name="environment-variables" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#environment-variables" name="environment-variables" shape="rect">2.3.&nbsp;Environment Variables</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The following environment variables can be set before launching the compute-sanitizer tool.
                           
                        </p>
                        <div class="tablenoborder"><a name="environment-variables__environment-variables-table" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="environment-variables__environment-variables-table" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 6. Environment Variables</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="33.33333333333333%" id="d54e1630" rowspan="1" colspan="1">Name</th>
                                    <th class="entry" valign="top" width="33.33333333333333%" id="d54e1633" rowspan="1" colspan="1">Description</th>
                                    <th class="entry" valign="top" width="33.33333333333333%" id="d54e1636" rowspan="1" colspan="1">Default/Values</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1630" rowspan="1" colspan="1">
                                       NV_COMPUTE_SANITIZER_BINARY_PATCHING
                                       
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1633" rowspan="1" colspan="1">
                                       <p class="p">
                                          Controls whether compute-sanitizer will instrument user kernel code.
                                          
                                       </p>
                                       <p class="p">
                                          This option is intended for debugging and should not be used by normal users.
                                          
                                       </p>
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1636" rowspan="1" colspan="1">
                                       <p class="p">
                                          Default if unset: enabled.
                                          
                                       </p>
                                       <p class="p">
                                          Valid values: any positive value between 0 and <samp class="ph codeph">INT_MAX</samp>.
                                          
                                       </p>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1630" rowspan="1" colspan="1">
                                       NV_COMPUTE_SANITIZER_LOCAL_CONNECTION_OVERRIDE
                                       
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1633" rowspan="1" colspan="1">
                                       <p class="p">
                                          Override the default local connection mechanism between frontend and target processes.
                                          The default mechanism is platform-dependent.
                                          This should only be used if there are connection problems between frontend and target processes in a local launch.
                                          
                                       </p>
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1636" rowspan="1" colspan="1">
                                       <p class="p">
                                          Default: unset (use default mechanism)
                                          
                                       </p>
                                       <p class="p">
                                          Set to "uds" to use Unix Domain Socket connections (available on Posix platforms, only).
                                          Set to "tcp" to use TCP (available on all platforms).
                                          Set to "named-pipes" to use Windows Named Pipes (available on Windows, only).
                                          
                                       </p>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1630" rowspan="1" colspan="1">
                                       NV_COMPUTE_SANITIZER_MAX_RACECHECK_CLUSTER_RECORDS
                                       
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1633" rowspan="1" colspan="1">
                                       <p class="p">
                                          Override the maximum number of racecheck cluster access records for
                                          early exit race detection. This option can be used to either increase
                                          the number of races the tool can detect, or to suppress early exit
                                          races (0 will display no early exit race).
                                          
                                       </p>
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1636" rowspan="1" colspan="1">
                                       <p class="p">
                                          Default if unset: 100.
                                          
                                       </p>
                                       <p class="p">
                                          Valid values: any positive value between 0 and <samp class="ph codeph">INT_MAX</samp>.
                                          
                                       </p>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1630" rowspan="1" colspan="1">
                                       NV_COMPUTE_SANITIZER_MAX_RACECHECK_HAZARDS
                                       
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1633" rowspan="1" colspan="1">
                                       <p class="p">
                                          Override the maximum number of racecheck hazards tool will process. This
                                          option can be used to either increase the number of races the tool can
                                          detect, or to reduce it and save host memory.
                                          
                                       </p>
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1636" rowspan="1" colspan="1">
                                       <p class="p">
                                          Default if unset: 10,000,000.
                                          
                                       </p>
                                       <p class="p">
                                          Valid values: any positive value between 0 and <samp class="ph codeph">INT_MAX</samp>.
                                          
                                       </p>
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1630" rowspan="1" colspan="1">
                                       NV_COMPUTE_SANITIZER_SHARED_ADDRESSING_SUPPORT
                                       
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1633" rowspan="1" colspan="1">
                                       <p class="p">
                                          Override shared memory addressing support.
                                          
                                       </p>
                                    </td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1636" rowspan="1" colspan="1">
                                       <p class="p">
                                          Default if unset: <samp class="ph codeph">auto</samp>.
                                          
                                       </p>
                                       <p class="p">
                                          Set to <samp class="ph codeph">none</samp> to disable shared addressing support.
                                          Set to <samp class="ph codeph">force</samp> to force shared addressing support.
                                          Set to <samp class="ph codeph">auto</samp> to enable shared memory addressing support if system
                                          supports HMM or ATS.
                                          
                                       </p>
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="memcheck-tool"><a name="memcheck-tool" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#memcheck-tool" name="memcheck-tool" shape="rect">3.&nbsp;Memcheck Tool</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="what-is-memcheck"><a name="what-is-memcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#what-is-memcheck" name="what-is-memcheck" shape="rect">3.1.&nbsp;What is Memcheck?</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">memcheck</dfn> tool is a run time error detection tool for
                           CUDA applications. The tool can precisely detect and report out of bounds
                           and misaligned memory accesses to global, local and shared memory in CUDA
                           applications. It can also detect and report hardware reported error
                           information. In addition, the memcheck tool can detect and report memory
                           leaks in the user application.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="supported-error-detection"><a name="supported-error-detection" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#supported-error-detection" name="supported-error-detection" shape="rect">3.2.&nbsp;Supported Error Detection</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The errors that can be reported by the memcheck tool are summarized in the table
                           below. The location column indicates whether the report originates from the
                           host or from the device. The precision of an error is explained in the paragraph
                           below.
                           
                        </p>
                        <div class="tablenoborder"><a name="supported-error-detection__memcheck-error-types" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="supported-error-detection__memcheck-error-types" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 7. Memcheck reported error types</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="16.666666666666664%" id="d54e1833" rowspan="1" colspan="1">Name</th>
                                    <th class="entry" valign="top" width="33.33333333333333%" id="d54e1836" rowspan="1" colspan="1">Description</th>
                                    <th class="entry" valign="top" width="16.666666666666664%" id="d54e1839" rowspan="1" colspan="1">Location</th>
                                    <th class="entry" valign="top" width="16.666666666666664%" id="d54e1842" rowspan="1" colspan="1">Precision</th>
                                    <th class="entry" valign="top" width="16.666666666666664%" id="d54e1845" rowspan="1" colspan="1">See also</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">Memory access error</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Errors due to
                                       out of bounds or misaligned accesses to memory by a global,
                                       local, shared or global atomic access.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Precise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1">&nbsp;</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">Hardware exception</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Errors that are reported
                                       by the hardware error reporting mechanism.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Imprecise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1">&nbsp;</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">Malloc/Free errors</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Errors that occur due to incorrect
                                       use of <samp class="ph codeph">malloc()/free()</samp>
                                       in CUDA kernels.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Precise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1"><a class="xref" href="index.html#device-side-allocation-checking" shape="rect">Device Side Allocation Checking</a></td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">CUDA API errors</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Reported when a CUDA API call in the application
                                       returns a failure.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Host</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Precise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1"><a class="xref" href="index.html#api-error-checking" shape="rect">CUDA API Error Checking</a></td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">cudaMalloc memory leaks</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Allocations of device memory using <samp class="ph codeph">cudaMalloc()</samp>
                                       that have not been freed by the application.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Host</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Precise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1"><a class="xref" href="index.html#leak-checking" shape="rect">Leak Checking</a></td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1833" rowspan="1" colspan="1"><dfn class="term">Device Heap Memory Leaks</dfn></td>
                                    <td class="entry" valign="top" width="33.33333333333333%" headers="d54e1836" rowspan="1" colspan="1">
                                       Allocations of device memory using <samp class="ph codeph">malloc()</samp>
                                       in device code that have not been freed by the application.
                                       
                                    </td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1839" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1842" rowspan="1" colspan="1">Imprecise</td>
                                    <td class="entry" valign="top" width="16.666666666666664%" headers="d54e1845" rowspan="1" colspan="1"><a class="xref" href="index.html#device-side-allocation-checking" shape="rect">Device Side Allocation Checking</a></td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <p class="p">
                           The memcheck tool reports two classes of errors
                           <dfn class="term">precise</dfn> and <dfn class="term">imprecise</dfn>.
                           
                        </p>
                        <p class="p"><dfn class="term">Precise</dfn> errors in memcheck are those that the tool can uniquely
                           identify and gather all information for.
                           For these errors, memcheck can report the block and thread coordinates
                           of the thread causing the failure, the program counter (PC) of the instruction performing the
                           access, as well as the address being accessed and its size and type. If the CUDA
                           application contains line number information (by either being compiled with device
                           side debugging information, or with line information), then the tool will also
                           print the source file and line number of the erroneous access.
                           
                        </p>
                        <p class="p"><dfn class="term">Imprecise</dfn> errors are errors reported by the hardware
                           error reporting mechanism that could not be precisely attributed to a particular
                           thread. The precision of the error varies based on the type of the error
                           and in many cases, memcheck may not be able to attribute the cause
                           of the error back to the source file and line.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="using-memcheck"><a name="using-memcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#using-memcheck" name="using-memcheck" shape="rect">3.3.&nbsp;Using Memcheck</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           The memcheck tool is enabled by default when running the
                           Compute Sanitizer application. It can also be explicitly enabled by using
                           the <samp class="ph codeph">--tool memcheck</samp> option.
                           <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer --tool memcheck [sanitizer_options] app_name [app_options]</strong>
</pre></div>
                        <p class="p"> When run in this way, the memcheck tool will look for precise, imprecise, malloc/free
                           and CUDA API errors. The reporting of device leaks must be explicitly enabled. Errors
                           identified by the memcheck tool are displayed on the screen after the application has
                           completed execution. See <a class="xref" href="index.html#understanding-memcheck-errors" shape="rect">Understanding Memcheck Errors</a> for more
                           information about how to interpret the messages printed by the tool. 
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="understanding-memcheck-errors"><a name="understanding-memcheck-errors" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#understanding-memcheck-errors" name="understanding-memcheck-errors" shape="rect">3.4.&nbsp;Understanding Memcheck Errors</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The memcheck tool can produce a variety of different errors. This is a short
                           guide showing some samples of errors and explaining how the information
                           in each error report can be interpreted.
                           
                        </p>
                        <ol class="ol">
                           <li class="li">
                              <div class="p"><dfn class="term">Memory access error</dfn>: Memory access errors are generated for errors
                                 that the memcheck tool can correctly attribute and identify the erroneous
                                 instruction. Below is an example of a precise memory access error.
                                 <pre class="pre screen" xml:space="preserve">
========= Invalid __global__ write of size 4 bytes
=========     at unaligned_kernel():0x160 in memcheck_demo.cu:6
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x7f6510c00001 is misaligned
</pre></div>
                              <div class="p">
                                 Let us examine this error line by line:
                                 <pre class="pre screen" xml:space="preserve">Invalid __global__ write of size 4 bytes</pre>
                                 
                                 The first line shows the memory segment, type and size being accessed.
                                 The memory segment is one of:
                                 <ul class="ul">
                                    <li class="li">__global__ : for device global memory</li>
                                    <li class="li">__shared__ : for per block shared memory</li>
                                    <li class="li">__local__  : for per thread local memory</li>
                                 </ul>
                                 
                                 In this case, the access was to device global memory.
                                 The next field contains information about the type of access,
                                 whether it was a read or a write. In this case, the access is a write.
                                 Finally, the last item is the size of the access in bytes. In this
                                 example, the access was 4 bytes in size.
                                 
                              </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">at unaligned_kernel():0x160 in memcheck_demo.cu:6</pre>
                                 
                                 The second line contains the CUDA kernel name, offset of the instruction, the source file
                                 and line number (if available).
                                 In this example, the instruction causing the access was at
                                 offset 0x160 inside the <samp class="ph codeph">unaligned_kernel</samp> CUDA kernel.
                                 Additionally, since the application was compiled with line number
                                 information, this instruction corresponds to line 6 in the memcheck_demo.cu
                                 source file.
                                 
                              </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">by thread (0,0,0) in block (0,0,0)</pre>
                                 
                                 The third line contains the thread indices and block indices of the
                                 thread on which the error was hit.
                                 In this example, the thread doing the erroneous access belonged to
                                 the first thread in the first block.
                                 </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">Address 0x7f6510c00001 is misaligned</pre>
                                 The fourth line contains the
                                 memory address being accessed and the type of access error. The type of access
                                 error can either be out of bounds access or misaligned access. In this example,
                                 the access was to address 0x7f6510c00001 and the access error was because this
                                 address was not aligned correctly. </div>
                              <p class="p"></p>
                           </li>
                           <li class="li">
                              <p class="p"><dfn class="term">Hardware exception</dfn>: Imprecise errors are generated for errors that
                                 the hardware reports to the memcheck tool. Hardware exceptions have a variety
                                 of formats and messages. Typically, the first line will provide some information
                                 about the type of error encountered.
                                 
                              </p>
                              <p class="p"></p>
                           </li>
                           <li class="li">
                              <div class="p"><dfn class="term">Malloc/free error</dfn>: Malloc/free errors refer to the errors in the
                                 invocation of device side <samp class="ph codeph">malloc()/free()</samp> in CUDA kernels. An
                                 example of a malloc/free error:
                                 <pre class="pre screen" xml:space="preserve">
========= Malloc/Free error encountered : Double free
=========     at 0x79d8
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x400aff920
</pre></div>
                              <div class="p"> We can examine this line by line.
                                 <pre class="pre screen" xml:space="preserve">Malloc/Free error encountered : Double free</pre>
                                 The first line
                                 indicates that this is a malloc/free error, and contains the type of error. This
                                 type can be: <ul class="ul">
                                    <li class="li">Double free – This indicates that the thread called
                                       <samp class="ph codeph">free()</samp> on an allocation that has already been
                                       freed. 
                                    </li>
                                    <li class="li">Invalid pointer to free – This indicates that <samp class="ph codeph">free</samp> was
                                       called on a pointer that was not returned by <samp class="ph codeph">malloc()</samp>. 
                                    </li>
                                    <li class="li">Heap corruption : This indicates generalized heap corruption, or cases
                                       where the state of the heap was modified in a way that memcheck did not
                                       expect.  
                                    </li>
                                 </ul>
                                 In this example, the error is due to calling <samp class="ph codeph">free()</samp> on a
                                 pointer which had already been freed. 
                              </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">at 0x79d8</pre>
                                 
                                 The second line gives the PC on GPU where the error was reported.
                                 This PC is usually inside of system code, and is not interesting
                                 to the user. The device frame backtrace will contain the location
                                 in user code where the <samp class="ph codeph">malloc()/free()</samp> call was
                                 made.
                                 
                              </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">by thread (0,0,0) in block (0,0,0)</pre>
                                 
                                 The third line contains the thread and block indices of the thread
                                 that caused this error. In this example, the thread has threadIdx = (0,0,0)
                                 and blockIdx = (0,0,0)
                                 </div>
                              <div class="p"><pre class="pre screen" xml:space="preserve">Address 0x400aff920</pre>
                                 
                                 This line contains the value of the pointer passed to <samp class="ph codeph">free()</samp>
                                 or returned by <samp class="ph codeph">malloc()</samp></div>
                              <p class="p"></p>
                           </li>
                           <li class="li">
                              <div class="p"><dfn class="term">Leak errors</dfn>: Errors are reported for allocations created using
                                 cudaMalloc and for allocations on the device heap that were not freed when the
                                 CUDA context was destroyed. An example of a cudaMalloc allocation leak report is
                                 the following: <pre class="pre screen" xml:space="preserve">
========= Leaked 64 bytes at 0x400200200
</pre>
                                 The
                                 error message reports information about the size of the allocation that was
                                 leaked as well as the address of the allocation on the device. </div>
                              <div class="p">
                                 A device heap leak message will be explicitly identified as such:
                                 <pre class="pre screen" xml:space="preserve">
========= Leaked 16 bytes at 0x4012ffff6 on the device heap
</pre></div>
                              <p class="p"></p>
                           </li>
                           <li class="li">
                              <div class="p"><dfn class="term">CUDA API error</dfn>: CUDA API errors are reported for CUDA
                                 API calls that return an error value. An example of a CUDA API error:
                                 <pre class="pre screen" xml:space="preserve">
========= Program hit invalid copy direction for memcpy (error 21) on CUDA API call to cudaMemcpy.
</pre>
                                 
                                 The message contains the returned value of the CUDA API call, as well as
                                 the name of the API function that was called.
                                 </div>
                           </li>
                        </ol>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="api-error-checking"><a name="api-error-checking" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#api-error-checking" name="api-error-checking" shape="rect">3.5.&nbsp;CUDA API Error Checking</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The memcheck tool supports reporting an error if a CUDA API call made by the user
                           program returned an error. The tool supports this detection for both
                           CUDA run time and CUDA driver API calls. In all cases, if the API function
                           call has a nonzero return value, Compute Sanitizer will print an error message
                           containing the name of the API call that failed and the return value of the
                           API call.
                           
                        </p>
                        <p class="p">
                           CUDA API error reports do not terminate the application, they merely provide
                           extra information. It is up to the application to check the
                           return status of CUDA API calls and handle error conditions appropriately.
                           
                        </p>
                        <div class="p">
                           The following API errors are not reported:
                           
                           <ul class="ul">
                              <li class="li"><samp class="ph codeph">cudaErrorNotReady</samp> for <samp class="ph codeph">cudaEventQuery</samp> and <samp class="ph codeph">cudaStreamQuery</samp> APIs.
                              </li>
                              <li class="li"><samp class="ph codeph">cudaErrorPeerAccessAlreadyEnabled</samp> for <samp class="ph codeph">cudaDeviceEnablePeerAccess</samp> API.
                              </li>
                              <li class="li"><samp class="ph codeph">cudaErrorPeerAccessNotEnabled</samp> for <samp class="ph codeph">cudaDeviceDisablePeerAccess</samp> API.
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="device-side-allocation-checking"><a name="device-side-allocation-checking" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#device-side-allocation-checking" name="device-side-allocation-checking" shape="rect">3.6.&nbsp;Device Side Allocation Checking</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">memcheck</dfn> tool checks accesses to allocations in the device heap.
                           
                        </p>
                        <p class="p">
                           These allocations are created by calling <samp class="ph codeph">malloc()</samp> inside a kernel.
                           This feature is implicitly enabled and can be disabled by specifying the
                           <samp class="ph codeph">--check-device-heap no</samp> option. This
                           feature is only activated for kernels in the application that call
                           <samp class="ph codeph">malloc()</samp>.
                           
                        </p>
                        <div class="p"> The tool will report an error if the application calls a <samp class="ph codeph">free()</samp> twice
                           for the same allocation, or if it calls <samp class="ph codeph">free()</samp> on an invalid pointer.
                           
                           <div class="note note"><span class="notetitle">Note:</span>  Make sure to look at the device side backtrace to find the location in the
                              application where the <samp class="ph codeph">malloc()/free()</samp> call was made.  
                           </div>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="leak-checking"><a name="leak-checking" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#leak-checking" name="leak-checking" shape="rect">3.7.&nbsp;Leak Checking</a></h3>
                     <div class="body conbody">
                        <p class="p">The <dfn class="term">memcheck</dfn> tool can detect leaks of allocated memory.
                        </p>
                        <p class="p">Memory leaks are device side allocations that have not been freed by the time
                           the context is destroyed. The <dfn class="term">memcheck</dfn> tool tracks  device memory
                           allocations created using the CUDA driver or runtime APIs. 
                        </p>
                        <p class="p">The <samp class="ph codeph">--leak-check full</samp> option must be specified to enable
                           leak checking.
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="padding"><a name="padding" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#padding" name="padding" shape="rect">3.8.&nbsp;Padding</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">memcheck</dfn> tool can automatically add padding to memory allocations in order to
                           improve out of bounds error detection for global memory.
                           
                        </p>
                        <p class="p">
                           By default, global memory buffers can be allocated back-to-back in the virtual address space. When
                           that happens, an overflow access into the first buffer will simply happen in the second buffer
                           and not be detected as out-of-bounds.
                           
                        </p>
                        <div class="fig fignone"><span class="desc figdesc">Example of device buffers allocated back-to-back:</span><br clear="none"></br><div class="imagecenter"><img class="image imagecenter" src="graphics/no-padding.png"></img></div><br clear="none"></br></div>
                        <p class="p">
                           Using the <samp class="ph codeph">--padding</samp> option will automatically extend the allocation size,
                           effectively creating a padding buffer after each allocation. This improves the out of bounds
                           error detection as accesses to the padding area will always be considered invalid. The example
                           below displays possible buffer addresses when using <samp class="ph codeph">--padding 32</samp>. Every allocation
                           is followed by a 32 bytes padding buffer. Writing or reading this buffer will cause an out-of-bounds
                           access to be reported.
                           
                        </p>
                        <div class="fig fignone"><span class="desc figdesc">Example of device buffers allocated with padding:</span><br clear="none"></br><div class="imagecenter"><img class="image imagecenter" src="graphics/padding.png"></img></div><br clear="none"></br></div>
                        <p class="p">
                           This option supports allocations created via the <samp class="ph codeph">cudaMalloc</samp> APIs, <samp class="ph codeph">cudaHostAlloc</samp>
                           and <samp class="ph codeph">cudaMallocHost</samp>.
                           
                        </p>
                        <p class="p">
                           This option does not support allocations created via <samp class="ph codeph">cudaHostRegister</samp> or the CUDA virtual
                           memory management APIs.
                           
                        </p>
                        <p class="p">
                           Be aware that using this option will result in an increased device memory pressure, potentially causing
                           additional CUDA out of memory errors.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="stream-ordered-races"><a name="stream-ordered-races" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#stream-ordered-races" name="stream-ordered-races" shape="rect">3.9.&nbsp;Stream-ordered race detection</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">memcheck</dfn> tool can detect stream-ordered allocations
                           races using the <samp class="ph codeph">--track-stream-ordered-races all</samp> option.
                           It will report accesses to stream-ordered allocations used outside
                           of their lifespan.
                           
                        </p>
                        <div class="p">
                           The tool is capable of detecting 2 types of races:
                           
                           <ul class="ul">
                              <li class="li">
                                 Use-before-alloc races (<samp class="ph codeph">--track-stream-ordered-races use-before-alloc</samp>)
                                 
                                 <p class="p">
                                    This race occurs when an allocation is used before it
                                    is available: an allocation created using
                                    <samp class="ph codeph">cudaMallocAsync</samp> on a stream cannot be
                                    used on another stream without a prior synchronization
                                    event after the allocation.
                                    
                                 </p>
                                 <p class="p">
                                    It also includes cases where an allocation is freed
                                    before it is available using <samp class="ph codeph">cudaFreeAsync</samp>.
                                    
                                 </p><br clear="none"></br><div class="imagecenter"><img class="image imagecenter" src="graphics/use-before-alloc.png" width="700"></img></div><br clear="none"></br></li>
                              <li class="li">
                                 Use-after-free races (<samp class="ph codeph">--track-stream-ordered-races use-after-free</samp>)
                                 
                                 <p class="p">
                                    This race occurs when an allocation is used after it is
                                    freed: an allocation freed using
                                    <samp class="ph codeph">cudaFreeAsync</samp> on a stream cannot be
                                    used on another stream without a following
                                    synchronization event before the free.
                                    
                                 </p><br clear="none"></br><div class="imagecenter"><img class="image imagecenter" src="graphics/use-after-free.png" width="700"></img></div><br clear="none"></br></li>
                           </ul>
                        </div>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="racecheck-tool"><a name="racecheck-tool" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#racecheck-tool" name="racecheck-tool" shape="rect">4.&nbsp;Racecheck Tool</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="what-is-racecheck"><a name="what-is-racecheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#what-is-racecheck" name="what-is-racecheck" shape="rect">4.1.&nbsp;What is Racecheck?</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">racecheck</dfn> tool is a run time shared memory data access hazard
                           detector. The primary use of this tool is to help identify memory access
                           race conditions in CUDA applications that use shared memory.
                           
                        </p>
                        <p class="p"> In CUDA applications, storage declared with the <samp class="ph codeph">__shared__</samp> qualifier is
                           placed on chip <dfn class="term">shared memory</dfn>. All threads in a thread block can access this
                           per block shared memory. Shared memory goes out of scope when the thread block completes
                           execution. As shared memory is on chip, it is frequently used for inter-thread
                           communication and as a temporary buffer to hold data being processed. As this data is
                           being accessed by multiple threads in parallel, incorrect program assumptions may result
                           in data races. Racecheck is a tool built to identify these hazards and help users write
                           programs free of shared memory races. 
                        </p>
                        <p class="p">
                           Currently, this tool only supports detecting accesses to on-chip shared memory.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="what-are-hazards"><a name="what-are-hazards" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#what-are-hazards" name="what-are-hazards" shape="rect">4.2.&nbsp;What are Hazards?</a></h3>
                     <div class="body conbody">
                        <p class="p"> A <dfn class="term">data access hazard</dfn> is a case where two threads attempt to access the same
                           location in memory resulting in non-deterministic behavior, based on the relative order
                           of the two accesses. These hazards cause <dfn class="term">data races</dfn> where the behavior or
                           the output of the application depends on the order in which all parallel threads are
                           executed by the hardware. Race conditions manifest as intermittent application failures
                           or as failures when attempting to run a working application on a different GPU. 
                        </p>
                        <div class="p">
                           The racecheck tool identifies three types of canonical hazards in a program.
                           These are :
                           
                           <ul class="ul">
                              <li class="li"> Write-After-Write (<dfn class="term">WAW</dfn>) hazards
                                 
                                 <p class="p"> This hazard occurs when two threads attempt to
                                    write data to the same memory location. The resulting value
                                    in that location depends on the relative order of the two
                                    accesses.
                                    
                                 </p>
                              </li>
                              <li class="li"> Write-After-Read (<dfn class="term">WAR</dfn>) hazards
                                 
                                 <p class="p"> This hazard occurs when two threads access the same memory location,
                                    with one thread performing a read and another a write. In
                                    this case, the writing thread is ordered before the reading
                                    thread and the value returned to the reading thread is
                                    not the original value at the memory location.
                                    
                                 </p>
                              </li>
                              <li class="li"> Read-After-Write (<dfn class="term">RAW</dfn>) hazards
                                 
                                 <p class="p"> This hazard occurs when two threads access the same memory
                                    location, with one thread performing a read and the other a write.
                                    In this case, the reading thread reads the value before the
                                    writing thread commits it.
                                    
                                 </p>
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="using-racecheck"><a name="using-racecheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#using-racecheck" name="using-racecheck" shape="rect">4.3.&nbsp;Using Racecheck</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           The racecheck tool is enabled by running the Compute Sanitizer application
                           with the <samp class="ph codeph">--tool racecheck</samp> option.
                           <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer --tool racecheck [sanitizer_options] app_name [app_options]</strong>
</pre></div>
                        <p class="p">
                           Once racecheck has identified a hazard, the user can make program modifications
                           to ensure this hazard is no longer present.
                           In the case of Write-After-Write hazards, the program should be modified
                           so that multiple writes are not happening to the same location.
                           In the case of Read-After-Write and Write-After-Read hazards, the reading
                           and writing locations should be deterministically ordered. In CUDA kernels,
                           this can be achieved by inserting a <samp class="ph codeph">__syncthreads()</samp> call
                           between the two accesses. To avoid races between threads within a single warp,
                           <samp class="ph codeph">__syncwarp()</samp> can be used.
                           
                        </p>
                        <div class="note note"><span class="notetitle">Note:</span> The racecheck tool does not perform any memory access error checking. It is
                           recommended that users first run the memcheck tool to ensure the application is free of
                           errors.  
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="racecheck-report-modes"><a name="racecheck-report-modes" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-report-modes" name="racecheck-report-modes" shape="rect">4.4.&nbsp;Racecheck Report Modes</a></h3>
                     <div class="body conbody">
                        <div class="p"> The racecheck tool can produce two types of output: 
                           <ul class="ul">
                              <li class="li"><dfn class="term">Hazard</dfn> reports 
                                 <p class="p"> These reports contain detailed information about
                                    one particular hazard. Each hazard report is byte accurate and represents
                                    information about conflicting accesses between two threads that affect this
                                    byte of shared memory. 
                                 </p>
                              </li>
                              <li class="li"><dfn class="term">Analysis</dfn> reports 
                                 <p class="p"> These reports contain a post analysis set of
                                    reports. These reports are produced by the racecheck tool by analysing
                                    multiple hazard reports and examining active device state. For example usage
                                    of analysis reports, see <a class="xref" href="index.html#understanding-racecheck-analysis-reports" shape="rect">Understanding Racecheck Analysis Reports</a>. 
                                 </p>
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="understanding-racecheck-analysis-reports"><a name="understanding-racecheck-analysis-reports" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#understanding-racecheck-analysis-reports" name="understanding-racecheck-analysis-reports" shape="rect">4.5.&nbsp;Understanding Racecheck Analysis Reports</a></h3>
                     <div class="body conbody">
                        <p class="p"> In <dfn class="term">analysis</dfn> reports, the racecheck tool produces a series of high-level
                           messages that identify the source locations of a particular race, based on observed
                           hazards and other machine state. 
                        </p>
                        <div class="p">
                           A sample racecheck analysis report is below:
                           
                           <pre class="pre screen" xml:space="preserve">
========= WARNING: Race reported between Write access at RAW()+0xf0 in raceGroupBasic.cu:40
=========     and Read access at RAW()+0x280 in raceGroupBasic:46 [4 hazards]
</pre></div>
                        <p class="p"> The analysis record contains high-level information about the hazard that is conveyed to
                           the end user. Each line contains information about a unique location in the application
                           which is participating in the race. 
                        </p>
                        <p class="p">
                           The first word on the first line indicates the severity of this report.
                           In this case, the message is at the WARNING level of severity.
                           For more information on the different severity levels, see <a class="xref" href="index.html#racecheck-severity-levels" shape="rect">Racecheck Severity Levels</a>.
                           Analysis reports are composed of one or more racecheck hazards, and the severity level
                           of the report is that of the hazard with the highest severity.
                           
                        </p>
                        <div class="p"> The first line additionally contains the type of access. The access can be either: 
                           <ul class="ul">
                              <li class="li">Read</li>
                              <li class="li">Write</li>
                           </ul>
                           
                           The next item on the line is the name of the kernel issuing the access and
                           the offset of the location where the access happened
                           from. In this case, the offset is 0xf0 in the RAW() kernel. If the application was compiled with line number
                           information, this line also contains the file name and line number of the access.
                           
                        </div>
                        <p class="p">
                           The next lines contain the location of the other offsets participating in the race condition.
                           In this case, there is only one other location which is the RAW() kernel at offset 0x280.
                           Similarly to the first line, file name and line number are printed if the application was
                           compiled with line number information.
                           Finally, the line also contains the number of hazards detected for this specific race condition.
                           
                        </p>
                        <p class="p">
                           A given analysis report will always contain at least one line which is performing a write
                           access. A common strategy to eliminate races which contain only write accesses is to
                           ensure that the write access is performed by only one thread. In the case of races
                           with multiple readers and one writer, introducing explicit program ordering
                           via a <samp class="ph codeph">__syncthreads()</samp> call can avoid the race condition.
                           For races between threads within the same warp, the <samp class="ph codeph">__syncwarp()</samp>
                           intrinsic can be used to avoid the hazard.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="understanding-racecheck-hazard-reports"><a name="understanding-racecheck-hazard-reports" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#understanding-racecheck-hazard-reports" name="understanding-racecheck-hazard-reports" shape="rect">4.6.&nbsp;Understanding Racecheck Hazard Reports</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           In <dfn class="term">hazard</dfn> reporting mode, the racecheck tool produces a series of messages
                           detailing information about hazards in the application. The tool is byte accurate and
                           produces a message for each byte on which a hazard was detected. Additionally, when enabled,
                           the host backtrace for the launch of the kernel will also be displayed.
                           
                        </p>
                        <div class="p">
                           A sample racecheck hazard is below:
                           <pre class="pre screen" xml:space="preserve">
========= ERROR: Potential WAW hazard detected at __shared__ 0x0 in block (0,0,0) :
=========     Write Thread (0,0,0) at WAW()+0x2f0 in raceWAW.cu:20
=========     Write Thread (1,0,0) at WAW()+0x2f0 in raceWAW.cu:20
=========     Current Value : 1, Incoming Value : 2
</pre></div>
                        <p class="p">
                           The hazard records are dense and capture a lot of interesting information.
                           In general terms, the first line contains information about the hazard
                           severity, type and address, as well as information about the thread
                           block where it occurred.
                           The next 2 lines contain detailed information about the two threads that were
                           in contention. These two lines are ordered chronologically, so the first entry
                           is for the access that occurred earlier and the second for the access that
                           occurred later. The final line is printed for some hazard types and captures
                           the actual data that was being written.
                           
                        </p>
                        <div class="p">
                           Examining this line by line, we have :
                           <pre class="pre screen" xml:space="preserve">ERROR: Potential WAW hazard detected at __shared__ 0x0 in block (0, 0, 0)</pre></div>
                        <p class="p">
                           The first word on this line indicates the severity of this hazard.
                           In this case, the message is at the ERROR level of severity.
                           For more information on the different severity levels, see <a class="xref" href="index.html#racecheck-severity-levels" shape="rect">Racecheck Severity Levels</a>.
                           
                        </p>
                        <div class="p">
                           The next piece of information here is the type of hazard. The racecheck tool
                           detects three types of hazards:
                           
                           <ul class="ul">
                              <li class="li">WAW or Write-After-Write hazards</li>
                              <li class="li">WAR or Write-After-Read hazards</li>
                              <li class="li">RAW or Read-After-Write hazards</li>
                           </ul>
                           
                           The type of hazard indicates the accesses types of the two threads that were in
                           contention. In this example, the hazard is of Write-After-Write type.
                           
                        </div>
                        <p class="p">
                           The next piece of information is the address in shared memory that was being
                           accessed. This is the offset in per block shared memory that was being accessed
                           by both threads. Since the racecheck tool is byte accurate, the message is only
                           for the byte of memory at given address. In this example, the byte being accessed
                           is byte 0x0 in shared memory.
                           
                        </p>
                        <p class="p">
                           Finally, the first line contains the block index of the thread block to which
                           the two racing threads belong.
                           
                        </p>
                        <div class="p"> The second line contains information about the first thread to write to this location.
                           <pre class="pre screen" xml:space="preserve">Write Thread (0, 0, 0) at WAW()+0x2f0 in raceWAW.cu:20(void)</pre>
                           The first
                           item on this line indicates the type of access being performed by this thread to the
                           shared memory address. In this example, the thread was writing to the location. The next
                           component is the index of the thread block. In this case, the thread is at index
                           (0,0,0). Following this, we have the name of the kernel and byte offset of the instruction
                           which did the access in the kernel. In this example, the offset is 0x2f0.
                           This is followed by the source file and line number (if line number information is available).</div>
                        <p class="p"> The third line contains similar information about the second thread that was causing
                           this hazard. This line has an identical format to the previous line. 
                        </p>
                        <div class="p"> The fourth line contains information about the data in the two accesses.
                           <pre class="pre screen" xml:space="preserve">Current Value : 1, Incoming Value : 2</pre>
                           If the second thread in the
                           hazard was performing a write access, i.e., the hazard is a Write-After-Write (WAW) or a
                           Write-After-Read (WAR), this line contains the value after the access by the first
                           thread as the <dfn class="term">Current Value</dfn> and the value that will be written by the
                           second access as the <dfn class="term">Incoming Value</dfn>. In this case, the first thread wrote
                           the value 1 to the shared memory location. The second thread is attempting to write the
                           value 2. 
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="racecheck-severity-levels"><a name="racecheck-severity-levels" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-severity-levels" name="racecheck-severity-levels" shape="rect">4.7.&nbsp;Racecheck Severity Levels</a></h3>
                     <div class="body conbody">
                        <p class="p"> Problems reported by racecheck can be of different severity levels. Depending on the
                           level, different actions are required from developers. By default, only issues of
                           severity level WARNING and ERROR are shown. The command line option
                           <samp class="ph codeph">--print-level</samp> can be used to set the lowest severity level that
                           should be reported. 
                        </p>
                        <div class="p">
                           Racecheck reports have one of the following severity levels:
                           
                           <ul class="ul">
                              <li class="li"><dfn class="term">INFO</dfn>: The lowest level of severity. This is for hazards that have
                                 no impact on program execution and hence are not contributing to data access
                                 hazards. It is still a good idea to find and eliminate such hazards. 
                              </li>
                              <li class="li">
                                 <p class="p"><dfn class="term">WARNING</dfn>: Hazards at this level of severity are determined to be
                                    programming model hazards, however may be intentionally created by the
                                    programmer. An example of this are hazards due to warp level programming
                                    that make the assumption that threads are proceeding in groups. Such hazards
                                    are typically only encountered by advanced programmers. In cases where a
                                    beginner programmer encounters such errors, he should treat them as sources
                                    of hazards. 
                                 </p>
                                 <p class="p">
                                    Starting with the Volta architecture, programmers cannot rely
                                    anymore on the assumption that threads within a warp execute in
                                    lock-step unconditionally. As a result, warnings due to warp-synchronous
                                    programming without explicit synchronization must be fixed when
                                    developing or porting applications from earlier architectures
                                    to Volta and above. Developers can use the <samp class="ph codeph">__syncwarp()</samp> intrinsic
                                    or the Cooperative Groups API.
                                    
                                 </p>
                              </li>
                              <li class="li"><em class="ph i">ERROR</em>: The highest level of severity. This corresponds to hazards that
                                 are very likely candidates for causing data access races. Programmers would be
                                 well advised to examine errors at this level of severity. 
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="racecheck-cuda-barrier"><a name="racecheck-cuda-barrier" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-cuda-barrier" name="racecheck-cuda-barrier" shape="rect">4.8.&nbsp;Racecheck support for <samp class="ph codeph">cuda::barrier</samp></a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Racecheck supports synchronization through <samp class="ph codeph">cuda::barrier</samp> on Ampere GPUs and newer.
                           
                        </p>
                        <div class="p">
                           The number of barriers tracked by the tool is based on the number of barriers present in the source
                           code as reported by compiler information. In some cases, the compiler may undercount this number.
                           Racecheck will report the following warning if more barriers are used than expected:
                           <pre class="pre screen" xml:space="preserve">========= Warning: Detected overflow of tracked cuda::barrier structures. Results might be incorrect. Try using --num-cuda-barriers to fix the issue</pre></div>
                        <p class="p">
                           The <samp class="ph codeph">--num-cuda-barriers</samp> option can be used to indicate the number of expected barriers
                           in the source code and workaround this issue.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="racecheck-asynchronous-copy"><a name="racecheck-asynchronous-copy" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-asynchronous-copy" name="racecheck-asynchronous-copy" shape="rect">4.9.&nbsp;Racecheck support for asynchronous copy</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Racecheck supports race detection on shared memory for asynchronous memory copy operations
                           from global to shared memory introduced in compute capability 8.0. These can take the form of
                           CUDA C++ <samp class="ph codeph">cuda::memcpy_async</samp> or the PTX <samp class="ph codeph">cp.async</samp>. Specifically,
                           racecheck is able to detect when the target of a asynchronous copy tracked by a pipeline (CUDA C++)
                           or async-group (PTX) was accessed before the required commit/wait to guarantee its completion.
                           In these cases, individual hazards when using <samp class="ph codeph">--racecheck-report hazard</samp> will bear the mention
                           <samp class="ph codeph">(invalid memcpy_async synchronization)</samp>.
                           These checks can be disabled by using <samp class="ph codeph">--racecheck-memcpy-async no</samp>.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="racecheck-cluster-races"><a name="racecheck-cluster-races" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-cluster-races" name="racecheck-cluster-races" shape="rect">4.10.&nbsp;Racecheck cluster entry and exit race detection</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Racecheck supports race detection on remote shared memory accesses
                           without appropriate cluster-wide synchronization. When a kernel
                           makes a remote shared memory access from one block to another (in
                           the same cluster), it needs to guarantee that the target block
                           exists, otherwise error <samp class="ph codeph">cudaErrorLaunchFailure</samp> is
                           raised. One way to achieve this is using cluster.sync() from the
                           <a class="xref" href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cluster-group" target="_blank" shape="rect">
                              Cluster Group API
                              </a>. Refer to the
                           <a class="xref" href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#distributed-shared-memory" target="_blank" shape="rect">
                              CUDA documentation about distributed shared memory
                              </a> for more information.
                           
                        </p>
                        <div class="p">
                           When running a program under Racecheck, instead of failing,
                           the tool will report these two types of illegal accesses:
                           
                           <ol class="ol">
                              <li class="li">
                                 Late entry race detection: a block is trying to access
                                 shared memory from another block in the cluster without an
                                 appropriate cluster-wide synchronization beforehand.
                                 
                              </li>
                              <li class="li">
                                 Early exit race detection: a block is trying to access
                                 shared memory from another block in the cluster without an
                                 appropriate cluster-wide synchronization before the target
                                 block exits.
                                 
                              </li>
                           </ol>
                           
                           A sample report for both races is below:
                           <pre class="pre screen" xml:space="preserve">
========= Potential invalid __shared__ read of size 4 bytes
=========     at RemoteAccess(int *, int)+0x170 in RaceCluster.cu:10
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x1000400 is located in a block that might not have entered yet
=========
========= Potential invalid __shared__ read of size 4 bytes
=========     at RemoteAccess(int *, int)+0x170 in RaceCluster.cu:10
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x1000400 is located in a block that might have already exited
=========
</pre></div>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="initcheck-tool"><a name="initcheck-tool" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#initcheck-tool" name="initcheck-tool" shape="rect">5.&nbsp;Initcheck Tool</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="what-is-initcheck"><a name="what-is-initcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#what-is-initcheck" name="what-is-initcheck" shape="rect">5.1.&nbsp;What is Initcheck? </a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">initcheck</dfn> tool is a run time uninitialized device global
                           memory access detector. This tool can identify when device global memory
                           is accessed without it being initialized via device side writes, or via
                           CUDA memcpy and memset API calls.
                           
                        </p>
                        <p class="p">
                           Currently, this tool only supports detecting accesses to device global memory.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="using-initcheck"><a name="using-initcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#using-initcheck" name="using-initcheck" shape="rect">5.2.&nbsp;Using Initcheck</a></h3>
                     <div class="body conbody">
                        <div class="p"> The initcheck tool is enabled by running the Compute Sanitizer application with the
                           <samp class="ph codeph">--tool initcheck</samp> option.
                           <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer --tool initcheck [sanitizer_options] app_name [app_options]</strong>
</pre></div>
                        <div class="note note"><span class="notetitle">Note:</span> The initcheck tool does not perform any memory access error checking. It is
                           recommended that users first run the memcheck tool to ensure the application is free of
                           errors.  
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="unused-memory"><a name="unused-memory" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#unused-memory" name="unused-memory" shape="rect">5.3.&nbsp;Unused memory detection</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           The initcheck tool can also be used to detect unused memory by using the <samp class="ph codeph">--track-unused-memory</samp> option.
                           <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer --tool initcheck --track-unused-memory yes app_name [app_options]</strong>
</pre></div>
                        <div class="p">
                           A sample unused memory report is below:
                           <pre class="pre screen" xml:space="preserve">
=========  Unused memory in allocation 0x7fed9f400000 of size 100 bytes
=========     Not written 80 bytes at offset 0x14 (0x7fed9f400014)
=========     80% of allocation were unused.
</pre></div>
                        <p class="p">
                           This report contains the address and size of the allocation, the number of bytes not used and their location.
                           The location can be a range if all unused bytes are not contiguous.
                           
                        </p>
                        <p class="p">
                           The behavior for this feature can be adjusted with the <samp class="ph codeph">--unused-memory-threshold</samp> option which takes the
                           minimum percentage at which reports should be printed. For instance, using a value of 81 or above would silence the
                           sample report above.
                           
                        </p>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="synccheck-tool"><a name="synccheck-tool" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#synccheck-tool" name="synccheck-tool" shape="rect">6.&nbsp;Synccheck Tool</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="what-is-synccheck"><a name="what-is-synccheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#what-is-synccheck" name="what-is-synccheck" shape="rect">6.1.&nbsp;What is Synccheck?</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The <dfn class="term">synccheck</dfn> tool is a runtime tool that can identify whether
                           a CUDA application is correctly using synchronization primitives, specifically
                           <samp class="ph codeph">__syncthreads()</samp> and <samp class="ph codeph">__syncwarp()</samp> intrinsics
                           and their Cooperative Groups API counterparts.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="using-synccheck"><a name="using-synccheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#using-synccheck" name="using-synccheck" shape="rect">6.2.&nbsp;Using Synccheck</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           The synccheck tool is enabled by running the Compute Sanitizer application
                           with the <samp class="ph codeph">--tool synccheck</samp> option.
                           <pre class="pre screen" xml:space="preserve">
<strong class="ph b">compute-sanitizer --tool synccheck [sanitizer_options] app_name [app_options]</strong>
</pre></div>
                        <div class="note note"><span class="notetitle">Note:</span> The synccheck tool does not perform any memory access error checking. It is
                           recommended that users first run the memcheck tool to ensure the application is free of
                           errors. 
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="understanding-synccheck-reports"><a name="understanding-synccheck-reports" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#understanding-synccheck-reports" name="understanding-synccheck-reports" shape="rect">6.3.&nbsp;Understanding Synccheck Reports</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           For each violation, the synccheck tool produces a report message that identifies
                           the source location of the violation and its classification.
                           
                        </p>
                        <div class="p">
                           A sample synccheck report is below:
                           
                           <pre class="pre screen" xml:space="preserve">
========= Barrier error detected. Divergent thread(s) in warp
=========     at ThreadDivergence(int *, int)+0xf0 in divergence.cu:79
=========     by thread (37,0,0) in block (0,0,0)
</pre></div>
                        <div class="p"> Each report starts with "Barrier error detected." In most cases, this is followed by a
                           classification of the detected barrier error. In this message, a CUDA block with
                           divergent threads was found. The following error classes can be reported: 
                           <ul class="ul">
                              <li class="li"><dfn class="term">Divergent thread(s) in block</dfn>: Divergence between threads within a
                                 block was detected for a barrier that does not support this on the current
                                 architecture. For example, this occurs when <samp class="ph codeph">__syncthreads()</samp> is
                                 used within conditional code but the conditional does not evaluate equally
                                 across all threads in the block. 
                              </li>
                              <li class="li"><dfn class="term">Divergent thread(s) in warp</dfn>: Divergence between threads within a
                                 single warp was detected for a barrier that does not support this on the current
                                 architecture. 
                              </li>
                              <li class="li"><dfn class="term">Invalid arguments</dfn>: A barrier instruction or primitive was used with
                                 invalid arguments. This can occur for example if not all threads reaching a
                                 <samp class="ph codeph">__syncwarp()</samp> declare themselves in the mask parameter.
                                 However, synccheck will not detect cases where not all the threads declared in
                                 the mask parameter reach the <samp class="ph codeph">__syncwarp()</samp>.
                                 
                              </li>
                           </ul>
                        </div>
                        <p class="p">
                           The next line states the offset within the function of the location where the access happened. In
                           this case, the offset is 0xf0. If the application was compiled with line number
                           information, this line would also contain the file name and line number of the access,
                           followed by the name of the kernel issuing the access.
                           
                        </p>
                        <p class="p">
                           The third line contains information on the thread and block for which
                           this violation was detected. In this case, it is thread 37 in block 0.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="synccheck-cuda-barrier"><a name="synccheck-cuda-barrier" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#synccheck-cuda-barrier" name="synccheck-cuda-barrier" shape="rect">6.4.&nbsp;Synccheck support for <samp class="ph codeph">cuda::barrier</samp></a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Synccheck supports synchronization through <samp class="ph codeph">cuda::barrier</samp> on Ampere GPUs and newer.
                           
                        </p>
                        <div class="p">
                           The number of barriers tracked by the tool is based on the number of barriers present in the source
                           code as reported by compiler information. In some cases, the compiler may undercount this number.
                           Synccheck will report the following warning if more barriers are used than expected:
                           <pre class="pre screen" xml:space="preserve">========= Warning: Detected overflow of tracked cuda::barrier structures. Results might be incorrect. Try using --num-cuda-barriers to fix the issue</pre></div>
                        <p class="p">
                           The <samp class="ph codeph">--num-cuda-barriers</samp> option can be used to indicate the number of expected barriers
                           in the source code and workaround this issue.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="synccheck-wgmma"><a name="synccheck-wgmma" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#synccheck-wgmma" name="synccheck-wgmma" shape="rect">6.5.&nbsp;Synccheck support for <samp class="ph codeph">wgmma</samp></a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Synccheck supports additional checks related to PTX <samp class="ph codeph">wgmma</samp> instructions for Hopper sm_90a architecture.
                           
                        </p>
                        <p class="p"><a class="xref" href="https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-warpgroup-level-matrix-instructions" target="_blank" shape="rect"><samp class="ph codeph">wgmma</samp></a>
                           instructions are executed across a
                           <a class="xref" href="https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-warpgroup-level-matrix-instructions-warpgroup" target="_blank" shape="rect">warpgroup</a>.
                           Each warp in the warpgroup are expected to execute the same <samp class="ph codeph">wgmma</samp> instructions in the same order with the same predicates, with all threads active or none.
                           Synccheck can detect and report cases where these rules are not respected, and will exit the entire warpgroup when detected.
                           In such cases, the report will start with "Warpgroup MMA sequence error detected" instead of "Barrier error detected",
                           followed by a description of the specific error encountered. The error is reported once per warp encountering the error.
                           
                        </p>
                        <p class="p">
                           The <samp class="ph codeph">--check-warpgroup-mma</samp> option can be used to enable or disable these checks.
                           
                        </p>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="compute-sanitizer-features"><a name="compute-sanitizer-features" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#compute-sanitizer-features" name="compute-sanitizer-features" shape="rect">7.&nbsp;Compute Sanitizer Features</a></h2>
                  <div class="topic concept nested1" id="nonblocking-mode"><a name="nonblocking-mode" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#nonblocking-mode" name="nonblocking-mode" shape="rect">7.1.&nbsp;Nonblocking Mode</a></h3>
                     <div class="body conbody">
                        <p class="p"> By default, the standalone Compute Sanitizer tool will launch
                           kernels in nonblocking mode. This allows the tool to support error reporting in
                           applications running concurrent kernels
                           
                        </p>
                        <p class="p">
                           To force kernels to execute serially, a user can use the
                           <samp class="ph codeph">--force-blocking-launches yes</samp> option.
                           One side effect is that when in blocking mode, only the
                           first thread to hit an error in a kernel will be reported.
                           Also, using this option or <samp class="ph codeph">--force-synchronization-limit</samp>
                           will disable CUDA reduced API serialization.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="stack-backtraces"><a name="stack-backtraces" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#stack-backtraces" name="stack-backtraces" shape="rect">7.2.&nbsp;Stack Backtraces</a></h3>
                     <div class="body conbody">
                        <p class="p">Compute Sanitizer can generate backtraces when given <samp class="ph codeph">--show-backtrace</samp>
                           option. Backtraces usually consist of two sections – a saved host backtrace that leads
                           up to the CUDA driver call site, and a device backtrace at the time of the error. Each
                           backtrace contains a list of frames showing the state of the stack at the time the
                           backtrace was created. 
                        </p>
                        <p class="p">To get function names in the host backtraces, the user application must be
                           built with support for symbol information in the host application. For more
                           information, see <a class="xref" href="index.html#compilation-options" shape="rect">Compilation Options</a></p>
                        <p class="p">
                           Backtraces are printed for most Compute Sanitizer tool outputs, and the information
                           generated varies depending on the type of output. The table below explains the
                           kind of host and device backtrace seen under different conditions.
                           
                        </p>
                        <div class="tablenoborder"><a name="stack-backtraces__compute-sanitizer-stack-backtrace-information" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="stack-backtraces__compute-sanitizer-stack-backtrace-information" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 8. Compute Sanitizer Stack Backtrace Information</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="25%" id="d54e3249" rowspan="1" colspan="1">Output Type</th>
                                    <th class="entry" valign="top" width="37.5%" id="d54e3252" rowspan="1" colspan="1">Host Backtrace</th>
                                    <th class="entry" valign="top" width="37.5%" id="d54e3255" rowspan="1" colspan="1">Device Backtrace</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Memory access error</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Kernel launch on host</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">Precise backtrace on device</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Hardware exception</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Kernel launch on host</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">Imprecise backtrace on device
                                       <a name="fnsrc_1" href="#fntarg_1" shape="rect"><sup>1</sup></a></td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Malloc/Free error</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Kernel launch on host</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">Precise backtrace on device</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">cudaMalloc allocation leak</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Callsite of cudaMalloc</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">N/A</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">CUDA API error</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Callsite of CUDA API call</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">N/A</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Compute Sanitizer internal error</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Callsite leading to internal error</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">N/A</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Device heap allocation leak</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">N/A</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">N/A</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="25%" headers="d54e3249" rowspan="1" colspan="1">Shared memory hazard</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3252" rowspan="1" colspan="1">Kernel launch on host</td>
                                    <td class="entry" valign="top" width="37.5%" headers="d54e3255" rowspan="1" colspan="1">N/A</td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <p class="p">
                           Note that for OptiX applications, the name of OptiX internal device functions will
                           be displayed as "NVIDIA Internal".
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="name-demangling"><a name="name-demangling" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#name-demangling" name="name-demangling" shape="rect">7.3.&nbsp;Name Demangling</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The Compute Sanitizer suite supports displaying mangled and demangled names for
                           CUDA kernels and CUDA device functions.
                           By default, tools display the fully demangled name, which contains the name
                           of the kernel as well as its prototype information. In the simple demangle
                           mode, the tools will only display the first part of the name. If demangling
                           is disabled, tools will display the complete mangled name of the kernel.
                           
                        </p>
                        <p class="p"></p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="dynamic-parallelism"><a name="dynamic-parallelism" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#dynamic-parallelism" name="dynamic-parallelism" shape="rect">7.4.&nbsp;Dynamic Parallelism</a></h3>
                     <div class="body conbody">
                        <p class="p">The Compute Sanitizer tool suite supports dynamic parallelism. The <dfn class="term">memcheck</dfn>
                           tool supports precise error reporting of out of bounds and misaligned accesses on
                           global, local and shared memory accesses, as well as on global atomic instructions for
                           applications using dynamic parallelism. In addition, the imprecise hardware exception
                           reporting mechanism is also fully supported. Error detection on applications using
                           dynamic parallelism requires significantly more memory on the device; as a result, in
                           memory constrained environments, <dfn class="term">memcheck</dfn> may fail to initialize with an
                           internal out of memory error. 
                        </p>
                        <p class="p">
                           For limitations, see the known limitations in the Release Notes section.
                           
                        </p>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="error-actions"><a name="error-actions" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#error-actions" name="error-actions" shape="rect">7.5.&nbsp;Error Actions</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           When encountering an error, Compute Sanitizer behavior depends on the type of error.
                           The default behavior of Compute Sanitizer is to continue execution on purely host
                           side errors. Hardware exceptions detected by the memcheck tool cause the
                           CUDA context to be destroyed. Precise errors (such as memory access and
                           malloc/free errors) detected by the memcheck tool cause the kernel to be terminated.
                           This terminates the kernel without running any subsequent instructions and the
                           application continues launching other kernels in the CUDA context.
                           The handling of memory access and malloc/free errors detected by the memcheck tool
                           can be changed using the <samp class="ph codeph">--destroy-on-device-error</samp> option.
                           
                        </p>
                        <p class="p">
                           The <samp class="ph codeph">--destroy-on-device-error kernel</samp> option is not supported
                           on Maxwell GPUs.
                           
                        </p>
                        <p class="p">
                           For racecheck detected hazards, the hazard is reported, but execution is
                           not affected.
                           
                        </p>
                        <div class="p">
                           For a full summary of error action, based on the type of the error see the
                           table below. The error action <dfn class="term">terminate kernel</dfn> refers to the
                           cases where the kernel is terminated early, and no subsequent instructions
                           are run. In such cases, the CUDA context is not destroyed and other kernels
                           continue execution and CUDA API calls can still be made.
                           
                           <div class="note note"><span class="notetitle">Note:</span> 
                              When kernel execution is terminated early, the application may not have
                              completed its computations on data. Any subsequent kernels that depend
                              on this data will have undefined behavior.
                              
                           </div>
                           
                           The action <dfn class="term">terminate CUDA context</dfn> refers to the cases where the
                           CUDA context is forcibly terminated. In such cases, all outstanding work for
                           the context is terminated and subsequent CUDA API calls will fail.
                           The action <dfn class="term">continue application</dfn> refers to cases where the
                           application execution is not impacted, and the kernel continues executing
                           instructions.
                           
                        </div>
                        <p class="p"></p>
                        <div class="tablenoborder"><a name="error-actions__compute-sanitizer-error-action" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="error-actions__compute-sanitizer-error-action" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 9. Compute Sanitizer Error Actions</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="30.76923076923077%" id="d54e3466" rowspan="1" colspan="1">Error Type</th>
                                    <th class="entry" valign="top" width="15.384615384615385%" id="d54e3469" rowspan="1" colspan="1">Location</th>
                                    <th class="entry" valign="top" width="23.076923076923077%" id="d54e3472" rowspan="1" colspan="1">Action</th>
                                    <th class="entry" valign="top" width="30.76923076923077%" id="d54e3475" rowspan="1" colspan="1">Comments</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Memory access error</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Terminate CUDA context</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">User can choose to instead terminate the kernel</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Hardware exception</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Terminate CUDA context</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">Subsequent calls on the CUDA context will fail</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Malloc/Free error</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Terminate CUDA context</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">User can choose to instead terminate the kernel</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">cudaMalloc allocation leak</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Host</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Continue application</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">Error reported. No other action taken.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">CUDA API error</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Host</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Continue application</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">Error reported. No other action taken.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Device heap allocation leak</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Continue application</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">Error reported. No other action taken.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Shared memory hazard</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Continue application</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">Error reported. No other action taken.</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Synchronization error</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Device</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Terminate CUDA context</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">User can choose to instead terminate the kernel</td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3466" rowspan="1" colspan="1">Compute Sanitizer internal error</td>
                                    <td class="entry" valign="top" width="15.384615384615385%" headers="d54e3469" rowspan="1" colspan="1">Host</td>
                                    <td class="entry" valign="top" width="23.076923076923077%" headers="d54e3472" rowspan="1" colspan="1">Undefined</td>
                                    <td class="entry" valign="top" width="30.76923076923077%" headers="d54e3475" rowspan="1" colspan="1">The application may behave in an undefined fashion</td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="escape-sequences"><a name="escape-sequences" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#escape-sequences" name="escape-sequences" shape="rect">7.6.&nbsp;Escape Sequences</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           The <samp class="ph codeph">--save</samp> and <samp class="ph codeph">--log-file</samp> options to Compute Sanitizer
                           accept the following escape sequences in the file name.
                           
                           <ul class="ul">
                              <li class="li"><samp class="ph codeph">%%</samp> : Replaced with a literal %. 
                              </li>
                              <li class="li"><samp class="ph codeph">%p</samp> : Replaced with the PID of the Compute Sanitizer frontend
                                 application. 
                              </li>
                              <li class="li"><samp class="ph codeph">%q{ENVVAR}</samp> : Replaced with the contents of the environment
                                 variable <samp class="ph codeph">ENVVAR</samp>. If the variable does not exist, this is
                                 replaced with an empty string. 
                              </li>
                              <li class="li"> Any other character following the % causes an error.
                                 
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="specifying-filters"><a name="specifying-filters" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#specifying-filters" name="specifying-filters" shape="rect">7.7.&nbsp;Specifying Filters</a></h3>
                     <div class="body conbody">
                        <p class="p"> Compute Sanitizer tools support filtering the choice of kernels which should be checked.
                           When a filter is specified, only kernels matching the filter will be checked. Filters
                           are specified using the <samp class="ph codeph">--kernel-name</samp> and <samp class="ph codeph">--kernel-name-exclude</samp>
                           options. By default, the Compute Sanitizer tools will check all kernels in the
                           application. 
                        </p>
                        <p class="p"> The <samp class="ph codeph">--kernel-name</samp> and <samp class="ph codeph">--kernel-name-exclude</samp> options can be specified
                           multiple times. If a kernel satisfies any filter, it will be checked by the running the
                           Compute Sanitizer tool. 
                        </p>
                        <p class="p">
                           The <samp class="ph codeph">--kernel-name</samp> and  <samp class="ph codeph">--kernel-name-exclude</samp> options take a filter
                           specification consisting of a list of comma separated key value pairs, specified as <samp class="ph codeph">key=value</samp>.
                           When using the regex filter key, multiple key value pairs need to be specified through multiple use of the option
                           instead.
                           In order for a filter to be matched, all components of the filter specification must be
                           satisfied. If a filter is incorrectly specified in any component, the entire
                           filter is ignored. For a full summary of valid key values, see the table below.
                           If a key has multiple strings, any of the strings can be used to specify that
                           filter component.
                           
                        </p>
                        <div class="tablenoborder"><a name="specifying-filters__compute-sanitizer-filter-keys" shape="rect">
                              <!-- --></a><table cellpadding="4" cellspacing="0" summary="" id="specifying-filters__compute-sanitizer-filter-keys" class="table" frame="border" border="1" rules="all">
                              <caption><span class="tablecap">Table 10. Compute Sanitizer Filter Keys</span></caption>
                              <thead class="thead" align="left">
                                 <tr class="row">
                                    <th class="entry" valign="top" width="21.428571428571427%" id="d54e3722" rowspan="1" colspan="1">Name</th>
                                    <th class="entry" valign="top" width="21.428571428571427%" id="d54e3725" rowspan="1" colspan="1">Key String</th>
                                    <th class="entry" valign="top" width="28.57142857142857%" id="d54e3728" rowspan="1" colspan="1">Value</th>
                                    <th class="entry" valign="top" width="28.57142857142857%" id="d54e3731" rowspan="1" colspan="1">Comments</th>
                                 </tr>
                              </thead>
                              <tbody class="tbody">
                                 <tr class="row">
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3722" rowspan="1" colspan="1">Kernel Name</td>
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3725" rowspan="1" colspan="1">kernel_name, kne</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3728" rowspan="1" colspan="1">Complete mangled kernel name</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3731" rowspan="1" colspan="1">User specifies the complete mangled kernel name.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3722" rowspan="1" colspan="1">Kernel Substring</td>
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3725" rowspan="1" colspan="1">kernel_substring, kns</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3728" rowspan="1" colspan="1">Any substring in mangled kernel name</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3731" rowspan="1" colspan="1">User specifies a substring in the mangled kernel name.
                                       
                                    </td>
                                 </tr>
                                 <tr class="row">
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3722" rowspan="1" colspan="1">Regex</td>
                                    <td class="entry" valign="top" width="21.428571428571427%" headers="d54e3725" rowspan="1" colspan="1">regex</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3728" rowspan="1" colspan="1">Any regex that can be matched in a substring of the mangled kernel name</td>
                                    <td class="entry" valign="top" width="28.57142857142857%" headers="d54e3731" rowspan="1" colspan="1">User specifies a regular expression searched in the mangled kernel name.
                                       
                                    </td>
                                 </tr>
                              </tbody>
                           </table>
                        </div>
                        <p class="p"> When using the <samp class="ph codeph">kernel-name</samp> filters, the Compute Sanitizer tools will
                           check all <samp class="ph codeph">device</samp> function calls made by the kernel. When using CUDA
                           Dynamic Parallelism (CDP), the Compute Sanitizer tools will not check child kernels
                           launched from a checked kernel unless the child kernel matches a filter. If a GPU
                           launched kernel that does not match a filter calls a device function that is reachable
                           from a kernel that does match a filter, the device function behaves as though it was
                           checked. In the case of some tools, this can result in undefined behavior. 
                        </p>
                        <div class="example">
                           <h3 class="title sectiontitle">Filter usage example</h3>
                           <p class="p"> We consider an application that launches three different kernels declared below. </p><pre xml:space="preserve">
            <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-attribute">__global__</span> <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">void</span> gamma(<span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">int</span> *bufer);
            <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-attribute">__global__</span> <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">void</span> delta(<span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">int</span> *bufer);
            <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-attribute">__global__</span> <span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">void</span> epsilon(<span xmlns:xslthl="http://xslthl.sf.net" class="xslthl-keyword">int</span> *bufer);
</pre><p class="p"> Their respective mangled names are <samp class="ph codeph">_Z5gammaPi</samp>, <samp class="ph codeph">_Z5deltaPi</samp>
                              and <samp class="ph codeph">_Z7epsilonPi</samp>. We only want to check the launches of the kernel epsilon. Here are different means to achieve it:
                              
                           </p>
                           <ul class="ul">
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">compute-sanitizer --kernel-name kne=_Z7epsilonPi</samp> Only epsilon is matching the specified filter,
                                    so only kernel launches of epsilon will be checked.
                                 </p>
                              </li>
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">compute-sanitizer --kernel-name kns=epsilon</samp> Since "epsilon" is a substring of "_Z7epsilonPi",
                                    and also happens to be the only kernel having this substring in its mangled name, only epsilon will be matched and checked.
                                 </p>
                              </li>
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">compute-sanitizer --kernel-name-exclude kns=delta,kne=_Z5gammaPi</samp> This time, we are using the exclude options.
                                    Only epsilon is not matched by the exclude option in this scenario, which means it will be the only one checked.
                                    We specified multiple filter separating them with a comma: this can be used with both <samp class="ph codeph">kernel-name</samp> and <samp class="ph codeph">kernel-name-exclude</samp>.
                                 </p>
                              </li>
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">compute-sanitizer --kernel-name-exclude kns=delta --kernel-name-exclude kne=_Z5gammaPi</samp> Same as above,
                                    except we used the exclude option twice to specify multiple filters instead of specifying them all at once. If needed,
                                    <samp class="ph codeph">kernel-name</samp> and <samp class="ph codeph">kernel-name-exclude</samp> can be used at the same time.
                                 </p>
                              </li>
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">compute-sanitizer --kernel-name regex='[a-z]{7}'</samp> For this example we are using the regex filter.
                                    It matches any kernel for which the regular expression can be matched anywhere inside the mangle named. The specified
                                    regex matches any 7 consecutive lowercase letter. Only <samp class="ph codeph">_Z7epsilonPi</samp> has 7 consecutive lowercase letter,
                                    and therefore is the only kernel matched by <samp class="ph codeph">--kernel-name</samp>.
                                 </p>
                              </li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="coredump"><a name="coredump" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#coredump" name="coredump" shape="rect">7.8.&nbsp;Coredump support</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Starting from CUDA 11.6, the compute-sanitizer tool can generate a CUDA coredump
                           once an error is detected by using the <samp class="ph codeph">--generate-coredump yes</samp> option.
                           Once the coredump is generated, the target application will abort.
                           
                        </p>
                        <div class="p">
                           The coredump file can be loaded in cuda-gdb using the following option:
                           <pre class="pre screen" xml:space="preserve">(cuda-gdb) <strong class="ph b">target cudacore core.name.nvcudmp</strong></pre>
                           
                           See the
                           <a class="xref" href="https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support" target="_blank" shape="rect">cuda-gdb documentation</a>
                           for more information.
                           
                        </div>
                        <p class="p">
                           The <samp class="ph codeph">--coredump-name</samp> option can be used to specify the file name of the coredump. See the "Naming of GPU core dump files" section
                           of the <a class="xref" href="https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support" target="_blank" shape="rect">cuda-gdb documentation</a>
                           for more information on template specifiers and default name.
                           
                        </p>
                        <div class="p">
                           The coredump feature has the following restrictions:
                           
                           <ul class="ul">
                              <li class="li">Only threads that encountered an error can be inspected in the generated coredump</li>
                              <li class="li">Maxwell GPUs are not supported</li>
                              <li class="li">The racecheck tool is not supported.</li>
                              <li class="li">Coredumps are not supported on WSL2.</li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="error-suppression"><a name="error-suppression" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#error-suppression" name="error-suppression" shape="rect">7.9.&nbsp;Error suppression</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           The compute-sanitizer tools can sometimes generate false positive reports. In these cases, a suppression file
                           can be provided as input to the tool to suppress the reporting of these false positives.
                           
                        </p>
                        <p class="p">
                           A suppression file can be generated by using the <samp class="ph codeph">--xml</samp> option of the compute-sanitizer tool on
                           the target application. Once generated, the XML file can be edited manually to be more generic.
                           
                        </p>
                        <p class="p">
                           On subsequent use of the tools, the suppression file can be provided as input using the <samp class="ph codeph">--suppressions</samp> option.
                           
                        </p>
                        <div class="p">
                           The following rules are applied when checking if a detected report should be suppressed:
                           
                           <ul class="ul">
                              <li class="li">The types of the report must match.</li>
                              <li class="li">If provided in the suppression file, integer fields must match exactly.</li>
                              <li class="li">If provided in the suppression file, a string field can be a regex.</li>
                              <li class="li">When comparing stack traces, the suppression trace needs to have the same number of frames or less than the report one.</li>
                              <li class="li">Stack frame comparisons include the following fields (if provided in the suppression): function name, file name and module
                                 name.
                              </li>
                           </ul>
                        </div>
                        <div class="p">
                           The following types of error can be suppressed:
                           
                           <ul class="ul">
                              <li class="li"><a class="xref" href="index.html#api-error-checking" shape="rect">CUDA API errors</a></li>
                              <li class="li"><a class="xref" href="index.html#initcheck-tool" shape="rect">Initcheck uninitialized memory accesses</a></li>
                              <li class="li"><a class="xref" href="index.html#understanding-racecheck-analysis-reports" shape="rect">Racecheck analysis reports</a></li>
                           </ul>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="optix"><a name="optix" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#optix" name="optix" shape="rect">7.10.&nbsp;OptiX support</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Starting from CUDA 11.6, the compute-sanitizer tool support OptiX 7 applications with memcheck and initcheck.
                           The option <samp class="ph codeph">--check-optix yes</samp> needs to be set for optix launches to be tracked with initcheck.
                           To get full device backtrace information, please make sure your OptiX modules are compiled with
                           <samp class="ph codeph">OPTIX_COMPILE_DEBUG_LEVEL_FULL</samp> set in the <samp class="ph codeph">debugLevel</samp> field in the
                           <samp class="ph codeph">OptixModuleCompileOptions</samp> structure.
                           
                        </p>
                        <div class="p">
                           When using compute-sanitizer on OptiX applciations, it is possible that some or all device
                           frames are located in OptiX internal code. Such frames have their name displayed as <samp class="ph codeph">NVIDIA Internal</samp>.
                           See the example below of an error reported in user code called from an internal OptiX function:
                           <pre class="pre screen" xml:space="preserve">
========= Invalid __global__ write of size 1 bytes
=========     at __raygen__placeholder_0x67b9a77bb7822a34+0x19b0 in /home/cuda/optixApp.cu:70
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x7f91edf00403 is out of bounds
=========     and is 262,132 bytes after the nearest allocation at 0x7f91edec0400 of size 16 bytes
=========     <strong class="ph b">Device Frame:NVIDIA Internal [0x520]</strong>
=========     Saved host backtrace up to driver entry point at kernel launch time
[...]
            </pre></div>
                        <div class="p">
                           Starting from CUDA 11.7, it is possible to detect leaks of <samp class="ph codeph">OptixModule</samp>, <samp class="ph codeph">optixPipeline</samp>,
                           <samp class="ph codeph">optixProgramGroup</samp> and <samp class="ph codeph">optixDenoiser</samp> with compute-sanitizer. This requires using
                           the <samp class="ph codeph">--check-optix-leaks yes</samp> option. Leaks will only reported if the <samp class="ph codeph">OptixDeviceContext</samp>
                           is destroyed with a call to <samp class="ph codeph">OptixDeviceContextDestroy</samp>. <samp class="ph codeph">OptixDeviceContext</samp> that are leaking
                           will have their associated CUDA buffers reported with a regular use of <samp class="ph codeph">--leak-check full</samp>.
                           See the example below of an <samp class="ph codeph">optixProgramGroup</samp> that was not destroyed being reported:
                           <pre class="pre screen" xml:space="preserve">
========= Leaked an OptixProgramGroup with handle 0x55dbffbd9840
=========     Saved host backtrace up to driver entry point at allocation time
[...]
            </pre></div>
                        <p class="p">The following feature set is supported per OptiX API version:</p>
                        <table cellpadding="4" cellspacing="0" summary="" border="1" class="simpletable">
                           <tr class="strow">
                              <td valign="top" class="stentry" rowspan="1" colspan="1"><strong class="ph b">OptiX API Version</strong></td>
                              <td valign="top" class="stentry" rowspan="1" colspan="1"><strong class="ph b">Kernel checks</strong></td>
                              <td valign="top" class="stentry" rowspan="1" colspan="1"><strong class="ph b">Resource leak check</strong></td>
                           </tr>
                           <tr class="strow">
                              <td valign="top" class="stentry" rowspan="1" colspan="1">7.0 - 8.0</td>
                              <td valign="top" class="stentry" rowspan="1" colspan="1">Yes</td>
                              <td valign="top" class="stentry" rowspan="1" colspan="1">Yes</td>
                           </tr>
                        </table>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="usage-guide"><a name="usage-guide" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#usage-guide" name="usage-guide" shape="rect">8.&nbsp;Usage Guide</a></h2>
                  <div class="body conbody"></div>
                  <div class="topic concept nested1" id="memory-footprint"><a name="memory-footprint" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#memory-footprint" name="memory-footprint" shape="rect">8.1.&nbsp;Memory Footprint</a></h3>
                     <div class="body conbody">
                        <p class="p">
                           Compute Sanitizer tools can have a large memory footprint due to their tracking data.
                           This can cause out of memory errors on applications performing a large number of concurrent kernel launches.
                           
                        </p><pre class="pre screen" xml:space="preserve">
========= Internal Sanitizer Error: The Sanitizer encountered an error while launching kernel_name and didn't track the launch. Errors might go undetected. (Unable to allocate enough memory to perform the requested operation)
</pre><p class="p">
                           The tools might also cause a failure to allocate host memory causing the application to crash.
                           
                        </p><pre class="pre screen" xml:space="preserve">
========= Error: process didn't terminate successfully
========= Target application returned an error
</pre><div class="p">
                           This issue can be resolved using one of the following command line options:
                           
                           <ul class="ul">
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">--force-synchronization-limit {number}</samp> forces a stream synchronization after a stream
                                    reaches the given number of launches without synchronizing.
                                    
                                 </p>
                              </li>
                              <li class="li">
                                 <p class="p"><samp class="ph codeph">--force-blocking-launches yes</samp> forces the serialization of of every kernel launch.
                                    This option is equivalent to <samp class="ph codeph">--force-synchronization-limit 1</samp>.
                                    
                                 </p>
                              </li>
                           </ul>
                        </div>
                        <p class="p">
                           Using  <a class="xref" href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading" target="_blank" shape="rect">CUDA lazy module loading</a>
                           will also help lower the memory footprint of the tools, both for host and device memory.
                           
                        </p>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="os-specific-behavior"><a name="os-specific-behavior" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#os-specific-behavior" name="os-specific-behavior" shape="rect">9.&nbsp;Operating System Specific Behavior</a></h2>
                  <div class="body conbody">
                     <p class="p">This section describes operating system specific behavior.</p>
                  </div>
                  <div class="topic concept nested1" id="os-specific-windows"><a name="os-specific-windows" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#os-specific-windows" name="os-specific-windows" shape="rect">9.1.&nbsp;Windows Specific Behavior</a></h3>
                     <div class="body conbody">
                        <ul class="ul">
                           <li class="li">
                              <p class="p">Timeout Detection and Recovery (TDR)</p>
                              <p class="p"> On Windows, GPUs have a timeout associated with them. GPU applications that take
                                 longer than the threshold (default of 2 seconds) will be killed by the operating
                                 system. Since the Compute Sanitizer tools increase the runtime of kernels, it is
                                 possible for a CUDA kernel to exceed the timeout and therefore be terminated due
                                 to the TDR mechanism. 
                              </p>
                              <p class="p"> For the purposes of debugging, the number of seconds before which the timeout is
                                 hit can be modified by setting the timeout value in seconds in the DWORD
                                 registry key <samp class="ph codeph">TdrDelay</samp> at:
                              </p><pre xml:space="preserve">HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers</pre><p class="p">
                                 More information about the registry keys to control the Timeout Detection and Recovery
                                 mechanism is available from MSDN at
                                 <a class="xref" href="http://msdn.microsoft.com/en-us/library/windows/hardware/ff569918%28v=vs.85%29.aspx" target="_blank" shape="rect">http://msdn.microsoft.com/en-us/library/windows/hardware/ff569918%28v=vs.85%29.aspx</a>.
                                 
                              </p>
                           </li>
                        </ul>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="tegra-setup"><a name="tegra-setup" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#tegra-setup" name="tegra-setup" shape="rect">9.2.&nbsp;Using the Compute Sanitizer on Jetson and Tegra devices</a></h3>
                     <div class="body conbody">
                        <p class="p">By default, on Jetson and Drive Tegra devices, GPU debugging is supported only if <samp class="ph codeph">compute-sanitizer</samp> is launched by a user who is a member of the <strong class="ph b">debug</strong> group.
                        </p>
                        <p class="p">To add the current user to the <strong class="ph b">debug</strong> group run this command:
                        </p><pre class="pre screen" xml:space="preserve"><strong class="ph b">sudo usermod -a -G debug $USER</strong></pre></div>
                  </div>
               </div>
               <div class="topic concept nested0" id="cuda-fortran-support"><a name="cuda-fortran-support" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#cuda-fortran-support" name="cuda-fortran-support" shape="rect">10.&nbsp;CUDA Fortran Support</a></h2>
                  <div class="body conbody">
                     <p class="p">This section describes support for CUDA Fortran.</p>
                  </div>
                  <div class="topic concept nested1" id="cuda-fortran-specific-behavior"><a name="cuda-fortran-specific-behavior" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#cuda-fortran-specific-behavior" name="cuda-fortran-specific-behavior" shape="rect">10.1.&nbsp;CUDA Fortran Specific Behavior</a></h3>
                     <div class="body conbody">
                        <ul class="ul">
                           <li class="li">By default, error reports printed by Compute Sanitizer contain 0-based C style values for
                              thread index (threadIdx) and block index (blockIdx).
                              For Compute Sanitizer tools to use Fortran style 1-based offsets,
                              use the <samp class="ph codeph">--language fortran</samp> option.
                              
                           </li>
                           <li class="li">The CUDA Fortran compiler may insert extra padding in shared memory. Accesses hitting this extra padding may not be reported
                              as an error.
                              
                           </li>
                        </ul>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="compute-sanitizer-tool-examples"><a name="compute-sanitizer-tool-examples" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#compute-sanitizer-tool-examples" name="compute-sanitizer-tool-examples" shape="rect">11.&nbsp;Compute Sanitizer Tool Examples</a></h2>
                  <div class="topic concept nested1" id="example-use-of-memcheck"><a name="example-use-of-memcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#example-use-of-memcheck" name="example-use-of-memcheck" shape="rect">11.1.&nbsp;Example Use of Memcheck</a></h3>
                     <div class="body conbody">
                        <div class="p">
                           This section presents a walk-through of running the memcheck tool from
                           Compute Sanitizer on a simple application called <samp class="ph codeph">memcheck_demo</samp>.
                           
                           <div class="note note"><span class="notetitle">Note:</span> Depending on the SM type of your GPU, your system output may vary.
                           </div>
                        </div>
                        <p class="p">The application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Memcheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                        <div class="p">
                           This application can be compiled using the provided Makefile:
                           <pre class="pre screen" xml:space="preserve">
make
</pre></div>
                     </div>
                     <div class="topic concept nested2" id="memcheck-demo-output"><a name="memcheck-demo-output" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#memcheck-demo-output" name="memcheck-demo-output" shape="rect">11.1.1.&nbsp;<samp class="ph codeph">memcheck_demo</samp> Output
                              </a></h3>
                        <div class="body conbody">
                           <p class="p">
                              When a CUDA application causes access violations, the kernel launch may report an illegal memory
                              access or misaligned address. Sticky errors will be reported for all subsequent kernel launches.
                              
                           </p>
                           <p class="p">
                              This sample application is causing two failures but there is no way to detect where the misaligned
                              address access is caused. The second kernel is also not able to run, as illustrated in the following
                              output:
                              
                           </p><pre class="pre screen" xml:space="preserve"><strong class="ph b">$ ./memcheck_demo</strong>
Mallocing memory
Running unaligned_kernel: misaligned address
Running out_of_bounds_kernel: misaligned address
</pre></div>
                     </div>
                     <div class="topic concept nested2" id="memcheck-demo-output-with-memcheck-release-build"><a name="memcheck-demo-output-with-memcheck-release-build" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#memcheck-demo-output-with-memcheck-release-build" name="memcheck-demo-output-with-memcheck-release-build" shape="rect">11.1.2.&nbsp;<samp class="ph codeph">memcheck_demo</samp> Output with Memcheck (Release Build)</a></h3>
                        <div class="body conbody">
                           <p class="p">
                              In this case, since the application is built in release mode, the
                              Compute Sanitizer output contains only the kernel names from the application causing
                              the access violation. Though the kernel name and error type are detected, there
                              is no line number information on the failing kernel. Also included in the output
                              are the host and device backtraces for the call sites where the functions were launched
                              
                           </p>
                           <p class="p">
                              Now run this application with Compute Sanitizer and check the output. By default, the
                              application will run so that the kernel is terminated on memory access errors, but other
                              work in the CUDA context can still proceed.
                              
                           </p>
                           <p class="p">
                              In the output below, the first kernel no longer reports an unspecified launch failure as
                              its execution has been terminated early after Compute Sanitizer detected the error. The
                              application continued to run the second kernel. The error detected in the second kernel
                              causes it to terminate early.
                              
                           </p><pre class="pre screen" xml:space="preserve">
<strong class="ph b">$ make run_memcheck</strong>
/usr/local/cuda/compute-sanitizer/compute-sanitizer --destroy-on-device-error kernel memcheck_demo
========= COMPUTE-SANITIZER
Mallocing memory
========= Invalid __global__ write of size 4 bytes
=========     at unaligned_kernel()+0x70
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x7f671ac00001 is misaligned
=========     and is inside the nearest allocation at 0x7fb654c00000 of size 4 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2774ec]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:__cudart803 [0xfccb]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaLaunchKernel [0x6a578]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaError cudaLaunchKernel&lt;char&gt;(char const*, dim3, dim3, void**, unsigned long, CUstream_st*) [0xb535]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:__device_stub__Z16unaligned_kernelv() [0xb22e]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:unaligned_kernel() [0xb28c]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:run_unaligned() [0xaf55]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:main [0xb0e2]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:../sysdeps/nptl/libc_start_call_main.h:58:__libc_start_call_main [0x2dfd0]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:../csu/libc-start.c:379:__libc_start_main [0x2e07d]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0xada5]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========
Running unaligned_kernel: no error
========= Invalid __global__ write of size 4 bytes
=========     at out_of_bounds_kernel()+0x90
=========     by thread (0,0,0) in block (0,0,0)
=========     and is 140,418,624,437,472 bytes before the nearest allocation at 0x7fb649a00000 of size 1,024 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2774ec]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:__cudart803 [0xfccb]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaLaunchKernel [0x6a578]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaError cudaLaunchKernel&lt;char&gt;(char const*, dim3, dim3, void**, unsigned long, CUstream_st*) [0xb535]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:__device_stub__Z20out_of_bounds_kernelv() [0xb34e]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:out_of_bounds_kernel() [0xb3ac]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:run_out_of_bounds() [0xb037]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:main [0xb0e7]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:../sysdeps/nptl/libc_start_call_main.h:58:__libc_start_call_main [0x2dfd0]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:../csu/libc-start.c:379:__libc_start_main [0x2e07d]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0xada5]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========
Running out_of_bounds_kernel: no error
========= ERROR SUMMARY: 2 errors
</pre></div>
                     </div>
                     <div class="topic concept nested2" id="memcheck-demo-output-with-memcheck-debug-build"><a name="memcheck-demo-output-with-memcheck-debug-build" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#memcheck-demo-output-with-memcheck-debug-build" name="memcheck-demo-output-with-memcheck-debug-build" shape="rect">11.1.3.&nbsp;<samp class="ph codeph">memcheck_demo</samp> Output with Memcheck (Debug Build)</a></h3>
                        <div class="body conbody">
                           <div class="p"> The application can be built with device side debug information and function symbols as:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <p class="p">
                              The source location of the error is now reported in the compute-sanitizer output:
                              
                           </p><pre class="pre screen" xml:space="preserve">
<strong class="ph b">$ make run_memcheck</strong>
========= COMPUTE-SANITIZER
========= Invalid __global__ write of size 4 bytes
=========     at unaligned_kernel()+0x160 in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo.cu:34
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x7f3d7ce00001 is misaligned
=========     and is inside the nearest allocation at 0x7f9544c00000 of size 4 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2774ec]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:__cudart803 [0xfccb]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaLaunchKernel [0x6a578]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaError cudaLaunchKernel&lt;char&gt;(char const*, dim3, dim3, void**, unsigned long, CUstream_st*) [0xb535]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:__device_stub__Z16unaligned_kernelv() [0xb22e]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:unaligned_kernel() [0xb28c]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:run_unaligned() [0xaf55]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:main [0xb0e2]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:../sysdeps/nptl/libc_start_call_main.h:58:__libc_start_call_main [0x2dfd0]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:../csu/libc-start.c:379:__libc_start_main [0x2e07d]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0xada5]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========
Running unaligned_kernel: no error
========= Invalid __global__ write of size 4 bytes
=========     at out_of_bounds_function()+0xb0 in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo.cu:39
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x87654320 is out of bounds
=========     and is 140,276,689,190,112 bytes before the nearest allocation at 0x7f953da00000 of size 1,024 bytes
=========     Device Frame:out_of_bounds_kernel()+0x30 in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo.cu:44
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2774ec]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:__cudart803 [0xfccb]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaLaunchKernel [0x6a578]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaError cudaLaunchKernel&lt;char&gt;(char const*, dim3, dim3, void**, unsigned long, CUstream_st*) [0xb535]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:__device_stub__Z20out_of_bounds_kernelv() [0xb34e]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:out_of_bounds_kernel() [0xb3ac]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:run_out_of_bounds() [0xb037]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:main [0xb0e7]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:../sysdeps/nptl/libc_start_call_main.h:58:__libc_start_call_main [0x2dfd0]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:../csu/libc-start.c:379:__libc_start_main [0x2e07d]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0xada5]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========
Running out_of_bounds_kernel: no error
========= ERROR SUMMARY: 2 errors
</pre></div>
                     </div>
                     <div class="topic concept nested2" id="leak-checking-in-compute-sanitizer"><a name="leak-checking-in-compute-sanitizer" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#leak-checking-in-compute-sanitizer" name="leak-checking-in-compute-sanitizer" shape="rect">11.1.4.&nbsp;Leak Checking in Compute Sanitizer</a></h3>
                        <div class="body conbody">
                           <p class="p">To print information about the allocations that have not been freed at the time
                              the CUDA context is destroyed, we can specify the <samp class="ph codeph">--leak-check full</samp>
                              option to Compute Sanitizer.
                           </p>
                           <p class="p">When running the program with the leak check option, the user is presented with
                              a list of allocations that were not destroyed, along with the size of the allocation
                              and the address on the device of the allocation. For allocations made on the host,
                              each leak report will also print a backtrace corresponding to the saved host stack
                              at the time the allocation was first made. Also presented is a summary of the total
                              number of bytes leaked and the corresponding number of allocations.
                           </p>
                           <p class="p">In this example, the program created an allocation using
                              <samp class="ph codeph">cudaMalloc()</samp> and has not called <samp class="ph codeph">cudaFree()</samp>
                              to release it, leaking memory. Notice that Compute Sanitizer still prints errors
                              it encountered while running the application. They are omitted in the output below
                              for the sake of clarity.
                              
                           </p><pre class="pre screen" xml:space="preserve">
<strong class="ph b">$ make run_leakcheck</strong>
========= COMPUTE-SANITIZER
...
========= Leaked 1,024 bytes at 0x7fab4fa00000
=========     Saved host backtrace up to driver entry point at cudaMalloc time
=========     Host Frame: [0x9b5c16]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame:__cudart612 [0x41f5e]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:__cudart618 [0x1080b]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:cudaMalloc [0x4f3ef]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:main [0xb0dd]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========     Host Frame:../sysdeps/nptl/libc_start_call_main.h:58:__libc_start_call_main [0x2dfd0]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:../csu/libc-start.c:379:__libc_start_main [0x2e07d]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0xada5]
=========                in /home/cuda/github/compute-sanitizer-samples/Memcheck/memcheck_demo
=========
========= LEAK SUMMARY: 1024 bytes leaked in 1 allocations
========= ERROR SUMMARY: 3 errors
</pre></div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="example-use-of-racecheck"><a name="example-use-of-racecheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#example-use-of-racecheck" name="example-use-of-racecheck" shape="rect">11.2.&nbsp;Example Use of Racecheck</a></h3>
                     <div class="body conbody">
                        <div class="section">
                           <div class="p">
                              This section presents two example usages of the racecheck tool from
                              Compute Sanitizer. The first example uses an application called
                              <samp class="ph codeph">block_error</samp>, which has shared memory hazards on the block level.
                              The second example uses an application called <samp class="ph codeph">warp_error</samp>,
                              which has shared memory hazards on the warp level.
                              
                              <div class="note note"><span class="notetitle">Note:</span> Depending on the SM type of your GPU, your system output may vary.
                              </div>
                           </div>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="racecheck-demo-block-error"><a name="racecheck-demo-block-error" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-demo-block-error" name="racecheck-demo-block-error" shape="rect">11.2.1.&nbsp;Block-level Hazards</a></h3>
                        <div class="body conbody">
                           <p class="p">The application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Racecheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <div class="section">
                              <p class="p"> Each kernel thread write some element in shared memory. Afterward, thread 0 computes
                                 the sum of all elements in shared memory and stores the result in global memory
                                 variable <samp class="ph codeph">sum_out</samp>. 
                              </p>
                              <div class="p">
                                 Running this application under the racecheck tool with the
                                 <samp class="ph codeph">--racecheck-report analysis</samp> option, the following
                                 error is reported:
                                 <pre class="pre screen" xml:space="preserve">
$ make run_block_error
========= COMPUTE-SANITIZER
========= Error: Race reported between Write access at sumKernel(int *, int *)+0x90 in /home/cuda/github/compute-sanitizer-samples/Racecheck/block_error.cu:41
=========     and Read access at sumKernel(int *, int *)+0x100 in /home/cuda/github/compute-sanitizer-samples/Racecheck/block_error.cu:51 [508 hazards]
=========
========= RACECHECK SUMMARY: 1 hazard displayed (1 error, 0 warnings)
</pre></div>
                              <p class="p">
                                 Racecheck reports races between thread 0 reading all shared memory elements
                                 in line 51 and each individual thread writing its shared memory entry in line 41.
                                 Accesses to shared memory between multiple threads, where at least one access
                                 is a write, can potentially race with each other.
                                 Since the races are between threads of different warps, the block-level synchronization
                                 barrier <samp class="ph codeph">__syncthreads()</samp> is required in line 42.
                                 
                              </p>
                              <p class="p">
                                 Note that a total of 508 hazards are reported: the kernel uses a single block
                                 of 128 threads. The data size written or read, respectively, by each thread is
                                 four bytes (one <samp class="ph codeph">int</samp>) and hazards are reported at the byte level.
                                 The writes by all threads race with the reads by thread 0, except for the four
                                 writes by thread 0 itself.
                                 
                              </p>
                           </div>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="racecheck-demo-warp-error"><a name="racecheck-demo-warp-error" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#racecheck-demo-warp-error" name="racecheck-demo-warp-error" shape="rect">11.2.2.&nbsp;Warp-level Hazards</a></h3>
                        <div class="body conbody">
                           <p class="p">The application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Racecheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <div class="section">
                              <p class="p">
                                 The kernel computes the some of all individual elements in shared memory two stages.
                                 First, each thread computes its local shared memory value in <samp class="ph codeph">smem_first</samp>.
                                 Second, a single thread of each warp is chosen with <samp class="ph codeph">if (tx % WARP_SIZE == 0)</samp>
                                 to sum all elements written by its warp, indexed <samp class="ph codeph">wx</samp>, and store the result
                                 in <samp class="ph codeph">smem_second</samp>.
                                 Finally, thread 0 of the kernel computes the sum of elements in <samp class="ph codeph">smem_second</samp>
                                 and writes the value into global memory.
                                 
                              </p>
                              <div class="p">
                                 Running this application under the racecheck tool with the
                                 <samp class="ph codeph">--racecheck-report hazard</samp> option, multiple
                                 hazards with WARNING severity are reported:
                                 <pre class="pre screen" xml:space="preserve">
========= Warning: (Warp Level Programming) Potential RAW hazard detected at __shared__ 0x8c in block (0,0,0) :
=========     Write Thread (35,0,0) at sumKernel(int *, int *)+0x90 in /home/cuda/github/compute-sanitizer-samples/Racecheck/warp_error.cu:44
=========     Read Thread (32,0,0) at sumKernel(int *, int *)+0x120 in /home/cuda/github/compute-sanitizer-samples/Racecheck/warp_error.cu:56
=========     Current Value : 35
</pre></div>
                              <p class="p"> To avoid the errors demonstrated in the <a class="xref" href="index.html#racecheck-demo-block-error" shape="rect">Block-level Hazards</a>
                                 example, the kernel uses the block-level barrier <samp class="ph codeph">__syncthreads()</samp> in
                                 line 60. However, racecheck still reports read-after-write (RAW) hazards between
                                 threads within the same warp, with severity WARNING. On architectures prior to SM
                                 7.0 (Volta), programmers commonly relied on the assumption that threads within a
                                 warp execute code in lock-step (warp-level programming). Starting with CUDA 9.0,
                                 programmers can use the new <samp class="ph codeph">__syncwarp()</samp> warp-wide barrier (instead
                                 of only <samp class="ph codeph">__syncthreads()</samp> beforehand) to avoid such hazards. This
                                 barrier should be inserted at line 45.
                              </p>
                           </div>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="example-use-of-initcheck"><a name="example-use-of-initcheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#example-use-of-initcheck" name="example-use-of-initcheck" shape="rect">11.3.&nbsp;Example Use of Initcheck</a></h3>
                     <div class="body conbody">
                        <div class="section">
                           <p class="p"> This section presents the usage of the initcheck tool from Compute Sanitizer. The
                              example uses an application called <samp class="ph codeph">memset_error</samp>. 
                           </p>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="initcheck-demo-memset-error"><a name="initcheck-demo-memset-error" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#initcheck-demo-memset-error" name="initcheck-demo-memset-error" shape="rect">11.3.1.&nbsp;Memset Error</a></h3>
                        <div class="body conbody">
                           <p class="p">The application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Initcheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <div class="section">
                              <p class="p">
                                 The example implements a very simple vector addition, where the thread index
                                 is added to each vector element. The vector contains <samp class="ph codeph">NumBlocks * NumThreads</samp>
                                 elements of type <samp class="ph codeph">int</samp>.
                                 The vector is allocated on the device and then initialized to 0 using <samp class="ph codeph">cudaMemset</samp>
                                 before the kernel is launched.
                                 
                              </p>
                              <div class="p">
                                 Running this application under the initcheck tool reports multiple errors
                                 like the following:
                                 <pre class="pre screen" xml:space="preserve">
$ make run_initcheck
========= Uninitialized __global__ memory read of size 4 bytes
=========     at vectorAdd(int *)+0x70 in /home/cuda/github/compute-sanitizer-samples/Initcheck/memset_error.cu:41
=========     by thread (31,0,0) in block (1,0,0)
=========     Address 0x7f3c7ec000fc
</pre></div>
                              <p class="p">
                                 The problem is that the call to <samp class="ph codeph">cudaMemset</samp> expects the size
                                 of the to-be set memory in bytes. However, the size is given in elements, as
                                 a factor of <samp class="ph codeph">sizeof(int)</samp> is missing while computing the parameter.
                                 As a result, 3/4 of the memory will have undefined values during the vector addition.
                                 
                              </p>
                           </div>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="example-use-of-synccheck"><a name="example-use-of-synccheck" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#example-use-of-synccheck" name="example-use-of-synccheck" shape="rect">11.4.&nbsp;Example Use of Synccheck</a></h3>
                     <div class="body conbody">
                        <div class="section">
                           <div class="p">
                              This section presents two example usages of the synccheck tool from
                              Compute Sanitizer. The first example uses an application called
                              <samp class="ph codeph">divergent_threads</samp>. The second example uses an application
                              called <samp class="ph codeph">illegal_syncwarp</samp>.
                              
                              <div class="note note"><span class="notetitle">Note:</span> Depending on the SM type of your GPU, your system output may vary.
                              </div>
                           </div>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="synccheck-demo-divergent-threads"><a name="synccheck-demo-divergent-threads" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#synccheck-demo-divergent-threads" name="synccheck-demo-divergent-threads" shape="rect">11.4.1.&nbsp;Divergent Threads</a></h3>
                        <div class="body conbody">
                           <p class="p">The divergent_threads application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Synccheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <div class="section">
                              <p class="p">
                                 In this example, we launch a kernel with a single block of 64 threads.
                                 The kernels loops over <samp class="ph codeph">DataBlocks</samp> blocks of input data <samp class="ph codeph">data_in</samp>.
                                 In each iteration, <samp class="ph codeph">NumThreads</samp> elements are added concurrently
                                 in shared memory. Finally, a single thread 0 computes the sum of all
                                 values in shared memory and writes it to <samp class="ph codeph">sum_out</samp>.
                                 
                              </p>
                              <div class="p">
                                 Running this application under the synccheck tool, 16 errors like the
                                 following are reported:
                                 <pre class="pre screen" xml:space="preserve">
$ make run_divergent_threads
========= Barrier error detected. Divergent thread(s) in warp
=========     at myKernel(int*, int*, int)+0x578 in divergent_thread.cu:54
=========     by thread (32,0,0) in block (0,0,0)
</pre></div>
                              <p class="p">
                                 The issue is with the <samp class="ph codeph">__syncthreads()</samp> in line 20
                                 when reading the last data block into shared memory.
                                 Note that the last data block only has 48 elements (compared to 64 elements
                                 for all other blocks). As a result, not all threads of the second warp
                                 execute this statement in convergence as required.
                                 
                              </p>
                              <div class="p">
                                 <div class="note note"><span class="notetitle">Note:</span> Calling <samp class="ph codeph">__syncthreads()</samp> without convergence is allowed on
                                    SM 7.0 and above. Synccheck will not report any error for this example on
                                    these architectures.
                                 </div>
                              </div>
                           </div>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="synccheck-demo-illegal-syncwarp"><a name="synccheck-demo-illegal-syncwarp" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#synccheck-demo-illegal-syncwarp" name="synccheck-demo-illegal-syncwarp" shape="rect">11.4.2.&nbsp;Illegal Syncwarp</a></h3>
                        <div class="body conbody">
                           <p class="p">The illegal_syncwarp application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Synccheck" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">
make dbg=1
</pre></div>
                           <div class="section">
                              <p class="p">
                                 This example only applies to devices of compute capability 7.0 (Volta) and above.
                                 
                                 The kernel is launched with a single warp (32 threads), but only thread 0-15 are part of the computation.
                                 Each of these threads initializes one shared memory element with its thread index.
                                 
                                 After the assignment, <samp class="ph codeph">__syncwarp()</samp> is used to ensure that the warp is converged
                                 and all writes are visible to other threads. The mask passed to <samp class="ph codeph">__syncwarp()</samp>
                                 is computed using <samp class="ph codeph">__ballot_sync()</samp>, which enables the bits for the first 16 threads in <samp class="ph codeph">mask</samp>.
                                 Finally, the first thread (index 0) computes the sum over all initialized shared memory elements
                                 and writes it to global memory.
                                 
                              </p>
                              <div class="p">
                                 Building the application with <samp class="ph codeph">-G</samp> to enable debug information and running
                                 it under the synccheck tool on SM 7.0 and above, multiple errors like the following are reported:
                                 <pre class="pre screen" xml:space="preserve">
$ make run_illegal_syncwarp
========= Barrier error detected. Invalid arguments
=========     at __cuda_sm70_warpsync+0x30
=========     by thread (0,0,0) in block (0,0,0)
=========     Device Frame:__syncwarp(unsigned int)+0xf0 in /usr/local/cuda/include/sm_30_intrinsics.hpp:110
=========     Device Frame:myKernel(int *)+0x3c0 in /home/cuda/github/compute-sanitizer-samples/Synccheck/illegal_syncwarp.cu:48
</pre></div>
                              <p class="p">
                                 The issue is with the <samp class="ph codeph">__syncwarp(mask)</samp> at line 48.
                                 All threads for which <samp class="ph codeph">tx &lt; (NumThreads / 2)</samp> holds true are enabled in the mask,
                                 which are threads 0-15. However, the if condition evaluates true for threads 0-16.
                                 As a result, thread 16 executes the <samp class="ph codeph">__syncwarp(mask)</samp> but does not declare
                                 itself in the mask parameter as required.
                                 
                              </p>
                           </div>
                        </div>
                     </div>
                  </div>
                  <div class="topic concept nested1" id="example-use-of-suppressions"><a name="example-use-of-suppressions" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#example-use-of-suppressions" name="example-use-of-suppressions" shape="rect">11.5.&nbsp;Example Use of suppressions</a></h3>
                     <div class="body conbody">
                        <div class="section">
                           <p class="p">
                              This section presents two example usages of the suppressions feature of
                              Compute Sanitizer. The first example displays an API suppression (in the
                              memcheck tool). The second example displays an initcheck report suppression.
                              
                           </p>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="suppressions-demo-api"><a name="suppressions-demo-api" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#suppressions-demo-api" name="suppressions-demo-api" shape="rect">11.5.1.&nbsp;API error suppression</a></h3>
                        <div class="body conbody">
                           <p class="p">
                              The API error application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Suppressions" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">make</pre></div>
                           <div class="section">
                              <div class="p">
                                 In this example, we have a simple loop where the application attempts
                                 to allocate a large decreasing size. We can expect the cudaMalloc API to
                                 fail several times before the size is small enough to fit on the GPU.
                                 <pre class="pre screen" xml:space="preserve">$ make run_memcheck
 /usr/local/cuda/compute-sanitizer/compute-sanitizer suppressions_demo
 ========= COMPUTE-SANITIZER
 ========= Program hit cudaErrorMemoryAllocation (error 2) due to "out of memory" on CUDA API call to cudaMalloc.</pre></div>
                              <div class="p">
                                 In order to generate a suppressions file, we need to use the <samp class="ph codeph">--xml</samp> option
                                 combined with the <samp class="ph codeph">--save</samp> option for the output file name. Running that command
                                 still prints out error as before, but it also creates an XML file and populates it with a record
                                 of the output.
                                 <pre class="pre screen" xml:space="preserve">$ make gen_supp
 /usr/local/cuda/compute-sanitizer/compute-sanitizer --save supp.xml --xml suppressions_demo
 ========= COMPUTE-SANITIZER
 ========= Program hit cudaErrorMemoryAllocation (error 2) due to "out of memory" on CUDA API call to cudaMalloc.
 [...]
 $ cat supp.xml
 
 &lt;?xml version="1.0" encoding="utf-8"?&gt;
   &lt;ComputeSanitizerOutput&gt;
     &lt;record&gt;
       &lt;kind&gt;Api&lt;/kind&gt;
       &lt;what&gt;
         &lt;text&gt;Program hit cudaErrorMemoryAllocation (error 2) due to out of memory on CUDA API call to cudaMalloc.&lt;/text&gt;
         &lt;api&gt;cudaMalloc&lt;/api&gt;
         &lt;error&gt;cudaErrorMemoryAllocation&lt;/error&gt;
         &lt;message&gt;out of memory&lt;/message&gt;
         &lt;result&gt;2&lt;/result&gt;
       &lt;/what&gt;
       &lt;hostStack&gt;
       [...]
       &lt;/hostStack&gt;
 </pre></div>
                              <div class="p">
                                 Now, we can use that file as input to run the tool, along with the <samp class="ph codeph">--suppressions</samp> option to ignore that error.
                                 <pre class="pre screen" xml:space="preserve">$ make run_memcheck_with_supp
/usr/local/cuda/compute-sanitizer/compute-sanitizer --suppressions supp.xml suppressions_demo
========= COMPUTE-SANITIZER
========= ERROR SUMMARY: 0 errors</pre></div>
                              <p class="p">
                                 The XML file can be edited to change which errors are ignored. For instance, a regular expression can be used in the api tag
                                 to suppress a range of API calls. For instance <samp class="ph codeph">cuda.*</samp> will ignore any errors related to an API starting with cuda.
                                 
                              </p>
                              <p class="p">
                                 Other tags that can be edited are the result and hoststack ones. Note that the host stack appears in reverse order and the
                                 suppressions
                                 feature will compare every stack frame that was recorded.
                                 
                              </p>
                           </div>
                        </div>
                     </div>
                     <div class="topic concept nested2" id="suppressions-demo-initcheck"><a name="suppressions-demo-initcheck" shape="rect">
                           <!-- --></a><h3 class="title topictitle2"><a href="#suppressions-demo-initcheck" name="suppressions-demo-initcheck" shape="rect">11.5.2.&nbsp;Initcheck error suppression</a></h3>
                        <div class="body conbody">
                           <p class="p">
                              The API error application can be found on the <a class="xref" href="https://github.com/NVIDIA/compute-sanitizer-samples/tree/master/Suppressions" target="_blank" shape="rect">compute-sanitizer github repository</a></p>
                           <div class="p">
                              This application can be compiled using the provided Makefile:
                              <pre class="pre screen" xml:space="preserve">make</pre></div>
                           <div class="section">
                              <div class="p">
                                 In this example, we have a simple multiplication kernel. A call to <samp class="ph codeph">cudaMemset</samp> is used
                                 to initialize the device memory to 0. However, it does not initialize the last byte of the array. The
                                 initcheck tool detects the unitialized access:
                                 <pre class="pre screen" xml:space="preserve">$ make run_initcheck
/usr/local/cuda/compute-sanitizer/compute-sanitizer --tool initcheck suppressions_initcheck_demo
========= COMPUTE-SANITIZER
========= Uninitialized __global__ memory read of size 4 bytes
=========     at mult(int *, int *, int)+0x60
=========     by thread (122,0,0) in block (0,0,0)
=========     Address 0x7f936fa001e8
[...]
========= ERROR SUMMARY: 1 error
</pre></div>
                              <div class="p">
                                 In a similar fashion to the previous example, we can use the <samp class="ph codeph">--xml</samp> option to
                                 generate a suppression file.
                                 <pre class="pre screen" xml:space="preserve">$ make initcheck_gen_supp
/usr/local/cuda/compute-sanitizer/compute-sanitizer --tool initcheck --save supp.xml --xml suppressions_initcheck_demo
========= COMPUTE-SANITIZER
========= Uninitialized __global__ memory read of size 4 bytes
[...]</pre></div>
                              <div class="p">
                                 Now, the error can be ignored using the XML file as input to the suppressions feature.
                                 <pre class="pre screen" xml:space="preserve">$ make run_initcheck_with_supp
/usr/local/cuda/compute-sanitizer/compute-sanitizer --tool initcheck --suppressions supp.xml suppressions_initcheck_demo
========= COMPUTE-SANITIZER
========= ERROR SUMMARY: 0 errors</pre></div>
                              <p class="p">
                                 As with the API suppressions, the XML file can be edited to make the suppressions detection more generic, by editing or removing
                                 the threadId, blockId, size and device stack tags.
                                 
                              </p>
                           </div>
                        </div>
                     </div>
                  </div>
               </div>
               <div class="topic concept nested0" id="notices-header"><a name="notices-header" shape="rect">
                     <!-- --></a><h2 class="title topictitle1"><a href="#notices-header" name="notices-header" shape="rect">Notices</a></h2>
                  <div class="topic reference nested1" id="notice"><a name="notice" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#notice" name="notice" shape="rect"></a></h3>
                     <div class="body refbody">
                        <div class="section">
                           <h3 class="title sectiontitle">Notice</h3>
                           <p class="p">ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND
                              SEPARATELY, "MATERIALS") ARE BEING PROVIDED "AS IS." NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE
                              WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS
                              FOR A PARTICULAR PURPOSE. 
                           </p>
                           <p class="p">Information furnished is believed to be accurate and reliable. However, NVIDIA Corporation assumes no responsibility for the
                              consequences of use of such information or for any infringement of patents or other rights of third parties that may result
                              from its use. No license is granted by implication of otherwise under any patent rights of NVIDIA Corporation. Specifications
                              mentioned in this publication are subject to change without notice. This publication supersedes and replaces all other information
                              previously supplied. NVIDIA Corporation products are not authorized as critical components in life support devices or systems
                              without express written approval of NVIDIA Corporation.
                           </p>
                        </div>
                     </div>
                  </div>
                  <div class="topic reference nested1" id="trademarks"><a name="trademarks" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#trademarks" name="trademarks" shape="rect"></a></h3>
                     <div class="body refbody">
                        <div class="section">
                           <h3 class="title sectiontitle">Trademarks</h3>
                           <p class="p">NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation
                              in the U.S. and other countries.  Other company and product names may be trademarks of
                              the respective companies with which they are associated.
                           </p>
                        </div>
                     </div>
                  </div>
                  <div class="topic reference nested1" id="copyright-past-to-present"><a name="copyright-past-to-present" shape="rect">
                        <!-- --></a><h3 class="title topictitle2"><a href="#copyright-past-to-present" name="copyright-past-to-present" shape="rect"></a></h3>
                     <div class="body refbody">
                        <div class="section">
                           <h3 class="title sectiontitle">Copyright</h3>
                           <p class="p">© <span class="ph">2019</span>-<span class="ph">2024</span> NVIDIA
                              Corporation and affiliates. All rights reserved.
                           </p>
                           <p class="p">This product includes software developed by the Syncro Soft SRL (http://www.sync.ro/).</p>
                        </div>
                     </div>
                  </div>
               </div>
               <div class="fn"><a name="fntarg_1" href="#fnsrc_1" shape="rect"><sup>1</sup></a>  In some cases, there may be no device backtrace
               </div>
               
               <hr id="contents-end"></hr>
               
            </article>
         </div>
      </div>
      <script language="JavaScript" type="text/javascript" charset="utf-8" src="../common/formatting/common.min.js"></script>
      <script language="JavaScript" type="text/javascript" charset="utf-8" src="../common/scripts/google-analytics/google-analytics-write.js"></script>
      <script language="JavaScript" type="text/javascript" charset="utf-8" src="../common/scripts/google-analytics/google-analytics-tracker.js"></script>
      <script type="text/javascript">_satellite.pageBottom();</script>
      <script type="text/javascript">var switchTo5x=true;</script><script type="text/javascript">stLight.options({publisher: "998dc202-a267-4d8e-bce9-14debadb8d92", doNotHash: false, doNotCopy: false, hashAddressBar: false});</script></body>
</html>