File: mysql-test-run.dox

package info (click to toggle)
mysql-8.0 8.0.43-3
links: PTS, VCS
area: main
in suites: sid
size: 1,273,924 kB
sloc: cpp: 4,684,605; ansic: 412,450; pascal: 108,398; java: 83,641; perl: 30,221; cs: 27,067; sql: 26,594; sh: 24,181; python: 21,816; yacc: 17,169; php: 11,522; xml: 7,388; javascript: 7,076; makefile: 2,194; lex: 1,075; awk: 670; asm: 520; objc: 183; ruby: 97; lisp: 86
file content (8394 lines) | stat: -rw-r--r-- 283,422 bytes
########## The MySQL Test Framework ##########


/**
  @page PAGE_MYSQL_TEST_RUN The MySQL Test Framework

  This manual describes the MySQL test framework, consisting of the
  test driver and the test script language.

  -----

  <h3>Table of Contents</h3>
  - @subpage PAGE_PREFACE
  - @subpage PAGE_INTRODUCTION
  - @subpage PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS
  - @subpage PAGE_RUNNING_TESTCASES
  - @subpage PAGE_WRITING_TESTCASES
  - @subpage PAGE_MYSQLTEST_PROGRAMS
  - @subpage PAGE_MYSQLTEST_LANGUAGE_REFERENCE
  - @subpage PAGE_UNIT_TESTS
  - @subpage PAGE_PLUGINS
*/

###############################################################################



## Preface

/**
  @page PAGE_PREFACE Preface

  MySQL distributions include a set of test cases and programs for
  running them. These tools constitute the MySQL test framework that
  provides a means for verifying that MySQL Server and its client
  programs operate according to expectations. The test cases consist
  mostly of SQL statements, but can also use test language constructs
  that control how to run tests and verify their results.

  This manual describes the MySQL test framework. It describes the
  programs used to run tests and the language used to write test
  cases.
*/

###############################################################################



## Introduction to the MySQL Test Framework

/**
  @page PAGE_INTRODUCTION Introduction to the MySQL Test Framework

  MySQL distributions include a test suite: a set of test cases and
  programs for running them. (If you find that the test suite is not
  included in your distribution, look for a similar distribution with
  <b>-test</b> in the name and install that as well.) These tools
  constitute the MySQL test framework that provides a means for
  verifying that MySQL Server and its client programs operate
  according to expectations. The test cases consist mostly of SQL
  statements, but can also use test language constructs that control
  how to run tests and verify their results. Distributions also
  provide facilities for running unit tests and creating new unit
  tests.

  This document describes the components of the MySQL test framework,
  how the test programs work, and the language used for writing test
  cases. It also provides a tutorial for developing test cases and
  executing them.

  The application that runs the test suite is named @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". Its location is the
  <b>mysql-test</b> directory, which is present both in source and
  binary MySQL Server distributions.

  On platforms other than Windows, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" is also available through the shortened name
  <b>mtr</b> in the same directory, as either a symbolic link or a
  copy.

  The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" application
  starts MySQL servers, restarts them as necessary when a specific
  test case needs different start arguments, and presents the test
  result. For each test case, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" invokes the @ref PAGE_MYSQLTEST "mysqltest"
  program (also referred to as the “test engine”) to read the test
  case file, intepret the test language constructs, and send SQL
  statements to the server.

  Input for each test case is stored in a file, and the expected
  result from running the test is stored in another file. The actual
  result is compared to the expected result after running the test.

  For a MySQL source distribution, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" is located in the <b>mysql-test</b> directory,
  and @ref PAGE_MYSQLTEST "mysqltest" is located in the <b>client</b>
  directory. The <b>mysql-test</b> and <b>client</b> directories
  are located in the root directory of the distribution.

  For a MySQL binary distribution, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" is located in the <b>mysql-test</b> directory,
  and @ref PAGE_MYSQLTEST "mysqltest" is located in the same directory
  where other client programs such as <b>mysql</b> or
  <b>mysqladmin</b> are installed. The locations of the
  <b>mysql-test</b> and other directories depend on the layout used
  for the distribution format.

  Within the <b>mysql-test</b> directory, test case input files and
  result files are stored in the <b>t</b> and <b>r</b>
  directories, respectively. The input and result files have the same
  basename, which is the test name, but have extensions of
  <b>.test</b> and <b>.result</b>, respectively. For example, for
  a test named “decimal”, the input and result files are
  <b>mysql-test/t/decimal.test</b> and <b>mysql-test/r/decimal.result</b>.

  Each test file is referred to as one test case, but usually consists
  of a sequence of related tests. An unexpected failure of a single
  statement in a test case makes the test case fail.

  There are several ways a test case can fail:

  <ul>
    <li>
      The @ref PAGE_MYSQLTEST "mysqltest" test engine checks the result
      codes from executing each SQL statement in the test input. If the
      failure is unexpected, the test case fails.
    </li>

    <li>
      A test case can fail if an error was expected but did not occur
      (for example, if an SQL statement succeeded when it should have
      failed).
    </li>

    <li>
      The test case can fail by producing incorrect output. As a test
      runs, it produces output (the results from <b>SELECT</b>,
      <b>SHOW</b>, and other statements). This output is compared
      to the expected result found in the <b>mysql-test/r</b> directory
      (in a file with a <b>.result</b> suffix). If the expected and
      actual results differ, the test case fails. The actual test result
      is written to a file in the <b>mysql-test/var/log</b> directory
      with a <b>.reject</b> suffix, and the difference between the
      <b>.result</b> and <b>.reject</b> files is presented for
      evaluation.
    </li>

    <li>
      A test case will fail if the MySQL server dies unexpectedly during
      the test. If this happens, the @ref PAGE_MYSQLTEST "mysqltest"
      test client will usually also report a failure due to loosing the
      connection.
    </li>

    <li>
      Finally, the test case will fail if the error log written by the
      MySQL server during the test includes warnings or errors which are
      not filtered (suppressed). See @ref PAGE_SUPPRESSING_ERRORS_WARNINGS
      for more about suppressing warnings.
    </li>
  </ul>

  This method of checking test results puts some restrictions on how
  test cases can be written. For example, the result cannot contain
  information that varies from run to run, such as the current time.
  However, if the information that varies is unimportant for test
  evaluation, there are ways to instruct the test engine to replace
  those fields in the output with fixed values.

  Because the test cases consist mostly of SQL statements in a text
  file, there is no direct support for test cases that are written in
  C, Java, or other languages. Such tests are not within the scope of
  this test framework. But the framework does support executing your
  own scripts and initiating them with your own data. Also, a test
  case can execute an external program, so in some respects the test
  framework can be extended for uses other than testing SQL
  statements. Finally, it is possible to embed small pieces of Perl
  code within the test. This can sometimes be used to perform actions
  or execute logic which is beyond the capabilities of the test
  language or SQL.
*/

###############################################################################



## MySQL Test Framework Components

/**
  @page PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS MySQL Test Framework Components

  <h3>Table of Contents</h3>
  - @subpage PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS
  - @subpage PAGE_REPORT_BUGS

  The MySQL test framework consists of programs that run tests, and
  directories and files used by those programs.

  Test Framework Programs
  -----------------------

  The MySQL test framework uses several programs:

  <ul>
    <li>
      The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" Perl script
      is the main application used to run the test suite. It invokes
      @ref PAGE_MYSQLTEST "mysqltest" to run individual test cases.
    </li>

    <li>
      @ref PAGE_MYSQLTEST "mysqltest" runs test cases.
    </li>

    <li>
      The @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" program is
      used for testing aspects of the MySQL client API that cannot be
      tested using @ref PAGE_MYSQLTEST "mysqltest" and its test
      language.
    </li>

    <li>
      The @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" Perl
      script performs stress-testing of the MySQL server.
    </li>

    <li>
      A unit-testing facility is provided so that individual unit test
      programs can be created for storage engines and plugins.
    </li>
  </ul>

  Test suite programs can be found in these locations:

  <ul>
    <li>
      For a source distribution, @ref PAGE_MYSQLTEST "mysqltest" is in
      the <b>client</b> directory. For a binary distribution, it is in
      the MySQL <b>bin</b> directory.
    </li>

    <li>
      For a source distribution, @ref PAGE_MYSQL_CLIENT_TEST
      "mysql_client_test" is in the <b>tests</b> directory. For a
      binary distribution, it is in the MySQL <b>bin</b> directory.
    </li>

    <li>
      The other programs are located in the <b>mysql-test</b>
      directory. For a source distribution, <b>mysql-test</b> is found
      under the source tree root. For a binary distribution, the
      location of <b>mysql-test</b> depends on the layout used for the
      distribution format.
    </li>
  </ul>


  Test Framework Directories and Files
  -------------------------------------

  The test suite is located in the <tt>mysql-test</tt> directory,
  which contains the following components:

  <ul>
    <li>
      The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" and @ref
      PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" programs that
      are used for running tests.
    </li>

    <li>
      The <b>t</b> directory contains test case input files. A test
      case file might also have additional files associated with it.
    </li>

    <ul>
      <li>
        A file name of the form <b><em>test_name</em>.test</b> is a
        test case file for a test named <b><em>test_name</em></b>. For
        example, <b>subquery.test</b> is the test case file for the
        test named <b>subquery</b>.
      </li>

      <li>
        A file name of the form <b><em>test_name</em>-master.opt</b>
        provides options to associate with the named test case.
        <b>mysql-test-run.pl</b> restarts the server with the options
        given in the file if the options are different from those
        required for the currently running server.

        Note that the <b>-master.opt</b> file is used for the “main”
        server of a test, even if no replication is involved.
      </li>

      <li>
        A file name of the form <b><em>test_name</em>-slave.opt</b>
        provides slave options.
      </li>

      <li>
        Server options which need to be passed during initialization
        of the server can be passed in both
        <b><em>test_name</em>-master.opt</b> and
        <b><em>test_name</em>-slave.opt</b>. Each bootstrap variable
        must be passed as the value of a <b>`--initialize`</b> option so
        that @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" can
        recognize that the variable should be used during server
        initialization, remove the existing data directory, and
        initialize a new data directory with the options provided.
        @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will also
        start the server afterwards, with the bootstrap options, and
        other options specified.
      </li>

      <li>
        A file name of the form <b><em>test_name</em>-client.opt </b>
        provides @ref PAGE_MYSQLTEST "mysqltest" client options.
      </li>

      <li>
        A file name of the form <b><em>test_name</em>.cnf</b> contains
        additional entries for the config file to be used for this
        test.

        <ul>
          <li>
            During a test run, each server is started with a different
            <b>defaults-group-suffix</b> value and this suffix value
            can be used with a group name to specify server specific
            options. For example, section <b>[mysqld.1]</b> is used to
            specify options for mysqld server with suffix value
            <b>".1"</b>.
          </li>

          <li>
            The <b>[mysqltest]</b> section is used to specify options
            for mysqltest client.
          </li>

          <li>
            The <b>[ENV]</b> section is used to specify environment
            variables for a test case.
          </li>

          <li>
            To include the options mentioned in another <b>.cnf</b>
            file, <b>!include path_to_cnf_file</b> is used. Path to
            the included <b>.cnf</b> should either be an absolute path
            or a relative path from current location or from
            <b>mysql-test</b> directory.
          </li>
        </ul>

        Here is a sample <b>.cnf</b> file.

        @verbatim
  !include include/default_my.cnf

  [mysqld.1]
  Options for server mysqld.1

  [mysqld.2]
  Options for server mysqld.2

  [mysqltest]
  ps-protocol

  ...
  ...
  ...

  [ENV]
  SERVER_MYPORT_1= @mysqld.1.port
  SERVER_MYPORT_2= @mysqld.2.port
        @endverbatim
      </li>

      <li>
        A file name of the form <b><em>test_name</em>.combinations</b>
        provides section of options for each test run.
      </li>

      <li>
        A file name of the form <b><em>suite</em>.opt</b> provides
        server options to all the tests within the suite it belongs
        to.

        If a test run is started with more than one server, the
        options specified in a <b><em>suite</em>.opt</b> are used by
        all of them. It is also possible to pass the server options
        needed during the server initialization in the
        <b><em>suite</em>.opt</b> file.

        The options mentioned in the <b><em>suite</em>.opt</b> file
        are overridden by those found in a
        <b><em>test_name</em>-master.opt</b> file or in a
        <b><em>test_name</em>-slave.opt</b> file.
      </li>

      <li>
        A file name of the form <b><em>test_name</em>-master.sh</b> is
        a shell script that will be executed before the server is
        started for the named test case. There may also be a
        <b><em>test_name</em>-slave.sh</b> which is executed before
        the slave server is started.

        These files will not work on Windows and may therefore be
        replaced with a more portable mechanism in the future.
      </li>

      <li>
        The <b>disabled.def</b> file contains information about
        deferred/disabled tests. When a test is failing because of a
        bug in the server and you want it to be ignored by @ref
        PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", list the test in
        this file.

        The format of a line in the <b>disabled.def</b> file looks
        like this, where fields are separated by one or more spaces.

        @verbatim
  suite_name.test_name [@platform|@!platform] : <BUG|WL>#<XXXX> [<comment>]
        @endverbatim

        Example:

        @verbatim
  main.1st @linux : BUG#18980 Test fails randomly
        @endverbatim

        <b><em>suite_name</em></b> is the suite name.
        <b><em>test_name</em></b> is the test case name.

        An additional element after the test name and before the
        colon you may add a <b>\@<em>platform</em></b> to have the
        test disabled only on a specific platform. Basic OS names as
        reported by <b>$^O</b> in Perl, or <b>'windows'</b> are
        supported. Alternatively, <b>@!<em>platform</em></b> will
        disable the test on all except the named platform.

        <b>BUG#<em>nnnnn</em></b> or <b>WL#<em>nnnn</em></b> indicates
        the bug or the WL related to the test that causes it to fail
        (and thus requires it to be disabled).

        The <b><em>comment</em></b> text following the
        <b>BUG#<em>nnnnn</em></b> or the <b>WL#<em>nnnn</em></b> is
        not part of the mandatory syntax for this file,
        @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will not
        actually care what is written here. Length of a
        <b><em>comment</em></b> text must not be more than <b>80</b>
        characters.

        A comment line can be written in the file by beginning the
        line with a “<b>#</b>” character.
      </li>
    </ul>

    <li>
      The <b>r</b> directory contains test case result files:

      <ul>
        <li>
          A file name of the form <b><em>test_name</em>.result</b> is
          the expected result for the named test case. A file
          <b>r/<em>test_name</em>.result</b> is the output that
          corresponds to the input in the test case file
          <b>t/<em>test_name</em>.test</b>.
        </li>

        <li>
          A file name of the form <b><em>test_name</em>.reject</b>
          contains output for the named test case if the test fails
          due to output mismatch (but not it it fails for other
          reasons).
        </li>
      </ul>

      For a test case that succeeds, the <b>.result</b> file
      represents both the expected and actual result. For a test case
      that fails, the <b>.result</b> file represents the expected
      result, and the <b>.reject</b> file represents the actual
      result.

      If a <b>.reject</b> file is created because a test fails, @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" removes the file
      later the next time the test succeeds.

      When <b>`--check-testcases`</b> option is enabled, a test case
      not having its corresponding <b>.result</b> file is marked as
      failed by MTR. See @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl".
    </li>

    <li>
      The <b>include</b> directory contains files that are included by
      test case files using the <b>source</b> command. These include
      files encapsulate operations of varying complexity into a single
      file so that you can perform the operations in a single step.
      See @ref PAGE_INCLUDE_FILES.
    </li>

    <li>
      The <b>lib</b> directory contains library files used by @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", and database
      initialization SQL code.
    </li>

    <li>
      The <b>std_data</b> directory contains data files used by some
      of the tests.
    </li>

    <li>
      The <b>var</b> directory is used during test runs for various
      kinds of files: log files, temporary files, trace files, Unix
      socket files for the servers started during the tests, and so
      forth. This directory cannot be shared by simultaneous test
      runs.
    </li>

    <li>
      The <b>suite</b> directory contains a set of subdirectories
      containing named test suites. Each subdirectory represents a
      test suite with the same name.

      A test suite directory contains the following components:

      <ul>
        <li>
          The <b>t</b> directory contains test cases.
        </li>

        <li>
          The <b>r</b> directory contains test case result files.
        </li>

        <li>
          The <b>include</b> directory contains files that are
          included by the test cases belonging to that suite.
        </li>

        <li>
          A file name of the form <b>combinations</b> provides section
          of options for each test run. See @ref
          PAGE_CONTROLLING_BINARY_LOG_FORMAT.
        </li>

        <li>
          A file name of the form <b><em>my</em>.cnf</b> contains
          additional entries for the config file used by all the tests
          within the suite it belongs to.

          A suite specific <b><em>my</em>.cnf</b> file is overridden
          by a <b><em>test_name</em>.cnf</b> file.
        </li>
      </ul>
    </li>

    <li>
      The <b>collections</b> directory contains collections of test
      runs that are run during integration and release testing of
      MySQL. They are not directly useful outside this context, but
      need to be part of the source repository and are included for
      reference.
    </li>
  </ul>

  Unit test-related files are located in the <b>unittest</b>
  directory. Additional files specific to storage engines and plugins
  may be present under the subdirectories of the <b>storage</b> or
  <b>plugin</b> directories.


  Test Execution and Evaluation
  -----------------------------

  There are a number of targets in the top-level <b>Makefile</b> that
  can be used to run sets of tests. <b>make test</b> only runs unit
  tests. Other targets run subsets of the tests, or run tests with
  specific options for the test programs. Have a look at the
  <b>Makefile</b> to see what targets are available.

  A “test case” is a single file. The case might contain multiple
  individual test commands. If any individual command fails, the
  entire test case is considered to fail. Note that “fail” means “does
  not produce the expected result.” It does <b><em>not</em></b>
  necessarily mean “executes without error,” because some tests are
  written precisely to verify that an illegal statement does in fact
  produce an error. In such an instance, if the statement executes
  successfully without producing the expected error, that is
  considered failure of the test.

  Test case output (the test result) consists of:

  <ul>
    <li>
      Input SQL statements and their output. Each statement is written
      to the result followed by its output. Columns in output
      resulting from SQL statements are separated by tab characters.
    </li>

    <li>
      The result from @ref PAGE_MYSQLTEST "mysqltest" commands such as
      <b>echo</b> and <b>exec</b>. The commands themselves are not
      echoed to the result, only their output.
    </li>
  </ul>

  The <b>disable_query_log</b> and <b>enable_query_log</b> commands
  control logging of input SQL statements. The
  <b>disable_result_log</b> and <b>enable_result_log</b> commands
  control logging of SQL statement results, and warning or error
  messages resulting from those statements.

  @ref PAGE_MYSQLTEST "mysqltest" reads a test case file from its
  standard input by default. The <b>`--test-file`</b> or <b>-x</b>
  option can be given to name a test case file explicitly.

  @ref PAGE_MYSQLTEST "mysqltest" writes test case output to the
  standard output by default. The <b>`--result-file`</b> or <b>-R</b>
  option can be used to indicate the location of the result file. That
  option, together with the <b>`--record`</b> option, determine how
  @ref PAGE_MYSQLTEST "mysqltest" treats the test actual and expected
  results for a test case:

  <ul>
    <li>
      If the test produces no results, @ref PAGE_MYSQLTEST "mysqltest"
      exits with an error message to that effect, unless
      <b>`--result-file`</b> is given and this file is empty.
    </li>

    <li>
      Otherwise, if <b>`--result-file`</b> is not given, @ref
      PAGE_MYSQLTEST "mysqltest" sends test results to the standard
      output.
    </li>

    <li>
      With <b>`--result-file`</b> but not <b>`--record`</b>, @ref
      PAGE_MYSQLTEST "mysqltest" reads the expected results from the
      given file and compares them with the actual results. If the
      results do not match, @ref PAGE_MYSQLTEST "mysqltest" writes a
      <b>.reject</b> file in the log directory and exits with an
      error. It will also print out a diff of the expected and actual
      result, provided it can find an appropriate
      <b>diff</b> tool to use.
    </li>

    <li>
      With both <b>`--result-file`</b> and <b>`--record`</b>, @ref
      PAGE_MYSQLTEST "mysqltest" updates the given file by writing the
      actual test results to it. The file does not need to pre-exist.
    </li>
  </ul>

  @ref PAGE_MYSQLTEST "mysqltest" itself knows nothing of the <b>t</b>
  and <b>r</b> directories under the <b>mysql-test</b> directory. The
  use of files in those directories is a convention that is used by
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", which invokes @ref
  PAGE_MYSQLTEST "mysqltest" with the appropriate options for each
  test case to tell @ref PAGE_MYSQLTEST "mysqltest" where to read input
  and write output.
*/

###############################################################################



## Test Framework System Requirements

/**
  @page PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS Test Framework System Requirements

  The @ref PAGE_MYSQLTEST "mysqltest" and @ref PAGE_MYSQL_CLIENT_TEST
  "mysql_client_test" programs are written in C++ and are available on
  any system where MySQL itself can be compiled, or for which a binary
  MySQL distribution is available.

  Other parts of the test framework such as @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" are Perl scripts and
  should run on systems with Perl installed.

  @ref PAGE_MYSQLTEST "mysqltest" uses the <b>diff</b> program to
  compare expected and actual test results. If <b>diff</b> is not
  found, @ref PAGE_MYSQLTEST "mysqltest" writes an error message and
  dumps the entire contents of the <b>.result</b> and
  <b>.reject</b> files so that you can try to determine why a test
  did not succeed. If your system does not have <b>diff</b>, you may
  be able to obtain it from one of these sites:

  - http://www.gnu.org/software/diffutils/diffutils.html
  - http://gnuwin32.sourceforge.net/packages/diffutils.htm

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" cannot function
  properly if started from within a directory whose full path includes
  a space character, due to the complexities of handling this
  correctly in all the different contexts it will be used.
*/

###############################################################################


## How to Report Bugs from Tests in the MySQL Test Suite

/**
  @page PAGE_REPORT_BUGS How to Report Bugs from Tests in the MySQL Test Suite

  If test cases from the test suite fail, you should do the following:

  <ul>
    <li>
      Do not file a bug report before you have found out as much as
      possible about what when wrong. See the instructions at
      https://dev.mysql.com/doc/refman/8.0/en/bug-reports.html.
    </li>

    <li>
      Make sure to include the output of @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl", as well as contents of all <b>.reject</b>
      files in the <b>mysql-test/var/log</b> directory, or the (often
      much shorter) diff that @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" reported.
    </li>

    <li>
      Check whether an individual test in the test suite also fails
      when run on its own:

      @verbatim
  shell> cd mysql-test
  shell> ./mysql-test-run.pl test_name
      @endverbatim

      If this fails, and you are compiling MySQL yourself, you should
      configure MySQL with <b>`--with-debug`</b> and run @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the
      <b>`--debug`</b> option. If this also fails, open a bug report
      and upload the trace file <b>mysql-test/var/tmp/master.trace</b>
      to the report, so that we can examine it. For instructions, see
      [How to Report Bugs or Problems]
      (https://dev.mysql.com/doc/refman/8.0/en/bug-reports.html).
      Please remember to also include a full description of your
      system, the version of the <b>mysqld</b> binary and how you
      compiled it.
    </li>

    <li>
      Run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the
      <b>`--force`</b> option to see whether any other tests fail.
    </li>

    <li>
      If you have compiled MySQL yourself, check the MySQL Reference
      Manual to see whether there are any platform-specific issues for
      your system. There might be configuration workarounds to deal
      with the problems that you observe. Also, consider using one of
      the binaries we have compiled for you at
      https://dev.mysql.com/downloads/. All our standard binaries
      should pass the test suite!
    </li>

    <li>
      If you get an error such as <b>Result length mismatch</b> or
      <b>Result content mismatch</b> it means that the output of the
      test was not an exact match for the expected output. This could
      be a bug in MySQL or it could be that your version of
      <b>mysqld</b> produces slightly different results under some
      circumstances.

      This output from @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
      should include a diff of the expected and actual result. If
      unable to produce a diff, it will instead print out both in
      full. In addition, the actual result is put in a file in the
      <b>r</b> directory with a <b>.result</b> extension.
    </li>

    <li>
      If a test fails completely, you should check the logs file in
      the <b>mysql-test/var/log</b> directory for hints of what went
      wrong.
    </li>

    <li>
      If you have compiled MySQL with debugging, you can try to debug
      test failures by running @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" with either or both of the <b>`--gdb`</b>
      and <b>`--debug`</b> options.

      If you have not compiled MySQL for debugging you should probably
      do so by specifying the [-DWITH_DEBUG]
      (https://dev.mysql.com/doc/refman/8.0/en/source-configuration-options.html#option_cmake_with_debug)
      option when you invoke <b>CMake</b>.
    </li>
  </ul>
*/

###############################################################################



## Running Test Cases

/**
  @page PAGE_RUNNING_TESTCASES Running Test Cases

  Typically, you run the test suite either from within a source tree
  (after MySQL has been built), or on a host where the MySQL server
  distribution has been installed. (If you find that the test suite is
  not included in your distribution, look for a similar distribution
  with <b>-test</b> in the name and install that as well.)

  To run tests, your current working directory should be the
  <b>mysql-test</b> directory of your source tree or installed
  distribution. In a source distribution, <b> mysql-test</b> is
  under the root of the source tree. In a binary distribution, the
  location of <b>mysql-test</b> depends on the distribution layout.
  The program that runs the test suite, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl", will figure out whether you are in a source
  tree or an installed directory tree.

  To run the test suite, change location into your <b>mysql-test</b>
  directory and invoke the @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" script:

  @verbatim
  shell> cd mysql-test
  shell> ./mysql-test-run.pl
  @endverbatim

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" accepts options on
  the command line. For example:

  @verbatim
  shell> ./mysql-test-run.pl --force --suite=binlog
  @endverbatim

  By default, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" exits if
  a test case fails. <b>`--force`</b> causes execution to continue
  regardless of test case failure.

  For a full list of the supported options, see @ref
  PAGE_MYSQL_TEST_RUN_PL.

  To run one or more specific test cases, name them on the @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" command line. Test case
  files have names like <b>t/<em>test_name</em>.test</b>, where
  <b><em>test_name</em></b> is the name of the test case, but each name
  given on the command line should be the test case name, not the full
  test case file name. The following command runs the test case named
  <b>rpl_abcd</b>, which has a test file of
  <b>t/rpl_abcd.test</b>:

  @verbatim
  shell> ./mysql-test-run.pl rpl_abcd
  @endverbatim

  To run a family of test cases for which the names share a common
  prefix, use the <b>`--do-test`</b> option:

  @verbatim
  shell> ./mysql-test-run.pl --do-test=prefix
  @endverbatim

  For example, the following command runs the events tests (test
  cases that have names beginning with <b>events</b>):

  @verbatim
  shell> ./mysql-test-run.pl --do-test=events
  @endverbatim

  To run a specific named testsuite with all the test cases in it,
  use the <b>`--suite`</b> option:

  @verbatim
  shell> ./mysql-test-run.pl --suite=suite_name
  @endverbatim

  For example, the following command runs the replication tests
  located in the <b>rpl</b> suite:

  @verbatim
  shell> ./mysql-test-run.pl --suite=rpl
  @endverbatim

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" starts the MySQL
  server, sets up the environment for calling the @ref PAGE_MYSQLTEST
  "mysqltest" program, and invokes @ref PAGE_MYSQLTEST "mysqltest" to
  run the test case. For each test case to be run, @ref PAGE_MYSQLTEST
  "mysqltest" handles operations such as reading input from the test
  case file, creating server connections, and sending SQL statements
  to servers.

  The language used in test case files is a mix of commands that the
  @ref PAGE_MYSQLTEST "mysqltest" program understands and SQL
  statements. Input that @ref PAGE_MYSQLTEST "mysqltest" doesn't
  understand is assumed to consist of SQL statements to be sent to the
  database server. This makes the test case language familiar to those
  that know how to write SQL and powerful enough to add the control
  needed to write test cases.

  You need not start a MySQL server first before running tests.
  Instead, the @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" program
  will start the server or servers as needed. Any servers started for
  the test run use ports in the range from 13000 by default.

  @section SECTION_RUNNING_TESTS_IN_PARALLEL Running Tests in Parallel

  It is possible to run more than one instance of @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" simultaneously on the
  same machine. Both will by default use server ports from 13000 but
  will coordinate used port numbers as well as check for availibility,
  to avoid conflicts.

  Running several instances from the same <b>mysql-test</b>
  directory is possible but problematic. You must the use the
  <b>`--vardir`</b> to set different log directories for each
  instance. Even so, you can get into trouble becuse they will write
  <b>.reject</b> files to the same directories.

  It is also possible to have a single @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" run tests in several threads in parallel.
  Execution of the tests will be distributed among the threads. This
  is achieved using the <b>`--parallel`</b> option, with the number
  of threads as argument. The special value <b>auto</b> will ask
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to pick a value
  automatically, based on system information. The parallel option may
  also be given using the environment variable <b>MTR_PARALLEL</b>.

  @note
  Test cases which use <b>not_parallel.inc</b> are run at the end with
  a parallel value of <b>1</b> overriding the <b>`--parallel`</b>
  option value. Such test cases will be executed at the end of the test
  run one at a time.
*/

###############################################################################



## Writing Test Cases

/**
  @page PAGE_WRITING_TESTCASES Writing Test Cases

  <h3>Table of Contents</h3>
  - @subpage PAGE_QUICK_START
  - @subpage PAGE_TESTCASE_CODING_GUIDELINES
  - @subpage PAGE_SAMPLE_TESTCASES
  - @subpage PAGE_CLEANUP_PREVIOUS_TEST_RUN
  - @subpage PAGE_GENERATE_TESTCASE_RESULT_FILE
  - @subpage PAGE_CHECK_EXPECTED_ERRORS
  - @subpage PAGE_CONTROL_INFORMATION_PRODUCED_TESTCASE
  - @subpage PAGE_DEALING_OUTPUT
  - @subpage PAGE_PASSING_OPTIONS
  - @subpage PAGE_TESTCASE_SPECIFIC_SERVER_OPTIONS
  - @subpage PAGE_TESTCASE_SPECIFIC_BOOTSTRAP_OPTIONS
  - @subpage PAGE_TESTCASE_SPECIFIC_CLIENT_OPTIONS
  - @subpage PAGE_INCLUDE_FILES
  - @subpage PAGE_BINARY_LOG_FORMAT
  - @subpage PAGE_WRITING_REPLICATION_TESTS
  - @subpage PAGE_THREAD_SYNCHRONIZATION
  - @subpage PAGE_SUPPRESSING_ERRORS_WARNINGS
  - @subpage PAGE_STOPPING_AND_RESTARTING_SERVER_DURING_TEST
  - @subpage PAGE_TIPS_WRITING_TESTCASES

  Normally, you run the test suite during the development process to
  ensure that your changes do not cause existing test cases to break.
  You can also write new test cases or add tests to existing cases.
  This happens when you fix a bug (so that the bug cannot reappear
  later without being detected) or when you add new capabilities to
  the server or other MySQL programs.

  This chapter provides guidelines for developing new test cases for
  the MySQL test framework.

  Some definitions:

  - One “test file” is one “test case”.

  - One “test case” might contain a “test sequence” (that is, a
    number of individual tests that are grouped together in the same
    test file).

  - A “command” is an input test that @ref PAGE_MYSQLTEST "mysqltest"
    recognizes and executes itself. A “statement” is an SQL statement
    or query that @ref PAGE_MYSQLTEST "mysqltest" sends to the MySQL
    server to be executed.
*/

###############################################################################



## Writing a Test Case: Quick Start

/**
  @page PAGE_QUICK_START Writing a Test Case: Quick Start

  The basic principle of test case evaluation is that output resulting
  from running a test case is compared to the expected result. This is
  just a <b>diff</b> comparison between the output and an expected-result
  file that the test writer provides. This simplistic method of
  comparison does not by itself provide any way to handle variation in
  the output that may occur when a test is run at different times.
  However, the test language provides commands for postprocessing
  result output before the comparison occurs. This enables you to
  manage certain forms of expected variation.

  Use the following procedure to write a new test case. In the
  examples, <b><em>test_name</em></b> represents the name of the test
  case. It is assumed here that you'll be using a development source
  tree, so that when you create a new test case, you can commit the
  files associated with it to the source repository for others to use.

  <ol>
    <li>
      Change location to the test directory
      <b>mysql-<em>version</em>/mysql-test</b>:

      @verbatim
  shell> cd mysql-version/mysql-test
      @endverbatim

      <b>mysql-<em>version</em></b> represents the root directory of
      your source tree.
    </li>

    <li>
      Create the test case in a file
      <b>t/<em>test_name</em>.test</b>. You can do this with
      any text editor. For details of the language used for writing
      @ref PAGE_MYSQLTEST "mysqltest" test cases, see @ref
      PAGE_MYSQLTEST_LANGUAGE_REFERENCE.
    </li>

    <li>
      Create an empty result file:

      @verbatim
  shell> touch r/test_name.result
      @endverbatim
    </li>

    <li>
      Run the test:

      @verbatim
  shell> ./mysql-test-run.pl test_name
      @endverbatim
    </li>

    <li>
      Assuming that the test case produces output, it should fail
      because the output does not match the result file (which is
      empty at this point). The failure results in creation of a
      reject file named
      <b>mysql-test/var/log/<em>test_name</em>.reject</b>.
      Examine this file. If the reject file appears to contain the
      output that you expect the test case to produce, copy its
      content to the result file:

      @verbatim
  shell> cp mysql-test/var/log/test_name.reject r/test_name.result
      @endverbatim

      Another way to create the result file is by invoking @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the
      <b>`--record`</b> option to record the test output in the
      result file:

      @verbatim
  shell> ./mysql-test-run.pl --record test_name
      @endverbatim
    </li>

    <li>
      Run the test again. This time it should succeed:

      @verbatim
  shell> ./mysql-test-run.pl test_name
      @endverbatim

      You can also run the newly created test case as part of
      the entire suite:

      @verbatim
  shell> ./mysql-test-run.pl
      @endverbatim
    </li>
  </ol>

  It is also possible to invoke the @ref PAGE_MYSQLTEST "mysqltest"
  program directly. If the test case file refers to environment
  variables, you will need to define those variables in your
  environment first. For more information about the @ref
  PAGE_MYSQLTEST "mysqltest" program, see @ref PAGE_MYSQLTEST.
*/

###############################################################################



## Test Case Coding Guidelines

/**
  @page PAGE_TESTCASE_CODING_GUIDELINES Test Case Coding Guidelines

  <h3>Table of Contents</h3>
  - @subpage PAGE_NAMING_ORGANIZATION_GUIDELINES
  - @subpage PAGE_CONTENT_FORMATTING_GUIDELINES
  - @subpage PAGE_NAMING_CONVENTIONS
*/

###############################################################################



## File Naming and Organization Guidelines

/**
  @page PAGE_NAMING_ORGANIZATION_GUIDELINES File Naming and Organization Guidelines

  Test case file names may use alphanumeric characters (<b>A-Z</b>,
  <b>a-z</b>, <b>0-9</b>), underscore (<b>'_'</b>) or dash
  (<b>'-'</b>), but should not start with underscore or dash. No other
  special characters are allowed, if used, then an error is thrown and
  the test run is aborted.

  Test names have traditionally used lowercase only, and we recommend
  continuing this for tests added to the common repository, though
  uppercase letters are also supported.

  Test cases are located in the <b>mysql-test/t</b> directory. Test
  case file names consist of the test name with a <b>.test</b>
  suffix. For example, a test named <b>foo</b> should be written in
  the file <b>mysql-test/t/foo.test</b>.

  In addition to this directory, tests are organized in test suites,
  located in subdirectories under the <b>suite</b> directory. For
  example, a test named <b>bar</b> under the replication suite
  <b>rpl</b> may be stored in the file <b>mysql-
  test/suite/rpl/t/bar.test</b>.

  In practice, the file would likely be called <b>rpl_bar.test</b>
  as tests in a suite usually also have the suite name as a prefix.
  This is just a convention from the time when suites were not
  supported, and not a requirement for test naming.

  One test case file can be a collection of individual tests that
  belong together. If one of the tests fails, the entire test case
  fails. Although it may be tempting to write each small test into a
  single file, that will be too inefficient and makes test runs
  unbearably slow. So make the test case files not too big, not too
  small.

  Each test case (that is, each test file) must be self contained and
  independent of other test cases. Do not create or populate a table
  in one test case and depend on the table in a later test case. If
  you have some common initialization that needs to be done for
  multiple test cases, create an include file. That is, create a file
  containing the initialization code in the <b>mysql-test/include</b>
  directory, and then put a <b>source</b> command in each test case
  that requires the code. For example, if several test cases need to
  have a given table created and filled with data, put the statements
  to do that in a file named <b>mysql-test/include/create_my_table.inc</b>.
  The <b>.inc</b> is a convention, not a requirement. Then put the
  following command in each test case file that needs the
  initialization code:

  @verbatim
  --source include/create_my_table.inc
  @endverbatim

  The file name in the <b>source</b> command is relative to the
  <b>mysql-test</b> directory. Remember to drop the table at the
  end of each test that creates it.
*/

###############################################################################



## Test Case Content-Formatting Guidelines

/**
  @page PAGE_CONTENT_FORMATTING_GUIDELINES Test Case Content-Formatting Guidelines

  When you write a test case, please keep in mind the following
  general guidelines.

  There are C/C++ coding guidelines in the MySQL Internals manual;
  please apply them when it makes sense to do so: @ref
  PAGE_CODING_GUIDELINES.

  Other guidelines may be found in this page, which discusses general
  principles of test-case writing: MySQL Internals:
  [How to Create Good Test Cases]
  (https://dev.mysql.com/doc/internals/en/good-tests.html)

  The following guidelines are particularly applicable to writing test
  cases:

  <ul>
    <li>
      To write a test case file, use any text editor that uses
      linefeed (newline) as the end-of-line character.
    </li>

    <li>
      Avoid lines longer than 80 characters unless there is no
      other choice.
    </li>

    <li>
      A comment in a test case is started with the "#" character.

      @ref PAGE_MYSQL_TEST_INPUT_CONVENTIONS discusses the details
      of input syntax for mysqltest test cases.
    </li>

    <li>
      Use spaces, not tabs.
    </li>

    <li>
      Write SQL statements using the same style as our manual:

      <ul>
        <li>
          Use uppercase for SQL keywords.
        </li>

        <li>
          Use lowercase for identifiers (names of objects such as
          databases, tables, columns, and so forth).
        </li>
      </ul>

      Ignore this guideline if your intent is to test lettercase
      processing for SQL statements, of course.

      Use lowercase for @ref PAGE_MYSQLTEST "mysqltest" commands
      (<b>echo</b>, <b>sleep</b>, <b>let,</b> and so forth).

      You may notice that many existing test cases currently do not
      follow the lettercase guideline and contain SQL statements
      written entirely in lowercase. Nevertheless, <b><em>please use
      the guideline for new tests</em></b>. Lettercase for older tests
      can be left as is, unless perhaps you need to revise them
      significantly.
    </li>

    <li>
      Break a long SQL statement into multiple lines to make it more
      readable. This means that you will need to write it using a
      <b>";"</b> delimiter at the end of the statement rather than
      using <b>"--"</b> at the beginning because the latter style
      works only for single-line statements.
    </li>

    <li>
      Include comments. They save the time of others. In particular:

      <ul>
        <li>
          Please include a header in test files that indicates the
          purpose of the test and references the corresponding worklog
          task, if any.
        </li>

        <li>
          Comments for a test that is related to a bug report should
          include the bug number and title.
        </li>
      </ul>

      Worklog and bug numbers are useful because they enable people
      who are interested in additional background related to the test
      case to know which worklog entries or bug reports to read.
    </li>
  </ul>

  Example SQL statement, formatted onto multiple lines for readability:

  @verbatim
  SELECT f1 AS "my_column", f10 ....
  FROM mysqltest1.t5
  WHERE (f2 BETWEEN 17 AND 25 OR f2 = 61)
    AND f3 IN (SELECT ....
               FROM mysqltest1.t4
               WHERE .....)
  ORDER BY ... ;
  @endverbatim

  Example test file header:

  @verbatim
  ########### suite/funcs_1/t/a_processlist_val_no_prot.test #############
  #                                                                      #
  # Testing of values within INFORMATION_SCHEMA.PROCESSLIST              #
  #                                                                      #
  # The prepared statement variant of this test is                       #
  # suite/funcs_1/t/b_processlist_val_ps.test.                           #
  #                                                                      #
  # There is important documentation within                              #
  #       suite/funcs_1/datadict/processlist_val.inc                     #
  #                                                                      #
  # Note(mleich):                                                        #
  #    The name "a_process..." with the unusual prefix "a_" is           #
  #    caused by the fact that this test should run as second test, that #
  #    means direct after server startup and a_processlist_priv_no_prot. #
  #    Otherwise the connection IDs within the processlist would differ. #
  #                                                                      #
  # Creation:                                                            #
  # 2007-08-09 mleich Implement this test as part of                     #
  #                   WL#3982 Test information_schema.processlist        #
  #                                                                      #
  ########################################################################
  @endverbatim

  Example test reference to bug report:

  @verbatim
  # Bug #3671 Stored procedure crash if function has "set @variable=param"
  @endverbatim
*/

###############################################################################



## Naming Conventions for Database Objects

/**
  @page PAGE_NAMING_CONVENTIONS Naming Conventions for Database Objects

  It is possible to run test cases against a production server. This
  is very unlikely to happen by accident, as @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will start its own server
  unless you use the <b>`--extern`</b> Even so, try to write test
  cases in a way that reduces the risk that running tests will alter
  or destroy important tables, views, or other objects. (<b>DROP
  DATABASE</b> statements are particularly dangerous if written using
  names that could exist on a customer's machine.) To avoid such
  problems, we recommend the following naming conventions:

  <ul>
    <li>
      User names: User names should begin with “mysql” (for example,
      <b>mysqluser1</b>, <b>mysqluser2</b>)
    </li>

    <li>
      Database names: Unless you have a special reason not to, use
      the default database named <b>test</b> that is already created
      for you. For tests that need to operate outside the <b>test</b>
      database, database names should contain “test” or begin with
      “mysql” (for example, <b>mysqltest1</b>, <b>mysqltest2</b>)
    </li>

    <li>
      Table names: <b>t1</b>, <b>t2</b>, <b>t3</b>, ...
    </li>

    <li>
      View names: <b>v1</b>, <b>v2</b>, <b>v3</b>, ...
    </li>
  </ul>

  For examples of how to name objects, examine the existing test
  cases. Of course, you can name columns and other objects inside
  tables as you wish.
*/

###############################################################################



## Sample Test Case

/**
  @page PAGE_SAMPLE_TESTCASES Sample Test Case

  Here is a small sample test case:

  @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1;
  SET @@sql_mode='NO_ENGINE_SUBSTITUTION';
  --enable_warnings

  SET SQL_WARNINGS=1;

  CREATE TABLE t1 (a INT);
  INSERT INTO t1 VALUES (1);
  INSERT INTO t1 VALUES ("hej");

  DROP TABLE t1;
  @endverbatim

  The first few lines try to clean up from possible earlier runs of
  the test case by dropping the <b>t1</b> table. The test case uses
  <b>disable_warnings</b> to prevent warnings from being written to
  the output because it is not of any interest at this point during
  the test to know whether the table <b>t1</b> was there. The value of
  <b>sql_mode</b> is changed to <b>NO_ENGINE_SUBSTITUTION</b> to avoid
  the error thrown by the second insert statement. After this, the
  test case uses <b>enable_warnings</b> so that subsequent warnings
  will be written to the output. The test case also enables verbose
  warnings in MySQL using the <b>SET SQL_WARNINGS=1;</b> statement.

  Next, the test case creates the table <b>t1</b> and tries some
  operations. Creating the table and inserting the first row are
  operations that should not generate any warnings. The second insert
  should generate a warning because it inserts a nonnumeric string
  into a numeric column. At the end, it drops the table <b>t1</b>. The
  output that results from running the test looks like this:

  @verbatim
  DROP TABLE IF EXISTS t1;
  SET @@sql_mode='NO_ENGINE_SUBSTITUTION';
  SET SQL_WARNINGS=1;
  CREATE TABLE t1 (a INT);
  INSERT INTO t1 VALUES (1);
  INSERT INTO t1 VALUES ("hej");
  Warnings:
  Warning 1366  Incorrect integer value: 'hej' for column 'a' at row 1
  DROP TABLE t1;
  @endverbatim

  Note that the result includes not only the output from SQL
  statements, but the statements themselves. Statement logging can be
  disabled with the <b>disable_query_log</b> test language command.
  There are several options for controlling the amount of output from
  running the tests.

  If there was a test failure, it will be reported to the screen. You
  can see the actual output from the last unsuccessful run of the test
  case in the reject file
  <b>mysql-test/var/log/<em>test_name</em>.reject</b>.
*/

###############################################################################



## Cleaning Up from a Previous Test Run

/**
  @page PAGE_CLEANUP_PREVIOUS_TEST_RUN Cleaning Up from a Previous Test Run

  For efficiency, the @ref PAGE_MYSQLTEST "mysqltest" test engine does
  not start with a clean new database for running each test case, so a
  test case generally starts with a “cleaning up section”. Assume that
  a test case will use two tables named <b>t1</b> and <b>t2</b>.
  The test case should begin by making sure that any old tables with
  those names do not exist:

  @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1,t2;
  --enable_warnings
  @endverbatim

  The <b>disable_warnings</b> command instructs the test engine not
  to log any warnings until an <b>enable_warnings</b> command occurs
  or the test case is ended. (MySQL generates a warning if the table
  <b>t1</b> or <b>t2</b> does not exist.) Surrounding this part of
  the test case with commands to disable and enable warnings makes its
  output the same regardless of whether the tables exist before the
  test is started. After ensuring that the tables do not exist, we are
  free to put in any SQL statements that create and use the tables
  <b>t1</b> and <b>t2</b>. The test case should also clean up at
  the end of the test by dropping any tables that it creates.

  Let's put in some SQL code into this test case:

  @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1,t2;
  --enable_warnings
  CREATE TABLE t1 (
    Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL,
    Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL
  );
  CREATE TABLE t2 (Period SMALLINT);

  INSERT INTO t1 VALUES (9410,9412);
  INSERT INTO t2 VALUES (9410),(9411),(9412),(9413);

  SELECT PERIOD FROM t1;
  SELECT * FROM t1;
  SELECT t1.* FROM t1;
  SELECT * FROM t1 INNER JOIN t2 USING (Period);

  DROP TABLE t1, t2;
  @endverbatim

  If a test case creates other objects such as stored programs or
  user accounts, it should take care to also clean those up at the
  beginning and end of the test. Temporary files should also be
  removed, either at the end or just after they have been used.
*/

###############################################################################



## Generating a Test Case Result File

/**
  @page PAGE_GENERATE_TESTCASE_RESULT_FILE Generating a Test Case Result File

  The test code we just wrote contains no checks of the result. The
  test will report a failure for one of two reasons:

  - An individual SQL statement fails with an error

  - The overall test case result does not match what was expected

  Note that these are the reasons why @ref PAGE_MYSQLTEST "mysqltest"
  would fail; if the test is run from @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" the test may fail for additional reasons.

  In the first case, @ref PAGE_MYSQLTEST "mysqltest" aborts with an
  error. The second case requires that we have a record of the
  expected result so that it can be compared with the actual result.
  To generate a file that contains the test result, run the test with
  the <b>`--record`</b> option, like this:

  @verbatim
  shell> cd mysql-test
  shell> ./mysql-test-run.pl --record foo
  @endverbatim

  Running the test as shown creates a result file named
  <b>mysql-test/r/foo.result</b> that has this content:

  @verbatim
  DROP TABLE IF EXISTS t1,t2;
  CREATE TABLE t1 (
  Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL,
  Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL
  );
  CREATE TABLE t2 (Period SMALLINT);
  INSERT INTO t1 VALUES (9410,9412);
  INSERT INTO t2 VALUES (9410),(9411),(9412),(9413);
  SELECT period FROM t1;
  period
  9410
  SELECT * FROM t1;
  Period  Varor_period
  9410    9412
  SELECT t1.* FROM t1;
  Period  Varor_period
  9410    9412
  SELECT * FROM t1 INNER JOIN t2 USING (Period);
  Period  Varor_period
  9410    9412
  DROP TABLE t1, t2;
  ok
  @endverbatim

  If we look at this result file, it contains the statements in the
  <b>foo.test</b> file together with the output from the
  <b>SELECT</b> statements. The output for each statement includes a
  row of column headings followed by data rows. Rows have columns
  separated by Tab characters.

  At this point, you should inspect the result file and determine
  whether its contents are as expected. If so, let it be part of your
  test case. If the result is not as expected, you have found a
  problem, either with the server or the test. Determine the cause of
  the problem and fix it. For example, the test might produce output
  that varies from run to run. To deal with this, you can postprocess
  the output before the comparison occurs. See @ref
  PAGE_DEALING_OUTPUT.
*/

###############################################################################



## Checking for Expected Errors

/**
  @page PAGE_CHECK_EXPECTED_ERRORS Checking for Expected Errors

  A good test suite checks not only that operations succeed as they
  ought, but also that they fail as they ought. For example, if a
  statement is illegal, the server should reject it with an error
  message. The test suite should verify that the statement fails and
  that it fails with the proper error message.

  The test engine enables you to specify “expected failures.” Let's
  say that after we create <b>t1</b>, we try to create it again
  without dropping it first:

  @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1,t2;
  --enable_warnings
  CREATE TABLE t1 (
    Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL,
    Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL
  );
  CREATE TABLE t2 (Period SMALLINT);

  INSERT INTO t1 VALUES (9410,9412);
  INSERT INTO t2 VALUES (9410),(9411),(9412),(9413);

  SELECT period FROM t1;
  SELECT * FROM t1;
  SELECT t1.* FROM t1;
  SELECT * FROM t1 INNER JOIN t2 USING (Period);

  CREATE TABLE t1 (something SMALLINT(4));
  @endverbatim

  The result is failure and an error:

  @verbatim
  At line 21: query 'CREATE TABLE t1 (something SMALLINT(4))'
  failed: 1050: Table 't1' already exists
  @endverbatim

  To handle this error and indicate that indeed we do expect it to
  occur, we can put an <b>error</b> command before the second
  <b>create table</b> statement. Either of the following commands
  test for this particular MySQL error:

  @verbatim
  --error 1050
  --error ER_TABLE_EXISTS_ERROR
  @endverbatim

  <b>1050</b> is the numeric error code and
  <b>ER_TABLE_EXISTS_ERROR</b> is the symbolic name. Symbolic names
  are more stable than error numbers because the numbers sometimes
  change, particularly for those created during recent development.
  For such errors, use of numbers rather than the names in a test case
  will require test to be revised should the numbers change.

  After we make a change to add an <b>error</b> command before the
  <b>CREATE TABLE</b> statement and run the test again, the end of
  the result will look like this:

  @verbatim
  CREATE TABLE t1 (something SMALLINT(4));
  ERROR 42S01: Table 't1' already exists
  @endverbatim

  In this case, the result shows the statement that causes the error,
  together with the resulting error message. The fact that @ref
  PAGE_MYSQLTEST "mysqltest" does not terminate and that the error
  message becomes part of the result indicates that the error was
  expected.

  You can also test for errors by specifying an SQLSTATE value. For
  MySQL error number <b>1050</b>, the corresponding SQLSTATE value
  is <b>42S01</b>. To specify an SQLSTATE value in an <b>error</b>
  command, use an <b>S</b> prefix:

  @verbatim
  --error S42S01
  @endverbatim

  A disadvantage of SQLSTATE values is that sometimes they correspond
  to more than one MySQL error code. Using the SQLSTATE value in this
  case might not be specific enough (it could let through an error
  that you do not actually expect).

  If you want to test for multiple errors, the <b>error</b> command
  allows multiple arguments, separated by commas. For example:

  @verbatim
  --error ER_NO_SUCH_TABLE,ER_KEY_NOT_FOUND
  @endverbatim

  For a list of MySQL error codes, symbolic names, and SQLSTATE values,
  see https://dev.mysql.com/doc/refman/8.0/en/error-messages-server.html.
  You can also examine the <b>mysqld_error.h</b> and
  <b>sql_state.h</b> files in the <b>include</b> directory of a MySQL
  source distribution.

  It is also possible to use symbolic error names to refer to client
  errors:

  @verbatim
  --error CR_SERVER_GONE_ERROR
  @endverbatim

  For a list of MySQL client error codes, see
  https://dev.mysql.com/doc/refman/8.0/en/error-messages-client.html.
  You can also examine the <b>errmsg.h</b> file in the <b>include</b>
  directory of a MySQL source distribution.

  The built-in variable <b>$mysql_errno</b> contains the numeric
  error returned by the most recent SQL statement sent to the server,
  or 0 if the statement executed successfully. This may be useful
  after statements that may or may not fail, or fail in more than one
  way (more than one argument to the <b>error</b> command), in case
  you need to perform different actions. Note that this applies to SQL
  statements, not to other commands.

  There is also a variable <b>$mysql_errname</b> which contains the
  symbolic name of the last error. In some cases the symbolic name is
  not available; in those cases the variable will contain the string
  <b>"<Unknown>"</b>. For new test development we recommend testing
  against the name rather than the number, since the number may change
  in future versions. If the last statement succeeded
  (<b>$mysql_errno</b> is 0), this variable is an empty string.
*/

###############################################################################



## Controlling the Information Produced by a Test Case

/**
  @page PAGE_CONTROL_INFORMATION_PRODUCED_TESTCASE Controlling the Information Produced by a Test Case

  By default, the @ref PAGE_MYSQLTEST "mysqltest" test engine produces
  output only from <b>select</b>, <b>show</b>, and other SQL
  statements that you expect to produce output (that is, statements
  that create a result set). It also produces output from certain
  commands such as <b>echo</b> and <b>exec</b>. @ref
  PAGE_MYSQLTEST "mysqltest" can be instructed to be more or less
  verbose.

  Suppose that we want to include in the result the number of rows
  affected by or returned by SQL statements. To do this, add the
  following line to the test case file preceding the first
  table-creation statement:

  @verbatim
  --enable_info
  @endverbatim

  After rerunning the test by invoking @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" with the <b>`--record`</b> option to record
  the new result, the result file will contain more information:

  @verbatim
  DROP TABLE IF EXISTS t1,t2;
  CREATE TABLE t1 (
  Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL,
  Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL
  );
  affected rows: 0
  CREATE TABLE t2 (Period SMALLINT);
  affected rows: 0
  INSERT INTO t1 VALUES (9410,9412);
  affected rows: 1
  INSERT INTO t2 VALUES (9410),(9411),(9412),(9413);
  affected rows: 4
  info: Records: 4  Duplicates: 0  Warnings: 0
  SELECT period FROM t1;
  period
  9410
  affected rows: 1
  SELECT * FROM t1;
  Period  Varor_period
  9410    9412
  affected rows: 1
  SELECT t1.* FROM t1;
  Period  Varor_period
  9410    9412
  affected rows: 1
  SELECT * FROM t1 INNER JOIN t2 USING (Period);
  Period  Varor_period
  9410    9412
  affected rows: 1
  DROP TABLE t1, t2;
  affected rows: 0
  ok
  @endverbatim

  To turn off the affected-rows reporting, add this command to the
  test case file:

  @verbatim
  --disable_info
  @endverbatim

  In general, options can be enabled and disabled for different parts
  of the test file. Suppose that we are interested in the internals of
  the database as well. We could enable the display of query metadata
  using <b>enable_metadata</b>. With this option enabled, the test
  output is a bit verbose. However, as mentioned earlier, the option
  can be enabled and disabled selectively so that it is enabled only
  for those parts of the test case where it interests you to know
  more.

  If you perform an operation for which you have no interest in seeing
  the statements logged to the result, you can disable statement
  logging. For example, you might be initializing a table where you
  don't really expect a failure, and you are not interested in seeing
  the initialization statements in the test result. You can use the
  <b>disable_query_log</b> command to temporarily disable recording
  of input SQL statements, and enable recording again with
  <b>enable_query_log</b>. You can disable the recording of the
  output from executing commands using <b>disable_result_log</b> and
  enable recording again with <b>enable_result_log</b>.
*/

###############################################################################



## Dealing with Output That Varies Per Test Run

/**
  @page PAGE_DEALING_OUTPUT Dealing with Output That Varies Per Test Run

  It is best to write each test case so that the result it produces
  does not vary for each test run, or according to factors such as the
  time of day, differences in how program binaries are compiled, the
  operating system, and so forth. For example, if the result contains
  the current date and time, the test engine has no way to verify that
  the result is correct.

  However, sometimes a test result is inherently variable according to
  external factors, or perhaps there is a part of a result that you
  simply do not care about. @ref PAGE_MYSQLTEST "mysqltest" provides
  commands that enable you to postprocess test output into a more
  standard format so that output variation across test runs will not
  trigger a result mismatch.

  One such command is <b>replace_column</b>, which specifies that
  you want to replace whatever is in a given column with a string.
  This makes the output for that column the same for each test run.

  To see how this command works, add the following row after the first
  insert in the test case:

  @verbatim
  INSERT INTO t1 VALUES (DATE_FORMAT(NOW(), '%s'),9999);
  @endverbatim

  Then record the test result and run the test again:

  @verbatim
  shell> ./mysql-test-run.pl --record foo
  shell> ./mysql-test-run.pl foo
  @endverbatim

  Most likely, a failure will occur and @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" will display the difference between the expected
  result and what we actually got, like this (the header has been
  simplified):

  @verbatim
  CURRENT_TEST: main.foo
  --- r/foo.result        2009-11-17 16:22:38
  +++ var/log/foo.reject        2009-11-17 16:22:47
  @@ -10,15 +10,15 @@
   SELECT period FROM t1;
   period
   9410
  -0038
  +0047
   SELECT * FROM t1;
   Period Varor_period
   9410   9412
  -0038   9999
  +0047   9999
   SELECT t1.* FROM t1;
   Period Varor_period
   9410   9412
  -0038   9999
  +0047   9999
   SELECT * FROM t1 INNER JOIN t2 USING (Period);
   Period Varor_period
   9410   9412

  mysqltest: Result content mismatch
  @endverbatim

  The actual numbers will likely be different for your case, and the
  format of the diff may also vary.

  If we are not really interested in the first column, one way to
  eliminate this mismatch is by using the <b>replace_column</b>
  command. The duration of the effect of this command is the next SQL
  statement, so we need one before each <b>select</b> statement:

  @verbatim
  --replace_column 1 SECONDS
  SELECT period FROM t1;

  --replace_column 1 SECONDS
  SELECT * FROM t1;

  --replace_column 1 SECONDS
  SELECT t1.* FROM t1;
  @endverbatim

  In the <b>replace_column</b> commands, <b>SECONDS</b> could be
  any string. Its only purpose is to map variable output onto a
  constant value. If we record the test result again, we will succeed
  each time we run the test after that. The result file will look like
  this:

  @verbatim
  DROP TABLE IF EXISTS t1,t2;
  CREATE TABLE t1 (
  Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL,
  Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL
  );
  affected rows: 0
  CREATE TABLE t2 (Period SMALLINT);
  affected rows: 0
  INSERT INTO t1 VALUES (9410,9412);
  affected rows: 1
  INSERT INTO t1 VALUES (DATE_FORMAT(NOW(), '%s'),9999);
  affected rows: 1
  INSERT INTO t2 VALUES (9410),(9411),(9412),(9413);
  affected rows: 4
  info: Records: 4  Duplicates: 0  Warnings: 0
  SELECT period FROM t1;
  period
  SECONDS
  SECONDS
  affected rows: 2
  SELECT * FROM t1;
  Period  Varor_period
  SECONDS 9412
  SECONDS 9999
  affected rows: 2
  SELECT t1.* FROM t1;
  Period  Varor_period
  SECONDS 9412
  SECONDS 9999
  affected rows: 2
  SELECT * FROM t1 INNER JOIN t2 USING (Period);
  Period  Varor_period
  9410    9412
  affected rows: 1
  DROP TABLE t1, t2;
  affected rows: 0
  ok
  @endverbatim
*/

###############################################################################



## Passing Options from mysql-test-run.pl to mysqld or mysqltest

/**
  @page PAGE_PASSING_OPTIONS Passing Options from mysql-test-run.pl to mysqld or mysqltest

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" supports several
  options that enable you to pass options to other programs. Each of
  these options takes a value consisting of one or more comma-separated
  options:

  <ul>
    <li>
      The <b>`--mysqld`</b> option tells @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" to start the <b>mysqld</b> server with the
      named option added. More than one such extra option may be
      provided. The following command causes <b>`--big-tables`</b>
      and <b>`--key_buffer_size`=16384</b> to be passed to
      <b>mysqld</b>:

      @verbatim
  shell> ./mysql-test-run.pl
           --mysqld=--big-tables
           --mysqld=--key_buffer_size=16384
      @endverbatim

      Note how <b>`--mysqld`</b> needs to be repeated for each
      server option to add. It does not work to add several server
      options with one <b>`--mysqld`</b> even if enclosed in
      quotation marks, as that will be interpreted as a single server
      option (including spaces).
    </li>

    <li>
      The <b>`--combination`</b> option is similar to
      <b>`--mysqld`</b>, but behaves differently. @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" executes multiple
      test runs, using the options for each instance of
      <b>`--combination`</b> in successive runs. The following
      command passes <b>`--big-tables`</b> to <b>mysqld</b> for the
      first test run, <b>`--innodb`</b> for the second run and
      <b>`--innodb-file-per-table`</b> for the last run:

      @verbatim
  shell> ./mysql-test-run.pl
           --combination=--big-tables
           --combination=--innodb
           --combination=--innodb-file-per-table
      @endverbatim

      If <b>`--combination`</b> is given only once, it behaves similar
      to <b>`--mysqld`</b> option.

      For test runs specific to a given test suite, an alternative to
      the use of the <b>`--combination`</b> option is to create a
      <b>combinations</b> file in the suite directory. The file should
      contain a section of options for each test run. For an example,
      see @ref PAGE_BINARY_LOG_FORMAT.

      <b>combinations</b> file can support <b>bootstrap</b> variables
      too. The <b>bootstrap</b> variable has to be passed as a value
      to the <b>`initialize`</b> keyword.

      For a test specific combinations, an alternative to the use of
      the <b>`--combination`</b> option is to create a
      <b><em>test_name</em>.combinations</b> file in the test
      directory.
    </li>

    <li>
      The <b>`--mysqltest`</b> option is used to pass extra options
      to @ref PAGE_MYSQLTEST "mysqltest".

      @verbatim
  shell> ./mysql-test-run.pl --mysqltest=options
      @endverbatim

      For an example, see @ref PAGE_MYSQL_TEST_RUN_PL.
    </li>
  </ul>
*/

###############################################################################



## Specifying Test Case-Specific Server Options

/**
  @page PAGE_TESTCASE_SPECIFIC_SERVER_OPTIONS Specifying Test Case-Specific Server Options

  Within a test case, many system variables can be set by using
  statements such as these:

  @verbatim
  SET sql_warnings= 1;
  SET sql_mode= 'NO_AUTO_VALUE_ON_ZERO';
  @endverbatim

  But sometimes you need to restart the server to use command-line
  options that are specific to a given test case. You can specify
  these options in a file named
  <b>mysql-test/t/<em>test_name-master</em>.opt</b>. When a
  file named <b>t/<em>test_name-master</em>.opt</b> exists,
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for
  extra options that the server needs to be run with when executing
  the <em>test_name</em> test case. If no server has yet been
  started or the current server is running with different options,
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" restarts the server
  with the new options.

  You may refer to environment variables in the option file, using the
  usual <b><em>$VAR_NAME</em></b> syntax. It is also possible to refer
  to optional variables using the syntax <b><em>$?VAR_NAME</em></b>.
  This will be replaced with an empty string if the variable is not
  set.

  As a special case, the option <b>`--skip-core-file`</b> will be
  interpreted by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl",
  which will then block the server from producing any core files or
  crash dumps during this test. This may be useful for tests that
  intentionally crash the server.

  Another special case is <b>`--testcase-timeout`=minutes</b> which
  can be used to set a different, longer timeout for a particular test
  case. The given timeout in minutes will be used for this test if
  it's longer than the default.

  Files in the <b>mysql-test/t</b> directory with names ending in
  <b>-slave.opt</b> are similar, but they are used for slave servers
  in replication tests.

  Sometimes it's also necessary to execute some external commands
  before starting the server, such as creating a directory. If you add
  a file named <b>t/<em>test_name-master</em>.sh</b>, it will
  be executed by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
  before it starts the server; a similar file may be created for the
  slave in replication tests.

  Because the <b>.sh</b> file is executed through <b>/bin/sh</b>, it
  cannot be used on Windows, and any tests using such a file will
  automatically be skipped if you run on Windows. For this reason,
  this mechanism may be replaced with a more portable one in some
  future release of MySQL.
*/

###############################################################################



## Specifying Test Case-Specific Bootstrap Options

/**
  @page PAGE_TESTCASE_SPECIFIC_BOOTSTRAP_OPTIONS Specifying Test Case-Specific Bootstrap Options

  If a test has to run with a particular value of a bootstrap variable
  such as <b>`--innodb-page-size`</b> or
  <b>`--innodb-data-file-path`</b>, the option can be passed on the
  command line while running @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl". For example, consider a test that can only run
  with <b>`--innodb-page-size`=8k</b>. The test can be run like this:

  @verbatim
  shell> ./mysql-test-run.pl test_name_8k --initialize=--innodb-page-size=8k
  @endverbatim

  This will initialize the data directory with 8k page size, and start the
  server with that value.

  It is also possible to pass bootstrap options in the
  <b>master.opt</b> file of the test, so that the test can run with
  the specified value of the bootstrap options without using any
  command line arguments. The usage is:

  @verbatim
  --initialize --innodb-page-size=8k
  @endverbatim

  or

  @verbatim
  --initialize=--innodb-page-size=8k
  @endverbatim

  Specifying bootstrap variables in the opt file is the preferred
  method.

  Each bootstrap variable must be specified as the value of a
  <b>`--initialize`</b> option in the opt file to ensure @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" recognizes that the
  variable must be used during server initialization. If there are
  bootstrap options in the file, the current data directory is
  deleted, and initialized again with the options set in the file. The
  server is also started with the bootstrap options passed along with
  the other options in the opt file.

  Similarly, bootstrap options passed in the <b>slave.opt</b> will be
  used to reinitialize the slave server in replication scenarios.

  Bootstrap options set in the opt file are passed to the server
  first, followed by the server options. So, server start up options
  have precedence over bootstrap options, regardless of the order in
  which they are set in the opt file.

  Note that since the bootstrap options are passed in the opt files,
  they have precedence over the command line bootstrap options. So,
  if a test has a master.opt file containing <b>`--innodb-page-size`=8k</b>,
  and while it is being run, <b>`--innodb-page-size`=4k</b> is passed on
  the command line, the test will run with <b>8k</b> page size.

  Some tests require the server to be restarted multiple times with
  different server options, as well different bootstrap options. In
  such cases, the existing data directory is deleted inside the test
  and a new data directory is initialized with the bootstrap options.
  When the server is started with the bootstrap options after this,
  the SQL queries will run with the specified options.

  @verbatim
  --exec $MYSQLD --no-defaults --initialize-insecure
  --lc_messages_dir=$MYSQL_SHAREDIR --innodb_page_size=8K
  --basedir=$MYSQLD_BASEDIR --datadir=$MYSQL_TMP_DIR/datadir1/data
  --init-file=$BOOTSTRAP_SQL --secure-file-priv=""
  @endverbatim
*/

###############################################################################



## Specifying Test Case-Specific Client Options

/**
  @page PAGE_TESTCASE_SPECIFIC_CLIENT_OPTIONS Specifying Test Case-Specific Client Options

  If a test case needs a specific client option, you can specify such
  options in a file named <b><em>testname</em>-client.opt</b>. When a
  test case has its own <b>-client.opt</b> file, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for any
  additional options that the @ref PAGE_MYSQLTEST "mysqltest" should
  run with while executing the test case.

  For example, consider a test case <b>example.test</b> needs
  <tt>latin1</tt> as default character set for client and connection,
  then that option can be specified in
  <b><em>example</em>-client.opt</b> file.

  @verbatim
  --default-character-set=latin1
  @endverbatim

  You may refer to environment variables in the option file, using the
  usual <b><em>$VAR_NAME</em></b> syntax. It is also possible to refer
  to optional variables using the syntax <b><em>$?VAR_NAME</em></b>.
  This will be replaced with an empty string if the variable is not
  set.
*/

###############################################################################



## Using Include Files to Simplify Test Cases

/**
  @page PAGE_INCLUDE_FILES Using Include Files to Simplify Test Cases

  The <b>include</b> directory contains many files intended for
  inclusion into test case files.

  These include files serve many purposes, but in general, they
  encapsulate operations of varying complexity into single files so
  that you can perform each operation in a single step. Include files
  are available for operations such as these:

  <ul>
    <li>
      Ensure that a given feature is available. The file checks to
      make sure that the feature is available and exits if not.
    </li>

    <ul>
      <li>
        Storage engine tests: These files have names of the form
        <b>have_<em>engine_name.inc</em></b>, such as
        <b>have_archive.inc</b> or <b>have_blackhole.inc</b>. The
        <b>MyISAM</b>, <b>MERGE</b>, and <b>MEMORY</b> storage
        engines are always supported and need not be checked.
      </li>

      <li>
        Windows specific tests: Include <b>have_windows.inc</b> file
        if a test runs only on windows machines.
      </li>

      <li>
        Debugging capabilities: Include the <b>have_debug.inc</b>
        file if a test requires that the server was built for
        debugging (that is, that the MySQL distribution was configured
        with the <b>`--with-debug`</b> option).
      </li>
    </ul>

    <li>
      Wait for a condition to become true. Set the
      <b>$wait_condition</b> variable to an SQL statement that
      selects a value and then include the <b>wait_condition.inc</b>
      file. The include file executes the statement in a loop with a
      0.1 second sleep between executions until the select value is
      nonzero. For example:

      @verbatim
  --let $wait_condition= SELECT c = 3 FROM t
  --source include/wait_condition.inc
      @endverbatim
    </li>

    <li>
      Control the binary log format. See @ref PAGE_BINARY_LOG_FORMAT.
    </li>

    <li>
      Control replication slave servers. See @ref
      PAGE_WRITING_REPLICATION_TESTS.
    </li>
  </ul>

  You can think of an include file as a rudimentary form of subroutine
  that is “called” at the point of inclusion. You can “pass
  parameters” by setting variables before including the file and
  referring to them within the file. You can “return” values by
  setting variables within the file and referring them following
  inclusion of the file.
*/

###############################################################################



## Controlling the Binary Log Format Used for Tests

/**
  @page PAGE_BINARY_LOG_FORMAT Controlling the Binary Log Format Used for Tests

  <h3>Table of Contents</h3>
  - @subpage PAGE_CONTROLLING_BINARY_LOG_FORMAT
  - @subpage PAGE_SPECIFY_BINARY_LOG_FORMAT

  The server can do binary logging using statement-based logging
  (SBL), which logs events as statements that produce data changes, or
  row-based logging (RBL), which logs events as changes to individual
  rows. It also supports mixed logging, which switches between SBL and
  RBL automatically as necessary.

  The server's global <b>binlog_format</b> system variable indicates
  which log format is in effect. It has possible values of
  <b>STATEMENT</b>, <b>ROW</b>, and <b>MIXED</b> (not case
  sensitive). This system variable can be set at server startup by
  specifying <b>`--binlog_format`=<em>value</em></b> on the
  command line or in an option file. A user who has the <b>SUPER</b>
  privilege can change the log format at runtime. For example:

  @verbatim
  SET GLOBAL binlog_format = STATEMENT;
  @endverbatim

  Some tests require a particular binary log format. You can exercise
  control over the binary log format in two ways:

  - To control the log format that the server uses for an entire
    test run, you can pass options to @ref PAGE_MYSQL_TEST_RUN_PL
    "mysql-test-run.pl" that tell it which format <b>mysqld</b>
    should use.

  - To verify that a particular log format is in effect for a specific
    test case, you can use an appropriate include file that checks
    the current format and exits if the format is other than what
    is required.

  The following sections describe how to use these techniques.
*/

###############################################################################



## Controlling the Binary Log Format Used for an Entire Test Run

/**
  @page PAGE_CONTROLLING_BINARY_LOG_FORMAT Controlling the Binary Log Format Used for an Entire Test Run

  To specify the binary log format for a test run, you can use the
  <b>`--mysqld`</b> or <b>`--combination`</b> option to tell @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to pass a logging option
  to <b>mysqld</b>. For example, the following command runs the tests
  from the <b>rpl</b> suite that have names that begin with
  <b>rpl_row</b>. The tests are run once with the binary log format
  set to <b>STATEMENT</b>:

  @verbatim
  shell> ./mysql-test-run.pl --suite=rpl --do-test=rpl_row
           --mysqld=--binlog_format=statement
  @endverbatim

  To run tests under multiple log formats, use two or more instances
  of the <b>`--combination`</b> option. The following command runs
  the same tests as the preceding command, but runs them once with the
  binary log format set to <b>ROW</b> and a second time with the
  format set to <b>MIXED</b>:

  @verbatim
  shell> ./mysql-test-run.pl --suite=rpl --do-test=rpl_row
           --combination=--binlog_format=row
           --combination=--binlog_format=mixed
  @endverbatim

  The <b>`--combination`</b> option must be given at least twice or
  it has no effect.

  As an alternative to using the <b>`--combination`</b> option, you
  can create a file named <b>combinations</b> in the test suite
  directory and list the options that you would specify using
  <b>`--combination`</b>, one line per option. For the preceding
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" command, the suite
  name is <b>rpl</b>, so you would create a file named
  <b>suite/rpl/combinations</b> with these contents:

  @verbatim
  [row]
  --binlog_format=row

  [mixed]
  --binlog_format=mixed
  @endverbatim

  Then invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" like this:

  @verbatim
  shell> ./mysql-test-run.pl --suite=rpl --do-test=row
  @endverbatim

  Similar to <b>combinations</b> file in the test suite directory, you
  can also create a file named <b><em>test_name</em>.combinations</b>
  in the test directory and list the options that you would specify
  using <b>`--combination`</b>, one line per option.

  If there exist both suite level <b>combinations</b> file and a
  <b><em>test_name</em>.combinations</b> file, then the
  <b><em>test_name</em>.combinations</b> file will take the priority
  i.e combinations specified in suite level <b>combinations</b> file
  will be ignored.

  The format of the <b>combinations</b> file is similar to that of
  <b>my.cnf</b> files (section names followed by options for each
  section). @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" displays
  the section name following the test name when it reports the test
  result.

  Any <b>`--combination`</b> options specified on the command line
  override those found in a <b>combinations</b> file.

  The <b>`--combination`</b> option and the <b>combinations</b>
  file have different scope. The <b>`--combination`</b> option
  applies globally to all tests run by a given invocation of @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". The
  <b>combinations</b> file is placed in a test suite directory and
  applies only to tests in that suite. The
  <b><em>test_name</em>.combinations</b> file is placed in a test
  directory and applies only to <b>test_name</b>.
*/

###############################################################################



## Specifying the Required Binary Log Format for Individual Test Cases

/**
  @page PAGE_SPECIFY_BINARY_LOG_FORMAT Specifying the Required Binary Log Format for Individual Test Cases

  To specify within a test case that a particular binary log format is
  required, include one of the following lines to indicate the format:

  @verbatim
  --source include/have_binlog_format_row.inc
  --source include/have_binlog_format_statement.inc
  --source include/have_binlog_format_mixed.inc
  @endverbatim

  The following files can be used for tests that support two binary
  log formats:

  @verbatim
  --source include/have_binlog_format_mixed_or_row.inc
  --source include/have_binlog_format_mixed_or_statement.inc
  --source include/have_binlog_format_row_or_statement.inc
  @endverbatim

  Before @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs the test
  case, it checks whether the value that it is using for the
  <b>binlog_format</b> system variable matches what the test
  requires, based on whether the test refers to one of the preceding
  include files. If <b>binlog_format</b> does not have an
  appropriate value, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
  skips the test.

  If a test supports all binary log formats, none of the
  <b>have_binlog_format_*.inc</b> include files should be used in
  the test file. A test that includes no such file is assumed to
  support all formats.
*/

###############################################################################



## Writing Replication Tests

/**
  @page PAGE_WRITING_REPLICATION_TESTS Writing Replication Tests

  If you are writing a replication test case, the test case file
  should begin with this command:

  @verbatim
  --source include/master-slave.inc
  @endverbatim

  To switch between the master and slave, use these commands:

  @verbatim
  connection master;
  connection slave;
  @endverbatim

  If you need to do something on an alternative connection, you can
  use <b>connection master1;</b> for the master and <b>connection
  slave1;</b> for the slave.

  To run the master with additional options for your test case, put
  them in command-line format in
  <b>t/<em>test_name-master</em>.opt</b>. When a file named
  <b>t/<em>test_name-master</em>.opt</b> exists, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for extra
  options that the server needs to be run with when executing the
  <em>test_name</em> test case. If no server has yet been started
  or the current server is running with different options, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" restarts the server with
  the new options.

  For the slave, similar principles apply, but you should list
  additional options in <b>t/<em>test_name-slave</em>.opt</b>.

  Several include files are available for use in tests that enable
  better control over the behavior of slave server I/O and SQL
  threads. The files are located in the <b>include</b> directory and
  have names of the form <b>wait_for_slave_*.inc</b>. By using these
  files, you can help make replication tests more stable because it
  will be more likely that test failures are due to replication
  failures, not due to problems with the tests themselves.

  The slave-control include files address the issue that it is not
  always sufficient to use a <b>START SLAVE</b> or <b>STOP
  SLAVE</b> statement by itself: When the statement returns, the
  slave may not have reached the desired operational state. For
  example, with <b>START SLAVE</b>, the following considerations
  apply:

  <ul>
    <li>
      It is not necessary to wait for the SQL thread after <b>START
      SLAVE</b> or <b>START SLAVE SQL_THREAD</b> because the thread
      will have started by the time statement returns.
    </li>

    <li>
      By contrast, it is necessary to wait for the I/O thread after
      <b>START SLAVE</b> or <b>START SLAVE IO_THREAD</b> because
      although the thread will have started when the statement returns,
      it may not yet have established the connection to the master.
    </li>
  </ul>

  To verify that a slave has reached the desired state, combine the
  use of <b>START SLAVE</b> or <b>STOP SLAVE</b> with an
  appropriate “wait” include file. The file contains code that waits
  until the state has been reached or a timeout occurs. For example,
  to verify that both slave threads have started, do this:

  @verbatim
  START SLAVE
  --source include/wait_for_slave_to_start.inc
  @endverbatim

  Similarly, to stop both slave threads, do this:

  @verbatim
  STOP SLAVE;
  --source include/wait_for_slave_to_stop.inc
  @endverbatim

  The following list describes the include files that are available
  for slave control:

  <ul>
    <li>
      <tt>wait_for_slave_to_start.inc</tt>

      Waits for both slave threads (I/O and SQL) to start. Should be
      preceded by a <b>START SLAVE</b> statement.
    </li>

    <li>
      <tt>wait_for_slave_to_stop.inc</tt>

      Waits for both slave threads (I/O and SQL) to stop. Should be
      preceded by a <b>STOP SLAVE</b> statement.
    </li>

    <li>
      <tt>wait_for_slave_sql_to_stop.inc</tt>

      Waits for the slave SQL thread to stop. Should be preceded by
      a <b>STOP SLAVE SQL_THREAD</b> statement.
    </li>

    <li>
      <tt>wait_for_slave_io_to_stop.inc</tt>

      Waits for the slave I/O thread to stop. Should be preceded by
      a <b>STOP SLAVE IO_THREAD</b> statement.
    </li>

    <li>
      <tt>wait_for_slave_param.inc</tt>

      Waits until <b>SHOW SLAVE STATUS</b> output contains a given
      value or a timeout occurs. Before including the file, you should
      set the <b>$slave_param</b> variable to the column name to
      look for in <b>SHOW SLAVE STATUS</b> output, and
      <b>$slave_param_value</b> to the value that you are waiting
      for the column to have.

      Example:

      @verbatim
  --let $slave_param= Slave_SQL_Running
  --let $slave_param_value= No
  --source include/slave_wait_slave_param.inc
      @endverbatim
    </li>

    <li>
      <tt>wait_for_slave_sql_error.inc</tt>

      Waits until the SQL thread for the current connection has gotten
      an error or a timeout occurs. Occurrence of an error is
      determined by waiting for the <b>Last_SQL_Errno</b> column of
      <b>SHOW SLAVE STATUS</b> output to have a nonzero value.
    </li>
  </ul>
*/

###############################################################################



## Thread Synchronization in Test Cases

/**
  @page PAGE_THREAD_SYNCHRONIZATION Thread Synchronization in Test Cases

  The Debug Sync facility allows placement of synchronization points
  in the code. They can be activated by statements that set the
  <b>debug_sync</b> system variable. An active synchronization point
  can emit a signal or wait for a signal to be emitted by another
  thread. This waiting times out after 300 seconds by default. The
  <b>`--debug-sync-timeout`=<em>N</em></b> option for
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" changes that
  timeout to <b><em>N</em></b> seconds. A timeout of zero disables
  the facility altogether, so that synchronization points will not
  emit or wait for signals, even if activated.

  The purpose of the timeout is to avoid a complete lockup in test
  cases. If for some reason the expected signal is not emitted by
  any thread, the execution of the affected statement will not block
  forever. A warning shows up when the timeout happens. That makes a
  difference in the test result so that it will not go undetected.

  For test cases that require the Debug Sync facility, include the
  following line in the test case file:

  @verbatim
  --source include/have_debug_sync.inc
  @endverbatim

  For a description of the Debug Sync facility and how to use
  synchronization points,
  see [MySQL Internals: Test Synchronization]
  (https://dev.mysql.com/doc/internals/en/test-synchronization.html).
*/

###############################################################################



## Suppressing Errors and Warning

/**
  @page PAGE_SUPPRESSING_ERRORS_WARNINGS Suppressing Errors and Warning

  After a test is finished, and if it didn't fail for some other
  reason, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will check
  the log written by the server(s) during the test for any warnings or
  errors. If it finds any, it will print them out and the test will be
  reported as failed.

  However, many warnings and also a few errors may occur normally
  without signifying a real problem. Also, many tests purposefully
  perform actions that will provoke warnings. For these not to result
  in a test failure, it is possible to specify that certain messages
  are to be suppressed.

  There is a list of global suppressions; warnings that will always be
  suppressed. These are listed in the file
  <b>include/mtr_warnings.sql</b>. Any error or warning that
  contains one of the listed strings will be ignored. It is not
  necessary to include the whole text, and regular expressions can be
  used, such as <b>.*</b> to match any substring or <b>[0-9]*</b>
  to match any number.

  You will rarely need to change or add to this global list, but you
  may need to suppress a particular error or warning for a new test
  case. A typical case is a test that performs some illegal action on
  purpose to test the respone; if this also results in a warning in
  the server log, this should not cause this particular test to fail.

  To add a suppresion for a warning text like for example <b>The
  table 't23456' is full</b> where the number in the table name can
  vary, add this line anywhere in the test case file:

  @verbatim
  call mtr.add_suppression("The table 't[0-9]*' is full");
  @endverbatim

  This may be put anythere in the test, but we recommend putting it
  either near the top, or just after the action that may result in the
  warning. If you're adding this line to an existing test, keep in
  mind that it will be echoed in the output, so the exact same line
  also needs to be added to the corresponding place in the result
  file. Alternatively, you may turn off logging like this, and will
  then not have to edit the result file:

  @verbatim
  --disable_query_log
  call mtr.add_suppression("The table 't[0-9]*' is full")
  --enable_query_log
  @endverbatim

  It is also possible to intruct @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" to skip the check for errors and warnings
  completely, by use of the <b>`--nowarnings`</b> command line
  option.
*/

###############################################################################



## Stopping and restarting a Server During a Test

/**
  @page PAGE_STOPPING_AND_RESTARTING_SERVER_DURING_TEST Stopping and Restarting a Server During a Test

  If a server dies during execution of a test case, this will be
  interpreted as a failure. However, there may be cases where you
  actually want to stop and possibly restart a server intentionally.
  It is possible to let the system know you expect a server to
  terminate, and to either wait or have it restarted immediately:

  Before you initiate the action that will stop the server, the test
  case should write either <b>restart</b> or <b>wait</b> to the
  file <b>$MYSQLTEST_VARDIR/tmp/mysqld.1.expect</b>. The easiest way
  to do this is using the <b>exec echo</b> construct like in the
  following example.

  If you write <b>restart</b> into this file, the server will be
  immediately restarted. If you write <b>wait</b> instead, the
  server will remain down, but can be restarted at a later time by
  writing <b>restart</b> to the file.

  If you write <b>restart:</b> or <b>restart_abort:</b> (please note
  the colon) followed by some command line parameters, those will
  be appended to the command line arguments of the server on restart.
  The former should be used when the server is expected to restart
  successfully while the latter comes with an implicit follow up wait
  and can be used when the server is expected to exit due to the
  provided arguments.

  @verbatim
  --exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
  --shutdown_server 10
  --source include/wait_until_disconnected.inc
  # Do something while server is down
  --enable_reconnect
  --exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
  --source include/wait_until_connected_again.inc
  @endverbatim

  The file name to write the command to will be
  <b>mysqld.2.expect</b> for the slave server in replication tests.
  Note that you have to use <b>$MYSQLTEST_VARDIR/tmp</b> for the
  directory here; if you use <b>$MYSQL_TMP_DIR</b> instead, it will
  not work when running tests in parallel.

  For your convenience, there are files which can be used to do
  what's shown in the example.

  <ul>
    <li>
      <tt>restart_mysqld.inc</tt>

      <b>include/restart_mysqld.inc</b> can be sourced in your
      own test case, to shutdown and start the server but
      the server is immediately started up again.

      @verbatim
  # Restart the server
  --source include/restart_mysqld.inc
      @endverbatim

      It is also possible to provide additional command line options
      to the restarted server, by writing a line with <b>restart:</b>
      (with a colon) followed by one or more options into the expect
      file. These extra options will be dropped if the same server
      is restarted again, unless they are repeated. A value for
      shutdown timeout can also be set with
      <b>$shutdown_server_timeout</b>, when this file is used.

      @verbatim
  # Restart server with extra parameters
  --let $shutdown_server_timeout= 10
  --let $restart_parameters= "restart: --innodb_autoinc_lock_mode=1"
  --source include/restart_mysqld.inc
      @endverbatim

    </li>

    <li>
      <tt>kill_and_restart_mysqld.inc</tt>

      Similar to <b>include/restart_mysqld.inc</b>,
      <b>include/kill_and_restart_mysqld.inc</b> can be used to
      restart the server, but setting <b>$shutdown_server_timeout</b>
      will have no effect.

      @verbatim
  # Restart server with extra parameters
  --let $restart_paramters= "restart: --innodb_autoinc_lock_mode=0"
  --source include/kill_and_restart_mysqld.inc
      @endverbatim

    </li>

    <li>
      <tt>shutdown_mysqld.inc and start_mysqld.inc</tt>

      If there is something to be done between the server shutdown
      and startup, there is a file
      <b>include/shutdown_mysqld.inc</b>
      which can be used to shutdown the server. A shutdown timeout
      value can be set and passed to
      <b>include/shutdown_mysqld.inc</b>.
      <b>include/start_mysqld.inc</b> can be used to start the
      server again after it has been shutdown.

      @verbatim
  # Timeout of 10 seconds
  --let $shutdown_server_timeout= 10
  --source include/shutdown_mysqld.inc
  # Do something while the server is down
  --source include/start_mysqld.inc
      @endverbatim

    </li>

    <li>
      <tt>kill_mysqld.inc</tt>

      To the kill the server,
      <b>include/kill_mysqld.inc</b> can be sourced, which
      internally calls <b>--shutdown_server 0</b>. Therefore,
      setting a value for shutdown timeout will have no effect.

      @verbatim
  --source include/kill_mysqld.inc
  # Do something while the server is down
  --source include/start_mysqld.inc
      @endverbatim

    </li>

    <li>
      <tt>wait_until_disconnected.inc</tt>

      It is recommended to use <b>include/wait_until_disconnected</b>
      in tandem with <b>--shutdown_server</b>. It ensures that all
      connections to the server are dropped before proceeding with
      the test. This file is sourced internally in
      <b>include/shutdown_mysqld.inc</b>,
      <b>include/kill_mysqld.inc</b>,
      <b>include/kill_and_restart_mysqld.inc</b>,
      and <b>include/restart_mysqld.inc</b>.

    </li>

    <li>
      <tt>wait_until_connected_again.inc</tt>

      <b>include/wait_until_connected_again.inc</b> is used after
      the server is started again, to ensure that the client
      connection is restored. It tries to establish the connection
      till a timeout occurs. This file is sourced internally in
      <b>include/start_mysqld.inc</b>,
      <b>include/restart_mysqld.inc</b>,
      and <b>include/kill_and_restart_mysqld.inc</b>.

    </li>

  </ul>

*/

###############################################################################



## Other Tips for Writing Test Cases

/**
  @page PAGE_TIPS_WRITING_TESTCASES Other Tips for Writing Test Cases

  <ul>
    <li>
      Writing loops

      If you need to do something in a loop, you can use something like this:

      @verbatim
  let $1= 10;
  while ($1)
  {
    # execute your statements here
    dec $1;
  }
      @endverbatim
    </li>

    <li>
      Pausing between statements

      To sleep between statements, use the <b>sleep</b> command. It
      supports fractions of a second. For example, <b>sleep 0.2</b>;
      sleeps 0.2 seconds.

      Try not to use the <b>sleep</b> command more than necessary.
      The more of them there are, the slower the test suite becomes.
      In some cases, heavy reliance on sleep operations is an indicator
      that the logic of a test should be reconsidered.
    </li>

    <li>
      Commenting the test result

      When the output in a result file is not understandable by
      inspection, it can be helpful to have the test case write
      comments to the result file that provide context. You can use
      the <b>echo</b> command for this:

      @verbatim
  --echo # Comment to explain the following output
      @endverbatim

      Of course, the same line (without <b>`--echo`</b>) will need
      to be put in the corresponding place in the result file, if you
      are adding this to an existing test.
    </li>

    <li>
      Sorting result sets

      If a test case depends on <b>SELECT</b> output being in a
      particular row order, use an <b>ORDER BY</b> clause. Do not
      assume that rows will be selected in the same order they are
      inserted, particularly for tests that might be run multiple
      times under conditions that can change the order, such as with
      different storage engines, or with and without indexing.

      If you do not want to add <b>ORDER BY</b> to the query, an
      alternative is to use the test command <b>sorted_result</b>
      which will sort the result of the following statement before
      putting it into the result file.
    </li>

    <li>
      Performing file system operations

      Avoid using <b>exec</b> or <b>system</b> to execute
      operating system commands for file system operations. This used
      to be very common, but OS commands tend to be platform specific,
      which reduces test portability. @ref PAGE_MYSQLTEST "mysqltest"
      now has several commands to perform these operations portably,
      so these commands should be used instead: <b>remove_file</b>,
      <b>chmod</b>, <b>mkdir</b>, and so forth.
    </li>

    <li>
      Local versus remote storage

      Some test cases depend on being run on local storage, and may
      fail when run on remote storage such as a network share. For
      example, if the test result can be affected by differences
      between local and remote file system times, the expected result
      might not be obtained. Failure of these test cases under such
      circumstances does not indicate an actual malfunction. It is not
      generally possible to determine whether tests are being run on
      local storage.
    </li>

    <li>
      Skipping consistency check or forcing restart after test

      As mentioned before, a test case should leave the server in a
      "clean" state for the next test. In cases where this is not
      possible or too costly, the test may in stead ask for the
      consistency check to be skipped, and the server(s) to be
      restarted. This is done by this executing this command at any
      time during the test:

      @verbatim
  call mtr.force_restart();
      @endverbatim

      This signals to @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
      that it should restart the server before the next test, and also
      skip the consistency check.
    </li>
  </ul>
*/

###############################################################################



## MySQL Test Programs

/**
  @page PAGE_MYSQLTEST_PROGRAMS MySQL Test Programs

  <h3>Table of Contents</h3>
  - @subpage PAGE_MYSQLTEST
  - @subpage PAGE_MYSQL_CLIENT_TEST
  - @subpage PAGE_MYSQL_TEST_RUN_PL
  - @subpage PAGE_MYSQL_STRESS_TEST_PL

  This chapter describes the test programs that run test cases. For
  information about the language used for writing test cases, see @ref
  PAGE_MYSQLTEST_LANGUAGE_REFERENCE.

  The test suite uses the following programs:

  <ul>
    <li>
      The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" Perl script
      is the main application used to run the MySQL test suite. It
      invokes @ref PAGE_MYSQLTEST "mysqltest" to run individual test
      cases. (Prior to MySQL 4.1, a similar shell script,
      <b>mysql-test-run</b>, can be used instead).
    </li>

    <li>
      @ref PAGE_MYSQLTEST "mysqltest" runs test cases.
    </li>

    <li>
      The @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" program is
      used for testing aspects of the MySQL client API that cannot be
      tested using @ref PAGE_MYSQLTEST "mysqltest" and its test
      language.
    </li>

    <li>
      The @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" Perl
      script performs stress-testing of the MySQL server.
    </li>
  </ul>
*/

###############################################################################



## mysqltest — Program to Run Test Cases

/**
  @page PAGE_MYSQLTEST mysqltest — Program to Run Test Cases

  The @ref PAGE_MYSQLTEST "mysqltest" program runs a test case against
  a MySQL server and optionally compares the output with a result
  file. This program reads input written in a special test language.
  Typically, you invoke @ref PAGE_MYSQLTEST "mysqltest" using @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" rather than invoking it
  directly.

  Features of @ref PAGE_MYSQLTEST "mysqltest":

  - Can send SQL statements to MySQL servers for execution

  - Can execute external shell commands

  - Can test whether the result from an SQL statement or shell
    command is as expected

  - Can connect to one or more standalone <b>mysqld</b> servers
    and switch between connections

  By default, @ref PAGE_MYSQLTEST "mysqltest" reads the test case on
  the standard input. To run @ref PAGE_MYSQLTEST "mysqltest" this way,
  you normally invoke it like this:

  @verbatim
  shell> mysqltest [options] [db_name] < test_file
  @endverbatim

  You can also name the test case file with a <b>`--test-
  file`=<em>file_name</em></b> option.

  The exit value from @ref PAGE_MYSQLTEST "mysqltest" is 0 for
  success, 1 for failure, and 62 if it skips the test case (for
  example, if after checking some preconditions it decides not to
  run the test).

  @ref PAGE_MYSQLTEST "mysqltest" supports the following options:

  <ul>
    <li>
      <tt>`--help`, -?</tt>

      Display a help message and exit.
    </li>

    <li>
      <tt>`--async-client`</tt>

      Enable asynchronous communication support in mysql protocol.
    </li>

    <li>
      <tt>`--basedir`=<b>dir_name</b>, -b <b>dir_name</b></tt>

      The base directory for tests.
    </li>

    <li>
      <tt>`--character-sets-dir`=<b>path</b></tt>

      The directory where character sets are installed.
    </li>

    <li>
      <tt>`--colored-diff`</tt>

      Colorize the diff part of the output.

      If enabled, @ref PAGE_MYSQLTEST "mysqltest" uses <b>diff</b>
      command with <b>`--color`='always'</b> option to print the
      colored diff. If the command fails or is not supported, @ref
      PAGE_MYSQLTEST "mysqltest" will report an error and abort the
      test run.

      @note
      <b>`--color`</b> option for <b>diff</b> command is available
      from <b>GNU diffutils</b> version <b>3.4</b>.
    </li>

    <li>
      <tt>`--compress`, -C</tt>

      Compress all information sent between the client and the server
      if both support compression.
    </li>

    <li>
      <tt>`--cursor-protocol`</tt>

      Use cursors for prepared statements.
    </li>

    <li>
      <tt>`--database`=<b>db_name</b>, -D <b>db_name</b></tt>

      The default database to use.
    </li>

    <li>
      <tt>`--debug`[=<b>debug_options</b>], -#[<b>debug_options</b>]</tt>

      Write a debugging log if MySQL is built with debugging support.
      The default <b><em>debug_options</em></b> value is
      <b>'d:t:S:i:O,/tmp/mysqltest.trace'</b>.
    </li>

    <li>
      <tt>`--debug-check`</tt>

      Print some debugging information when the program exits.
    </li>

    <li>
      <tt>`--debug-info`</tt>

      Print debugging information and memory and CPU usage statistics
      when the program exits.
    </li>


    <li>
      <tt>`--default-character-set`=<b>charset_name</b></tt>

      Use <b>charset_name</b> as the default character set for the
      client and connection. For more information, see [Connection
      Character Sets and Collations]
      (https://dev.mysql.com/doc/refman/8.0/en/charset-connection.html).
    </li>

    <li>
      <tt>`--explain-protocol`</tt>

      Run <b>EXPLAIN EXTENDED</b> on all <b>SELECT</b>,
      <b>INSERT</b>, <b>REPLACE</b>, <b>UPDATE</b> and
      <b>DELETE</b> queries.
    </li>

    <li>
      <tt>`--host`=<b>host_name</b>, -h <b>host_name</b></tt>

      Connect to the MySQL server on the given host.
    </li>

    <li>
      <tt>`--include`=<b>file_name</b>, -i <b>file_name</b></tt>

      Include the contents of the given file before processing the
      contents of the test file. The included file should have the
      same format as other @ref PAGE_MYSQLTEST "mysqltest" test files.
      This option has the same effect as putting a <b>`--source`
      <em>file_name</em></b> command as the first line of the test
      file.
    </li>

    <li>
      <tt>`--init-command`=<b>str</b></tt>

      SQL statement passed as an argument, to execute after connecting
      to the MYSQL server. If auto-reconnect is enabled, the statement
      is executed again after reconnection occurs.
    </li>
    <li>
      <tt>`--json-explain-protocol`</tt>

      Run <b>EXPLAIN EXTENDED</b> on all <b>SELECT</b>,
      <b>INSERT</b>, <b>REPLACE</b>, <b>UPDATE</b> and
      <b>DELETE</b> queries. The <b>json-explain-protocol</b>
    </li>

    <li>
      <tt>`--logdir`=<b>dir_name</b></tt>

      The directory to use for log files.
    </li>

    <li>
      <tt>`--mark-progress`</tt>

      Write the line number and elapsed time to
      <b><em>test_file</em>.progress.</b>
    </li>

    <li>
      <tt>`--max-connect-retries`=<b>num</b></tt>

      The maximum number of connection attempts when connecting to
      server.
    </li>

    <li>
      <tt>`--max-connections`=<b>num</b></tt>

      The maximum number of simultaneous server connections per client
      (that is, per test). If not set, the maximum is 128. Minimum
      allowed limit is 8, maximum is 5120.
    </li>

    <li>
      <tt>`--no-defaults`</tt>

      Do not read default options from any option files. If used, this
      must be the first option.
    </li>

    <li>
      <tt>`--plugin-dir`=<b>path</b></tt>

      The directory in which to look for plugins. It may be necessary
      to specify this option if the <b><em>default_auth</em></b>
      argument is used for the <b>connect()</b> command to specify
      an authentication plugin but @ref PAGE_MYSQLTEST "mysqltest"
      does not find it.
     </li>

    <li>
      <tt>`--password`[=<b>password</b>], -p[<b>password</b>]</tt>

      The password to use when connecting to the server. If you use
      the short option form (<b>-p</b>), you <b><em>cannot</em></b>
      have a space between the option and the password. If you omit
      the <b><em>password</em></b> value following the
      <b>`--password`</b> or <b>-p</b> option on the command line,
      you are prompted for one.
    </li>

    <li>
      <tt>`--port`=<b>port_num</b>, -P <b>port_num</b></tt>

      The TCP/IP port number to use for the connection.
    </li>

    <li>
      <tt>`--protocol`={TCP|SOCKET|PIPE|MEMORY}</tt>

      Choose the protocol for communication with the server.
      <b>SOCKET</b> is default.
    </li>

    <li>
      <tt>`--ps-protocol`</tt>

      Use the prepared-statement protocol for communication.
    </li>

    <li>
      <tt>`--quiet`</tt>

      Suppress all normal output. This is a synonym for
      <b>`--silent`</b>.
    </li>

    <li>
      <tt>`--record`, -r</tt>

      Record the output that results from running the test file into
      the file named by the <b>`--result-file`</b> option, if that
      option is given. It is an error to use this option without also
      using <b>`--result-file`</b>.
    </li>

    <li>
      <tt>`--result-file`=<b>file_name</b>, -R <b>file_name</b></tt>

      This option specifies the file for test case expected results.
      <b>`--result-file`</b>, together with <b>`--record`</b>,
      determines how @ref PAGE_MYSQLTEST "mysqltest" treats the test
      actual and expected results for a test case:

      <ul>
        <li>
          If the test produces no results, @ref PAGE_MYSQLTEST
          "mysqltest" exits with an error message to that effect,
          unless <b>`--result-file`</b> is given and the named file
          is an empty file.
        </li>

        <li>
          Otherwise, if <b>`--result-file`</b> is not given, @ref
          PAGE_MYSQLTEST "mysqltest" sends test results to the
          standard output.
        </li>

        <li>
          With <b>`--result-file`</b> but not <b>`--record`</b>,
          @ref PAGE_MYSQLTEST "mysqltest" reads the expected results
          from the given file and compares them with the actual
          results. If the results do not match, @ref PAGE_MYSQLTEST
          "mysqltest" writes a <b>.reject</b> file in the log
          directory, outputs a diff of the two files, and exits with
          an error.
        </li>

        <li>
          With both <b>`--result-file`</b> and <b>`--record`</b>,
          @ref PAGE_MYSQLTEST "mysqltest" updates the given file by
          writing the actual test results to it.
        </li>
      </ul>
    </li>

    <li>
      <tt>`--server-public-key-path`=<b>file_name</b></tt>

      The path name to a file containing the server RSA public key.
      The file must be in PEM format. The public key is used for RSA
      encryption of the client password for connections to the server
      made using accounts that authenticate with the
      <b>sha256_password</b> plugin. This option is ignored for
      client accounts that do not authenticate with that plugin. It is
      also ignored if password encryption is not needed, as is the
      case when the client connects to the server using an SSL
      connection.

      The server sends the public key to the client as needed, so it
      is not necessary to use this option for RSA password encryption
      to occur. It is more efficient to do so because then the server
      need not send the key.

      For additional discussion regarding use of the
      <b>sha256_password plugin</b>, including how to get the RSA
      public key, see [The SHA-256 Authentication Plugin]
      (https://dev.mysql.com/doc/refman/8.0/en/sha256-authentication-plugin.html).

    </li>

    <li>
      <tt>`--silent`, -s</tt>

      Suppress all normal output.
    </li>

    <li>
      <tt>`--skip-safemalloc`</tt>

      Do not use memory allocation checking.
    </li>

    <li>
      <tt>`--socket`=<b>path</b>, -S <b>path</b></tt>

      The socket file to use when connecting to <b>localhost</b>
      (which is the default host).
    </li>

    <li>
      <tt>`--sp-protocol`</tt>

      Execute DML statements within a stored procedure. For every DML
      statement, @ref PAGE_MYSQLTEST "mysqltest" creates and invokes a
      stored procedure that executes the statement rather than
      executing the statement directly.
    </li>

    <li>
      <tt>`--tail-lines`=<b>n</b></tt>

      Specify how many lines of the result to include in the output if
      the test fails because an SQL statement fails. The default is 0,
      meaning no lines of result printed.
    </li>

    <li>
      <tt>`--test-file`=<b>file_name</b>, -x <b>file_name</b></tt>

      Read test input from this file. The default is to read from the
      standard input.
    </li>

    <li>
      <tt>`--timer-file`=<b>file_name</b>, -m <b>file_name</b></tt>

      If given, the number of millisecond spent running the test will
      be written to this file. This is used by @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" for its reporting.
    </li>

    <li>
      <tt>`--tls-version`=<b>protocol_list</b></tt>

      The protocols permitted by the client for encrypted connections.
      The value is a comma-separated list containing one or more of
      these protocols: TLSv1, TLSv1.1, TLSv1.2.
    </li>

    <li>
      <tt>`--tmpdir`=<b>dir_name</b>, -t <b>dir_name</b></tt>

      The temporary directory where socket files are created.
    </li>

    <li>
      <tt>`--trace-exec`</tt>

      If enabled, this option causes @ref PAGE_MYSQLTEST "mysqltest"
      to immediately display the output from executed programs to
      stdout.
    </li>

    <li>
      <tt>`--user`=<b>user_name</b>, -u <b>user_name</b></tt>

      The MySQL user name to use when connecting to the server.
    </li>

    <li>
      <tt>`--verbose, -v`</tt>

      Verbose mode. Print out more information about what the program
      does.
    </li>

    <li>
      <tt>`--version, -V`</tt>

      Display version information and exit.
    </li>

    <li>
      <tt>`--view-protocol`</tt>

      Every <b>SELECT</b> statement is wrapped inside a view.
    </li>
  </ul>
*/

###############################################################################



## mysql_client_test — Test Client API

/**
  @page PAGE_MYSQL_CLIENT_TEST mysql_client_test — Test Client API

  The @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" program is used
  for testing aspects of the MySQL client API that cannot be tested
  using @ref PAGE_MYSQLTEST "mysqltest" and its test language. @ref
  PAGE_MYSQL_CLIENT_TEST "mysql_client_test" is run as part of the
  test suite.

  The source code for the programs can be found in in
  <b>tests/mysql_client_test.c</b> in a source distribution. The
  program serves as a good source of examples illustrating how to use
  various features of the client API.

  @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" is used in a test by
  the same name in the main tests suite of @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" but may also be run directly. Unlike the other
  programs listed here, it does not read an external description of
  what tests to run. Instead, all tests are coded into the program,
  which is written to cover all aspects of the C language API.

  @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" supports the
  following options:

  <ul>
    <li>
      <tt>`--help`, -?</tt>

      Display a help message and exit.
    </li>

    <li>
      <tt>`--basedir`=<b>dir_name</b>, -b <b>dir_name</b></tt>

      The base directory for the tests.
    </li>

    <li>
      <tt>`--count`=<b>count</b>, -t <b>count</b></tt>

      The number of times to execute the tests.
    </li>

    <li>
      <tt>`--database`=<b>db_name</b>, -D <b>db_name</b></tt>

      The database to use.
    </li>

    <li>
      <tt>`--debug`[=<b>debug_options</b>], -#[<b>debug_options</b>]</tt>

      Write a debugging log if MySQL is built with debugging support.
      The default <b>debug_options</b> value is
      <tt>'d:t:o,/tmp/mysql_client_test.trace'</tt>.
    </li>

    <li>
      <tt>`--getopt-ll-test`=<b>option</b>,
          -g <b>option</b></tt>

      Option to use for testing bugs in the <tt>getopt</tt> library.
    </li>

    <li>
      <tt>`--host`=<b>host_name</b>, -h <b>host_name</b></tt>

      Connect to the MySQL server on the given host.
    </li>

    <li>
      <tt>`--password`[=<b>password</b>], -p[<b>password]</b></tt>

      The password to use when connecting to the server. If you use
      the short option form (<b>-p</b>), you <tt><em>cannot</em></tt>
      have a space between the option and the password. If you omit
      the <b>password</b> value following the
      <b>`--password`</b> or <b>-p</b> option on the command line,
      you are prompted for one.
    </li>

    <li>
      <tt>`--port`=<b>port_num</b>, -P <b>port_num</b></tt>

      The TCP/IP port number to use for the connection.
    </li>

    <li>
      <tt>`--show-tests`, -T</tt>

      Show all test names.
    </li>

    <li>
      <tt>`--silent`, -s</tt>

      Be more silent.
    </li>

    <li>
      <tt>`--socket`=<b>path</b>, -S <b>path</b></tt>

      The socket file to use when connecting to <b>localhost</b>
      (which is the default host).
    </li>

    <li>
      <tt>`--testcase`, -c</tt>

      The option is used when called from @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl", so that @ref PAGE_MYSQL_CLIENT_TEST
      "mysql_client_test" may optionally behave in a different way
      than if called manually, for example by skipping some tests.
      Currently, there is no difference in behavior but the option is
      included to make this possible.
    </li>

    <li>
      <tt>`--user`=<b>user_name</b>, -u <b>user_name</b></tt>

      The MySQL user name to use when connecting to the server.
    </li>

    <li>
      <tt>`--vardir`=<b>dir_name</b>, -v <b>dir_name</b></tt>

      The data directory for tests. The default is
      <b>mysql-test/var</b>.
    </li>
  </ul>
*/

###############################################################################



## mysql-test-run.pl — Run MySQL Test Suite

/**
  @page PAGE_MYSQL_TEST_RUN_PL mysql-test-run.pl — Run MySQL Test Suite

  The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" Perl script is
  the main application used to run the MySQL test suite. It invokes
  @ref PAGE_MYSQLTEST "mysqltest" to run individual test cases.

  Invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" in the
  <b>mysql-test</b> directory like this:

  @verbatim
  shell> mysql-test-run.pl [options] [test_name] ...
  @endverbatim

  Each <b>test_name</b> argument names a test case. The test
  case file that corresponds to the test name is
  <b>t/<em>test_name</em>.test</b>.

  For each <b>test_name</b> argument, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs the named test case.
  With no <b>test_name</b> arguments, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs all <b>.test</b>
  files in the <b>t</b> subdirectory.

  If no suffix is given for the test name, a suffix of <b>.test</b>
  is assumed. Any leading path name is ignored. These commands are
  equivalent:

  @verbatim
  shell> mysql-test-run.pl mytest
  shell> mysql-test-run.pl mytest.test
  shell> mysql-test-run.pl t/mytest.test
  @endverbatim

  A suite name can be given as part of the test name. That is, the
  syntax for naming a test is:

  @verbatim
  [suite_name.]test_name[.suffix]
  @endverbatim

  If a suite name is given, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" looks in that suite for the test. The test file
  corresponding to a test named <b>suite_name.test_name</b> is
  found in
  <b>suite/<em>suite_name</em>/t/<em>test_name</em>.test</b>.
  There is also an implicit suite name <b>main</b> for the tests in
  the top <b>t</b> directory. With no suite name, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" looks in the default list
  of suites for a match and runs the test in any suites where it finds
  the test. Suppose that the default suite list is <b>main</b>,
  <b>binlog</b>, <b>rpl</b>, and that a test <b>mytest.test</b>
  exists in the <b>main</b> and <b>rpl</b> suites. With an
  argument of <b>mytest</b> or <b>mytest.test</b>, @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will run
  <b>mytest.test</b> from the <b>main</b> and <b>rpl</b> suites.

  To run a family of test cases for which the names share a common
  prefix, use the <b>`--do-test`=<em>prefix</em></b> option.
  For example, <b>`--do-test`=rpl</b> runs the replication tests
  (test cases that have names beginning with <b>rpl</b>).
  <b>`--skip-test`</b> has the opposite effect of skipping test
  cases for which the names share a common prefix.

  The argument for the <b>`--do-test`</b> and <b>`--skip-test`</b>
  options also allows more flexible specification of which tests to
  perform or skip. If the argument contains a pattern metacharacter
  other than a lone period, it is interpreted as a Perl regular
  expression and applies to test names that match the pattern. If the
  argument contains a lone period or does not contain any pattern
  metacharacters, it is interpreted the same way as previously and
  matches test names that begin with the argument value. For example,
  <b>`--do-test`=testa</b> matches tests that begin with testa,
  <b>`--do-test`=main.testa</b> matches tests in the <b>main</b>
  test suite that begin with <b>testa</b>, and
  <b>`--do-test`=main.*testa</b> matches test names that contain
  <b>main</b> followed by <b>testa</b> with anything in between.
  In the latter case, the pattern match is not anchored to the
  beginning of the test name, so it also matches names such as
  <b>xmainytesta</b>.

  It is also possible to put a list of test names in a file and have
  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" run those tests,
  using the option <b>`--do-test-list`=file</b>. The tests should be
  listed one per line in the file. A line beginning with <b>#</b>
  indicates a comment and is ignored. Similary an empty line in the
  file is also ignored.

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" also supports a
  <b>`--do-suite`</b> option, which is similar to <b>`--do-test`</b>
  but permits specifying entire suites of tests to run.

  To perform setup prior to running tests, @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" needs to invoke <b>mysqld</b> with the
  <b>`--initialize`</b> and <b>`--skip-grant-tables`</b> options.
  If MySQL was built with the compiler flag
  <b>-DDISABLE_GRANT_OPTIONS</b>, then <b>`--initialize`</b>,
  <b>`--skip-grant-tables`</b>, and <b>`--init-file`</b> will be
  disabled. To handle this, set the <b>MYSQLD_BOOTSTRAP</b>
  environment variable to the full path name of a server that has all
  options enabled. @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
  will use that server to perform setup; it is not used to run the
  tests.

  The <b>init_file</b> test will fail if <b>`--init-file`</b> is
  disabled. This is an expected failure in this case.

  To run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" on Windows,
  you'll need either Cygwin or ActiveState Perl to run it. You may
  also need to install the modules required by the script. To run the
  test script, change location into the <b>mysql-test</b> directory,
  set the <b>MTR_VS_CONFIG</b> environment variable to the
  configuration you selected earlier (or use the <b>`--vs-config`</b>
  option), and invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl".
  For example (using <b>Cygwin</b> and the <b>bash</b> shell):

  @verbatim
  shell> cd mysql-test
  shell> export MTR_VS_CONFIG=debug
  shell> ./mysqltest-run.pl --force --timer
  shell> ./mysqltest-run.pl --force --timer --ps-protocol
  @endverbatim

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" uses several
  environment variables. Some of them are listed in the following
  table. Some of these are set from the outside and used by @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", others are set by @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" instead, and may be
  referred to in tests.

  <table>
    <tr>
      <th>Variable</th>
      <th>Description</th>
    </tr>
    <tr>
      <td><tt>EXE_MYSQL</tt></td>
      <td>
        Path to the mysql client executable
      </td>
    </tr>
    <tr>
      <td><tt>MTR_BUILD_THREAD</tt></td>
      <td>
        If set, defines which port number range is used for the server
      </td>
    </tr>
    <tr>
      <td><tt>MTR_MAX_PARALLEL</tt></td>
      <td>
        If set, defines maximum number of parallel threads if
        <b>`--parallel`=auto</b> is given
      </td>
    </tr>
    <tr>
      <td><tt>MTR_MEM</tt></td>
      <td>
        If set to anything, will run tests with files in "memory"
        using tmpfs or ramdisk. Not available on Windows. Same as
        <b>`--mem`</b> option
      </td>
    </tr>
    <tr>
      <td><tt>MTR_<b>NAME</b>_TIMEOUT</tt></td>
      <td>
        Setting of a timeout in minutes or seconds, corresponding to
        command line option <b>`--name-timeout`</b>. Avaliable timeout
        names are <b>TESTCASE</b>, <b>SUITE</b> (both in minutes) and
        <b>START</b>, <b>SHUTDOWN</b>, <b>CTEST</b> (all in seconds).
        <b>MTR_CTEST_TIMEOUT</b> is for <b>ctest</b> unit tests.
      </td>
    </tr>
    <tr>
      <td><tt>MTR_PARALLEL</tt></td>
      <td>
        If set, defines number of parallel threads executing tests. Same
        as <tt>`--parallel`</tt> option
      </td>
    </tr>
    <tr>
      <td><tt>MTR_PORT_BASE</tt></td>
      <td>
        If set, defines which port number range is used for the server
      </td>
    </tr>
    <tr>
      <td><tt>MTR_RECORD</tt></td>
      <td>
        Set to 1 if a MTR run is started with <b>`--record`</b>
        option, 0 otherwise.
      </td>
    </tr>
    <tr>
      <td><tt>MTR_UNIQUE_IDS_DIR</tt></td>
      <td>
        The method in which the free ports are allocated is by maintaining
        a list under the unique ids directory. In case there are multiple
        chroot environments on the same host, then the same set of ports may
        get allocated for all environments, because there will be
        multiple unique ids directories in different physical locations,
        but MTR assumes it is the same directory.
        This will lead to a conflict while reserving ports for the
        different MTR runs. To avoid this problem, when using chroot
        environments, <b>MTR_UNIQUE_IDS_DIR</b> environment variable can
        be set to a common location, so that all the MTR processes will
        use the same unique ids directory.
        Users have to make sure this variable is set to the same path
        on all environments, and that this common path is mounted on
        all the environments.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL</tt></td>
      <td>
        Full path to <b>mysql</b> client executable ,along with options passed to server.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_BIN_PATH</tt></td>
      <td>
        Path to the directory having mysql binaries.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_CLIENT_BIN_PATH</tt></td>
      <td>
        Path to the directory having client binaries.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_CONFIG_EDITOR</tt></td>
      <td>
        Path name to <b>mysql_config_editor</b> binary.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_OPTIONS</tt></td>
      <td>
        Options passed to server when connecting from mysql client.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_TEST</tt></td>
      <td>
        Path name to @ref PAGE_MYSQLTEST "mysqltest" binary
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_TEST_DIR</tt></td>
      <td>
        Full path to the <tt>mysql-test</tt> directory where tests
        are being run from
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_TEST_LOGIN_FILE</tt></td>
      <td>
        Path name to login file used by <b>mysql_config_editor</b>.
        If not set, the default is <b>$HOME/.mylogin.cnf</b>, or
        <b>\%APPDATA%\\MySQL\\\.mylogin.cnf</b> on Windows.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQL_TMP_DIR</tt></td>
      <td>
        Path to temp directory used for temporary files during tests
      </td>
    </tr>
    <tr>
      <td><tt>MYSQLD</tt></td>
      <td>
        Full path to server executable used in tests.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQLD_BOOTSTRAP</tt></td>
      <td>
        Full path name to <b>mysqld</b> that has all options enabled
      </td>
    </tr>
    <tr>
      <td><tt>MYSQLD_BOOTSTRAP_CMD</tt></td>
      <td>
        Full command line used for initial database setup for this
        test batch
      </td>
    </tr>
    <tr>
      <td><tt>MYSQLD_CMD</tt></td>
      <td>
        Command line for starting server as used in tests, with the
        minimum set of required arguments.
      </td>
    </tr>
    <tr>
      <td><tt>MYSQLTEST_VARDIR</tt></td>
      <td>
        Path name to the <b>var</b> directory that is used for logs,
        temporary files, and so forth
      </td>
    </tr>
    <tr>
      <td><tt>NUMBER_OF_CPUS</tt></td>
      <td>
        Defines number of processors.
      </td>
    </tr>
    <tr>
      <td><tt>TSAN_OPTIONS</tt></td>
      <td>
        Path name to a file containing ThreadSanitizer suppressions.
      </td>
    </tr>
  </table>

  The variable <b>MTR_PORT_BASE</b> is a more logical replacement
  for the original variable <b>MTR_BUILD_THREAD</b>. It gives the
  actual port number directly (will be rounded down to a multiple of
  10). If you use <b>MTR_BUILD_THREAD</b>, the port number is found
  by multiplying this by 10 and adding 10000.

  The variable <b>$MYSQL_OPTIONS</b> contains only the options used while
  connecting to server from mysql client . whereas variable <b>$MYSQL</b>
  contains path to mysql client executable along with options .

  Tests sometimes rely on certain environment variables being defined.
  For example, certain tests assume that <b>MYSQL_TEST</b> is
  defined so that @ref PAGE_MYSQLTEST "mysqltest" can invoke itself
  with <b>exec $MYSQL_TEST</b>.

  Other tests may refer to the last three variables listed in the
  preceding table, to locate files to read or write. For example,
  tests that need to create files will typically put them in
  <b>$MYSQL_TMP_DIR/file_name</b>.

  The variable <b>$MYSQLD_CMD</b> will include any server options
  added with the <b>`--mysqld`</b> option to @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", but will not include
  server options added specifically for the currently running test.

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" supports the
  options in the following list. An argument of -- tells @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" not to process any
  following arguments as options.

  <ul>
    <li>
      <tt>`--help, -h`</tt>

      Display a help message and exit.
    </li>

    <li>
      <tt>`--accept-test-fail`</tt>

      Do not print an error and do not give exit 1 if some tests failed,
      but test run was completed. This also enables --force.
    </li>

    <li>
      <tt>`--big-test`</tt>

      Allow tests marked as "big" to run. Tests can be thus marked by
      including the line <b>`--source` include/big_test.inc</b>, and
      they will only be run if this option is given, or if the
      environment variable <b>BIG_TEST</b> is set to 1.

      This is typically done for tests that take very long to run, or
      that use very much resources, so that they are not suitable for
      running as part of a normal test suite run.

      If both <b>`--big-test`</b> and <b>`--only-big-tests`</b>
      are given, <b>`--only-big-tests`</b> is ignored.

      @note
      This option is enabled by default when test cases are specified
      on command line.
    </li>

    <li>
      <tt>`--boot-dbx`</tt>

      Run the <b>mysqld</b> server used for bootstrapping the database
      through the <b>dbx</b> debugger.
    </li>

    <li>
      <tt>`--boot-ddd`</tt>

      Run the <b>mysqld</b> server used for bootstrapping the database
      through the <b>ddd</b> debugger.
    </li>

    <li>
      <tt>`--boot-gdb`</tt>

      Run the <b>mysqld</b> server used for bootstrapping the database
      through the <b>gdb</b> debugger.

      See also the <b>`--manual-boot-gdb`</b> option.
    </li>

    <li>
      <tt>`--build-thread`=<b>number</b></tt>

      Specify a number to calculate port numbers from. The formula is
      <b>10 * build_thread + 10000</b>. Instead of a number, it can
      be set to <b>auto</b>, which is also the default value, in
      which case @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will
      allocate a number unique to this host.

      The value (number or <b>auto</b>) can also be set with the
      <b>MTR_BUILD_THREAD</b> environment variable.

      This option is kept for backward compatibility. The more logical
      <b>`--port-base`</b> is recommended instead.
    </li>

    <li>
      <tt>`--callgrind`</tt>

      Instructs <b>valgrind</b> to use <b>callgrind</b>.
    </li>

    <li>
      <tt>`--charset-for-testdb`=<b>charset_name</b></tt>

      Specify the default character set for the test database. The
      default value is latin1.
    </li>

    <li>
      <tt>`--check-testcases`</tt>

      Check test cases for side effects. This is done by checking the
      system state before and after each test case. If there is any
      difference, the test case is marked as failed because of it.

      Similarly, when <b>`--check-testcases`</b> option is enabled,
      MTR does additional check for missing <b>.result</b> file and a
      test case not having its corresponding <b>.result</b> file is
      marked as failed.

      This check is enabled by default. To disable it, use the
      <b>`--nocheck-testcases`</b> option.
    </li>

    <li>
      <tt>`--clean-vardir`</tt>

      Clean up the <b>var</b> directory with logs and test results
      etc. after the test run, but only if there were no test
      failures. This option only has effect if also running with
      option <b>`--mem`</b>. The intent is to alleviate the problem
      of using up memory for test results, in cases where many
      different test runs are being done on the same host.
    </li>

    <li>
      <tt>`--client-bindir`=<b>path</b></tt>

      The path to the directory where client binaries are located.
    </li>

    <li>
      <tt>`--client-dbx`</tt>

      Start @ref PAGE_MYSQLTEST "mysqltest" in the <b>dbx</b>
      debugger.
    </li>

    <li>
      <tt>`--client-ddd`</tt>

      Start @ref PAGE_MYSQLTEST "mysqltest" in the <b>ddd</b>
      debugger.
    </li>

    <li>
      <tt>`--client-debugger`=<b>debugger</b></tt>

      Start @ref PAGE_MYSQLTEST "mysqltest" in the named debugger.
    </li>

    <li>
      <tt>`--client-gdb`</tt>

      Start @ref PAGE_MYSQLTEST "mysqltest" in the <b>gdb</b> debugger.
    </li>

    <li>
      <tt>`--client-libdir`=<b>path</b></tt>

      The path to the directory where client libraries are located.
    </li>

    <li>
      <tt>`--colored-diff`</tt>

      Colorize the diff part of the output.

      When this option is enabled, @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" passes <b>`--colored-diff`</b> option to
      @ref PAGE_MYSQLTEST "mysqltest". If colored diff is enabled,
      @ref PAGE_MYSQLTEST "mysqltest" uses <b>diff</b> command with
      <b>`--color`='always'</b> option to print the colored diff.

      @note
      <b>`--color`</b> option for <b>diff</b> command is available
      from <b>GNU diffutils</b> version <b>3.4</b>.
    </li>

    <li>
      <tt>`--combination`=<b>value</b></tt>

      Extra option to pass to <b>mysqld</b>. The value should consist
      of a single <b>mysqld</b> option including dashes. This option
      is similar to <b>`--mysqld`</b> but has a different effect.
      @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" executes
      multiple test runs, using the options for each instance of
      <b>`--combination`</b> in successive runs. If
      <b>`--combination`</b> is given only once, it has no effect and
      behaves similar to <b>`--mysqld`</b> option.

      If a test being run, passes the same <b>`--mysqld`</b> option from an
      opt file which is also passed from <b>`--combination`</b>, then that
      <b>`--combination`</b> run is skipped because opt file always takes
      higher precedence over command line options. The test will just run
      once using the options provided in the opt file.

      For test runs specific to a given test suite, an alternative to
      the use of <b>`--combination`</b> is to create a
      <b>combinations</b> file in the suite directory. The file
      should contain a section of options for each test run. See @ref
      PAGE_PASSING_OPTIONS.

      For test specific combinations, onc can use
      <b><em>test_name</em>.combinations</b> file in the test
      directory.
    </li>

    <li>
      <tt>`--comment`=<b>str</b></tt>

      Write <b><em>str</em></b> to the output within lines filled
      with #, as a form of banner.
    </li>

    <li>
      <tt>`--compress`</tt>

      Compress all information sent between the client and the server
      if both support compression.
    </li>

    <li>
      <tt>`--cursor-protocol`</tt>

      Pass the <b>`--cursor-protocol`</b> option to @ref
      PAGE_MYSQLTEST "mysqltest" (implies <b>`--ps-protocol`</b>).
    </li>

    <li>
      <tt>`--dbx`</tt>

      Start <b>mysqld</b> in the <b>dbx</b> debugger.
    </li>

    <li>
      <tt>`--ddd`</tt>

      Start <b>mysqld</b> in the <b>ddd</b> debugger.
    </li>

    <li>
      <tt>`--debug`</tt>

      Dump trace output for all clients and servers.
    </li>

    <li>
      <tt>`--debugger`=<b>debugger</b></tt>

      Start <b>mysqld</b> using the named debugger.
    </li>

    <li>
      <tt>`--debug-common`</tt>

      This option works similar to <b>`--debug`</b> but turns on
      debug only for the debug macro keywords <b>query</b>,
      <b>info</b>, <b>error</b>, <b>enter</b>, <b>exit</b>
      which are considered the most commonly used.
    </li>

    <li>
      <tt>`--debug-server`</tt>

      Runs <b>mysqld.debug</b> (if available) instead of
      <b>mysqld</b> as server. If it does find <b>mysqld.debug</b>,
      it will search for plugin libraries in a subdirectory
      <b>debug</b> under the directory where it's normally located.
      This option does not turn on trace output and is independent of
      the <b>debug</b> option.
    </li>

    <li>
      <tt>`--debug-sync-timeout`=<b>seconds</b></tt>

      Controls whether the Debug Sync facility for testing and
      debugging is enabled. The option value is a timeout in seconds.
      The default value is 300. A value of 0 disables Debug Sync. The
      value of this option also becomes the default timeout for
      individual synchronization points.

      @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" passes
      <b>`--loose-debug-sync-timeout`=<em>seconds</em></b> to
      <b>mysqld</b>. The <b>`--loose`</b> prefix is used so that
      <b>mysqld</b> does not fail if Debug Sync is not compiled in.

      For information about using the Debug Sync facility for testing,
      see @ref PAGE_THREAD_SYNCHRONIZATION.
    </li>

    <li>
      <tt>`--default-myisam`</tt>

      Use <b>MyISAM</b> as the default storage engine for all except
      <b>InnoDB</b>-specific tests. This option is off by default.
    </li>

    <li>
      <tt>`--defaults-file`=<b>file_name</b></tt>

      Use the named file as fixed config file template for all tests.
    </li>

    <li>
      <tt>`--defaults_extra_file`=<b>file_name</b></tt>

      Add setting from the named file to all generated configs.
    </li>

    <li>
      <tt>`--do-suite`=<b>prefix or regex</b></tt>

      Run all test cases from suites having a name that begins with
      the given <b><em>prefix</em></b> value or matches the regular
      expression. If the argument matches no existing suite, @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" aborts.

      The argument for the <b>`--do-suite`</b> option allows more
      flexible specification of which tests to perform. See the
      description of the <b>`--do-test`</b> option for details.

      In case a suite or a list of suites is specified using the
      <b>`--suite`</b> option along with a pattern or regular
      expression given through <b>`--do-suite`</b>, only suite(s)
      from the list which match the prefix or regular expression
      are picked up.

      When the <b>`--do-suite`</b> option is specified more than once
      on the command line, only the last occurrence is considered.
    </li>

    <li>
      <tt>`--do-test`=<b>prefix or regex</b></tt>

      Run all test cases having a name that begins with the given
      <b><em>prefix</em></b> value or matches the regular expression.
      This option provides a convenient way to run a family of
      similarly named tests.

      The argument for the <b>`--do-test`</b> option allows more
      flexible specification of which tests to perform. If the
      argument contains a pattern metacharacter other than a lone
      period, it is interpreted as a Perl regular expression and
      applies to test names that match the pattern. If the argument
      contains a lone period or does not contain any pattern
      metacharacters, it is interpreted the same way as previously and
      matches test names that begin with the argument value. For
      example, <b>`--do-test`=testa</b> matches tests that begin with
      <b>testa</b>, <b>`--do-test`=main.testa</b> matches tests in the
      <b>main</b> test suite that begin with <b>testa</b>, and
      <b>`--do-test`=main.*testa</b> matches test names that contain
      <b>main</b> followed by <b>testa</b> with anything in between.
      In the latter case, the pattern match is not anchored to the
      beginning of the test name, so it also matches names such as
      <b>xmainytestz</b>.
    </li>

    <li>
      <tt>`--do-test-list`=<b>file</b></tt>

      Run all tests listed in the file <b><em>file</em></b>. In this
      file, tests should be listed one per line in any of the
      following forms.

      @verbatim
  <test_name>
  <test_name>.test
  <suite_name>.<test_name>
  <suite_name>.<test_name>.test
  path_to_the_test_file
      @endverbatim

      Path to the test file should either be an absolute path or a
      relative path from base directory.

      To run a test present in a non-default test suite, either the
      test name qualified with the suite name i.e
      <b>suite_name.test_name[.test]</b> or a <b>path</b> to the
      test file should be mentioned.

      A line beginning with <b>#</b> will be ignored and is considered
      as a comment. Similarly an empty line in the file is also
      ignored.

      For example, consider a test list file <b>test_list</b>
      containing following tests.

      @verbatim
  # Main suite test '1st.test'
  1st
  1st.test
  main.1st
  main.1st.test
  mysql-test/t/1st.test

  # GIS suite test 'geohash_functions.test'
  gis.geohash_functions
  mysql-test/suite/gis/t/geohash_functions.test

  # Non-default suite 'test_services'
  test_services.test_services
  mysql-test/suite/test_services/t/test_services.test

  # Internal suite test 'i_main.user_var'
  i_main.user_var
  internal/mysql-test/suite/i_main/t/user_var.test
      @endverbatim

      Following command runs the list of tests mentioned in the file.

      @verbatim
  ./mysql-test-run.pl --do-test-list=test_list
      @endverbatim

      If both <b>`--suite`</b> and <b>`--do-test-list`</b> are given,
      and if suite name is <em>not</em> part of the test names listed
      in the file, then those tests are searched only in the suites
      mentioned in <b>`--suite`</b> option.
    </li>

    <li>
      <tt>`--enable-disabled`</tt>

      Ignore any <b>disabled.def</b> file, and run also tests marked
      as disbaled. Success or failure of those tests will be reported
      the same way as other tests.

      @note
      This option is enabled by default when test cases are specified
      on command line.
    </li>

    <li>
      <tt>`--explain-protocol`</tt>

      Run <b>EXPLAIN EXTENDED</b> on all <b>SELECT</b>, <b>INSERT</b>,
      <b>REPLACE</b>, <b>UPDATE</b> and <b>DELETE</b> queries.
    </li>

    <li>
      <tt>`--extern` <b>option=value</b></tt>

      Use an already running server. The option/value pair is what is
      needed by the <b>mysql</b> client to connect to the server. Each
      <b>`--extern`</b> can only take one option/value pair as
      argument, so it you need more you need to repeat
      <b>`--extern`</b> for each of them.

      @verbatim
  ./mysql-test-run.pl --extern socket=var/tmp/mysqld.1.sock main.1st
      @endverbatim

      During server initialization, @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" creates a default database <b>test</b> which
      is used while establishing an initial connection with the
      server. When running a test case with <b>--extern</b> option, if
      <b>test</b> database doesn't exist on an already running server,
      then the test case will fail with following error.

      @verbatim
  mysqltest: Could not open connection 'default': 1049 Unknown database 'test'
      @endverbatim

      So if you're running a test case on an already running server
      using <b>--extern</b> option, it is necessary to create
      <b>test</b> database on the server before running the test case
      if it doesn't exist.

      Similarly, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
      creates <b>mtr</b> database during server initialization which
      contains tables and procedures used to suppress global or test
      specific warnings. If a test case refers to this database and
      the database doesn't exist on running server, then the test case
      will fail.

      @note
      If a test case has an <b>.opt</b> file that requires the server
      to be restarted with specific options, the file will not be
      used. The test case likely will fail as a result.
    </li>

    <li>
      <tt>`--fast`</tt>

      Do not perform controlled shutdown when servers need to be
      restarted or at the end of the test run. This is equivalent to
      using <b>`--shutdown-timeout`=0</b>.
    </li>

    <li>
      <tt>`--force`</tt>

      Normally, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" exits
      if a test case fails. <b>`--force`</b> causes execution to
      continue regardless of test case failure.
    </li>

    <li>
      <tt>`--force-restart`</tt>

      Always restart the server(s) between each tast case, whether
      it's needed or not. Will also restart between repeated runs of
      the same test case. This may be useful e.g. when looking for the
      source of a memory leak, as there will only have been one test
      run before the server exits.

      Enabling this option will cause @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl" to restart the server(s) on a newly
      intialized data directory.

      This option can also be mentioned in
      <b><em>test-name</em>-master.opt</b> file for a test case to
      restart the server(s) before it runs.
    </li>

    <li>
      <tt>`--gcov`</tt>

      Run tests with the <b>gcov</b> test coverage tool.
    </li>

    <li>
      <tt>`--gdb`</tt>

      Start <b>mysqld</b> in the <b>gdb</b> debugger.
    </li>

    <li>
      <tt>`--gprof`</tt>

      Run tests with the <b>gprof</b> profiling tool.
    </li>

    <li>
      <tt>`--include-ndbcluster`, `--include-ndb`</tt>

      Run also tests that need Cluster.
    </li>

    <li>
      <tt>`--initialize`=<b>value</b></tt>

      Extra bootstrap options that need be to be passed during
      the server initialization process can be passed as a value
      to <b>`--initialize`</b>. The server is then started with
      the specified value. Only one option may be specified in
      <b>value</b>; to specify more than one, use additional
      <b>`--initialize`</b> options.

      Options passed as a value to <b>`--mysqld`</b> will have
      precedence over the options passed to <b>`--initialize`</b>,
      regardless of the order in which they passed on the command
      line.
    </li>

    <li>
      <tt>`--json-explain-protocol`</tt>

      Run <b>EXPLAIN FORMAT=JSON</b> on all <b>SELECT</b>,
      <b>INSERT</b>, <b>REPLACE</b>, <b>UPDATE</b> and <b>DELETE</b>
      queries.
    </li>

    <li>
      <tt>`--manual-boot-gdb`</tt>

      This option is similar to <b>`--boot-gdb`</b> but attaches the
      debugger to the server during the bootstrapping process,
      permitting the use of a remote debugger.
    </li>

    <li>
      <tt>`--manual-dbx`</tt>

      Use a server that has already been started by the user in the
      <b>dbx</b> debugger.
    </li>

    <li>
      <tt>`--manual-ddd`</tt>

      Use a server that has already been started by the user in the
      <b>ddd</b> debugger.
    </li>

    <li>
      <tt>`--manual-debug`</tt>

      Use a server that has already been started by the user in a
      debugger.
    </li>

    <li>
      <tt>`--manual-gdb`</tt>

      Use a server that has already been started by the user in the
      <b>gdb</b> debugger.
    </li>

    <li>
      <tt>`--mark-progress`</tt>

      Marks progress with timing (in milliseconds) and line number in
      <b>var/log/<em>testname</em>.progress</b>.
    </li>

    <li>
      <tt>`--max-connections`=<b>num</b></tt>

      The maximum number of simultaneous server connections that may
      be used per test. If not set, the maximum is 128. Minimum
      allowed limit is 8, maximum is 5120. Corresponds to the same
      option for @ref PAGE_MYSQLTEST "mysqltest".
    </li>

    <li>
      <tt>`--max-save-core`=<b>N</b></tt>

      Limit the number of core files saved, to avoid filling up disks
      in case of a frequently crashing server. Defaults to 5, set to 0
      for no limit. May also be set with the environment variable
      <tt>MTR_MAX_SAVE_CORE</tt>.
    </li>

    <li>
      <tt>`--max-save-datadir`=<b>N</b></tt>

      Limit the number of data directories saved after failed tests,
      to avoid filling up disks in case of frequent failures. Defaults
      to 20, set to 0 for no limit. May also be set with the
      environment variable <b>MTR_MAX_SAVE_DATADIR</b>.
    </li>

    <li>
      <tt>`--max-test-fail`=<b>N</b></tt>

      Stop execution after the specified number of tests have failed,
      to avoid using up resources (and time) in case of massive
      failures. retries are noe counted. Defaults to 10, set to 0 for
      no limit. May also be set with the environment variable
      <b>MTR_MAX_TEST_FAIL</b>.
    </li>

    <li>
      <tt>`--mem`</tt>

      This option is not supported on Windows and on MacOS.

      Run the test suite in memory, using tmpfs or ramdisk. This can
      decrease test times significantly, in particular if you would
      otherwise be running over a remote file system. @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" attempts to find a
      suitable location using a built-in list of standard locations
      for tmpfs and puts the <b>var</b> directory there. This option
      also affects placement of temporary files, which are created in
      <b>var/tmp</b>.

      The default list includes <b>/dev/shm</b>, <b>/run/shm</b> and
      <b>/tmp</b>. You can also enable this option by setting the
      environment variable <b>MTR_MEM[=<em>dir_name</em>]</b>. If
      <b><em>dir_name</em></b> is given, it is added to the beginning
      of the list of locations to search, so it takes precedence over
      any built-in locations.

      Once you have run tests with <b>`--mem`</b> within a
      <b>mysql-test</b> directory, a soflink <b>var</b> will have been
      set up to the temporary directory, and this will be re-used the
      next time, until the soflink is deleted. Thus, you do not have
      to repeat the <b>`--mem`</b> option next time.
    </li>

    <li>
      <tt>`--mysqld`=<b>value</b></tt>

      Extra option to pass to <b>mysqld</b>. Only one option may be
      specified in <b>value</b>; to specify more than one, use
      additional <b>--mysqld</b> options. See @ref PAGE_PASSING_OPTIONS.
    </li>

    <li>
      <tt>`--mysqld-env`=<b>variable</b>=<b>value</b></tt>

      Sets (or changes) an environment variable before starting
      <b>mysqld</b>. Varibles set in the environment from which you
      run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will
      normally also be propagated to <b>mysqld</b>, but there may be
      cases where you want a setting just for a single run, or you may
      not want the setting to affect other programs. You may use
      additional <b>`--mysqld-env`</b> options to set more than one
      variable.
    </li>

    <li>
      <tt>`--mysqltest`=<b>options</b></tt>

      Extra options to pass to @ref PAGE_MYSQLTEST "mysqltest".
    </li>

    <li>
      <tt>`--ndb-connectstring`=<b>str</b></tt>

      Pass <b>`--ndb-connectstring`=<em>str</em></b> to the master
      MySQL server. This option also prevents @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" from starting a
      cluster. It is assumed that there is already a cluster running
      to which the server can connect with the given connectstring.
    </li>

    <li>
      <tt>`--no-skip`</tt>

      This option forces all tests to run, ignoring any
      <b>`--skip`</b> commands used in the test. This ensures that all
      tests are run. An excluded list (<b>excludenoskip.list</b>) is
      maintained to track which tests should continue to be skipped.
      The <b>`--no-skip`</b> option continues to skip the tests that
      are named in the excluded list. In case the test or file invoking
      the <b>`--skip`</b> command is not present in the excluded list, the
      outcome of the test will be identified as <b>[ noskip-pass ]</b> or
      <b>[ noskip-fail ]</b> .The default value of <b>--no-skip</b>
      introduced variable is OFF, which implies users are not forced to
      run all tests unless the <b>--no-skip</b> is explicitly used.

      @verbatim
  shell> mysql-test-run.pl --suite=innodb --no-skip
      @endverbatim
    </li>

    <li>
      <tt>`--nocheck-testcases`</tt>

      Disable the check for test case side effects. For additional
      information, see the description of the
      <b>`--check-testcases`</b> option.
    </li>

    <li>
      <tt>`--noreorder`</tt>

      Do not reorder tests to reduce number of restarts, but run them
      in exactly the order given. If a whole suite is to be run, the
      tests are run in alphabetic order, though similiar combinations
      will be grouped together. If more than one suite is listed, the
      tests are run one suite at a time, in the order listed.
    </li>

    <li>
      <tt>`--notimer`</tt>

      Cause @ref PAGE_MYSQLTEST "mysqltest" not to generate a timing
      file. The effect of this is that the report from each test case
      does not include the timing in milliseconds as it normally does.
    </li>

    <li>
      <tt>`--nounit-tests`</tt>

      Do not run unit tests, overriding default behavior or setting of
      the <b>MTR_UNIT_TESTS</b> variable.
    </li>

    <li>
      <tt>`--nowarnings`</tt>

      Do not look for and report errors and warning in the server logs.

      @note
      In addition to being passed as a command-line option,
      <b>`--nowarnings`</b> can also be passed in the <b>.opt</b> file of
      a test case.
    </li>

    <li>
      <tt>`--only-big-tests`</tt>

      This option causes only big tests to run. Normal (non-big) tests
      are skipped. If both <b>`--big-test`</b> and
      <b>`--only-big-tests`</b> are given, <b>`--only-big-tests`</b>
      is ignored.
    </li>

    <li>
      <tt>`--opt-trace-protocol`</tt>

      Prints the @ref PAGE_OPT_TRACE "optimizer trace" of SQL statements in a test.
      Running any MTR test with this option executes the following SQL
      statement after every <b>DML statement</b> that it encounters:

      @verbatim
  SELECT trace FROM information_schema.optimizer_trace /* injected by --opt-trace-protocol */;
      @endverbatim
    </li>

    <li>
      <tt>`--parallel`={<b>N</b>|auto}</tt>

      Run tests using <b><em>N</em></b> parallel threads. By default,
      1 thread is used. Use <b>`--parallel`=auto</b> to set
      <b><em>N</em></b> automatically.

      Setting the <b>MTR_PARALLEL</b> environment variable to
      <b><em>N</em></b> has the same effect as specifying
      <b>`--parallel`=<em>N</em></b>.

      The <b>MTR_MAX_PARALLEL</b> environment variable, if set,
      specifies the maximum number of parallel workers that can be
      spawned when the <b>`--parallel`=auto</b> option is specified.
      If <b>`--parallel`=auto</b> is not specified,
      <b>MTR_MAX_PARALLEL</b> variable has no effect.
    </li>

    <li>
      <tt>`--port-base`=<b>P</b></tt>

      Specify base of port numbers to be used; a block of 10 will be
      allocated. <b><em>P</em></b> should be divisible by 10; if it is
      not, it will be rounded down. If running with more than one
      parallel test thread, thread 2 will use the next block of 10 and
      so on.

      If the port number is given as <b>auto</b>, which is also the
      default, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will
      allocate a number unique to this host. The value may also be
      given with the environment variable <b>MTR_PORT_BASE</b>.

      <b>`--port-base`</b> was added as a more logical alternative to
      <b>`--build-thread`</b>. If both are used, <b>`--port-base`</b>
      takes precedence.
    </li>

    <li>
      <tt>`--print-testcases`</tt>

      Do not run any tests, but print details about all tests, in the
      order they would have been run.
    </li>

    <li>
      <tt>`--ps-protocol`</tt>

      Pass the <b>`--ps-protocol`</b> option to @ref PAGE_MYSQLTEST
      "mysqltest".
    </li>

    <li>
      <tt>`--quiet`</tt>

      Reuse the output buffer and maintain a single line for reporting
      successful tests, skipped tests and disabled tests. Failed tests
      and the necessary information about the failure will be printed
      on a separate line.

      When <b>`--quiet`</b> option is enabled, MTR prints only the
      test name and result of the test run.
    </li>

    <li>
      <tt>`--record`</tt>

      Pass the <b>`--record`</b> option to @ref PAGE_MYSQLTEST
      "mysqltest". This option requires a specific test case to be
      named on the command line.
    </li>

    <li>
      <tt>`--reorder`</tt>

      Reorder tests to minimize the number of server restarts needed.
      This is the default behavior. There is no guarantee that a
      particular set of tests will always end up in the same order.
    </li>

    <li>
      <tt>`--repeat`=<b>N</b></tt>

      Run each test <b><em>N</em></b> number of times, in parallel if
      <b>`--parallel`</b> option value is greater than 1.
    </li>

    <li>
      <tt>`--report-features`</tt>

      Display the output of <b>SHOW ENGINES</b> and <b>SHOW
      VARIABLES</b>. This can be used to verify that binaries are
      built with all required features.
    </li>

    <li>
      <tt>`--report-times`</tt>

      At the end of the test run, write a summary of how much time was
      spent in various phases of execution. If you run with
      <b>`--parallel`</b>, the total will exceed the wall clock time
      passed, since it will be summed over all threads.

      The times reported should only be treated as approximations, and
      the exact points where the time is taken may also change between
      releases. If the test run is aborted, including if a test fails
      and <b>`--force`</b> is not in use, the time report will not
      be produced.
    </li>

    <li>
      <tt>`--report-unstable-tests`</tt>

      The option can be used to distinguish unstable tests which pass
      on at least one retry attempt from hard failing ones. Unstable
      tests are reported separately in the end summary in the format:
      <b>test_name(num_of_failures/num_of_attempts)</b>.

      In case all failures encountered are due to unstable tests, MTR
      will print the below warning and exit with a zero status code.

      @verbatim
  mysql-test-run: WARNING: There are failures due to unstable test cases.
  However, the tests are not hard-failing.
      @endverbatim

      This option has no effect unless <b>`--force`</b> is used and
      both <b>`--retry`</b> and <b>`--retry-failure`</b> are set to
      values greater than 1.
    </li>

    <li>
      <tt>`--retry`=<b>N</b></tt>

      If a test fails, it is retried up to a maximum of
      <b><em>N</em></b> runs, but will terminate after 2 failures.
      Default is 3, set to 1 or 0 for no retries. This option has no
      effect unless <b>`--force`</b> is also used; without it, test
      execution will terminate after the first failure.

      The <b>`--retry`</b> and <b>`--retry-failure`</b> options do not
      affect how many times a test repeated with <b>`--repeat`</b> may
      fail in total, as each repetition is considered a new test case,
      which may in turn be retried if it fails.
    </li>

    <li>
      <tt>`--retry-failure`=<b>N</b></tt>

      Allow a failed and retried test to fail more than the default 2
      times before giving it up. Setting it to 0 or 1 effectively
      turns off retries
    </li>

    <li>
      <tt>`--sanitize`</tt>

      Scan the server log files for warnings from various sanitizers.
      Use of this option assumes that MySQL was configured with
      [-DWITH_ASAN]
      (https://dev.mysql.com/doc/refman/8.0/en/source-configuration-options.html#option_cmake_with_asan)
      or [-DWITH_UBSAN]
      (https://dev.mysql.com/doc/refman/8.0/en/source-configuration-options.html#option_cmake_with_ubsan).

      The <b>TSAN_OPTIONS</b> environment variable can be set to
      specify the path name of a file containing ThreadSanitizer
      suppressions.
    </li>

    <li>
      <tt>`--shutdown-timeout`=<b>seconds</b></tt>

      Max number of seconds to wait for servers to do controlled
      shutdown before killing them. Default is 10.
    </li>

    <li>
      <tt>`--skip-combinations`</tt>

      Do not apply combinations; ignore combinations file or option.
    </li>

    <li>
      <tt>`--skip-ndbcluster`, `--skip-ndb`</tt>

      Do not start NDB Cluster; skip Cluster test cases. This option
      only has effect if you do have NDB, if not it will have no
      effect as it cannot run those tests anyway.
    </li>

    <li>
      <tt>`--skip-ndbcluster-slave`, `--skip-ndb-slave`</tt>

      Do not start an NDB Cluster slave.
    </li>

    <li>
      <tt>`--skip-rpl`</tt>

      Skip replication test cases.
    </li>

    <li>
      <tt>`--skip-test`=<b>regex</b></tt>

      Specify a regular expression to be applied to test case names.
      Cases with names that match the expression are skipped. tests to
      skip.

      The argument for the <b>`--skip-test`</b> option allows more
      flexible specification of which tests to skip. If the argument
      contains a pattern metacharacter other than a lone period, it is
      interpreted as a Perl regular expression and applies to test
      names that match the pattern. See the description of the
      <b>`--do-test`</b> option for details.
    </li>

    <li>
      <tt>`--skip-test-list`=<b>file</b></tt>

      Specify a file listing tests that should be skipped (disabled).

      The file has the same format as the <b>disabled.def</b> file
      listing disabled tests. With this option, disabling can be done
      on a case by case basis.
    </li>

    <li>
      <tt>`--skip-*`</tt>

      <b>`--skip-*`</b> options not otherwise recognized by @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" are passed to the
      master server.
    </li>

    <li>
      <tt>`--sp-protocol`</tt>

      Pass the <b>`--sp-protocol`</b> option to @ref PAGE_MYSQLTEST
      "mysqltest".
    </li>

    <li>
      <tt>`--start`</tt>

      Initialize and start servers with the startup settings for the
      specified test case. You can use this option to start a server
      to which you can connect later. For example, after building a
      source distribution you can start a server and connect to it
      with the <b>mysql</b> client like this:

      @verbatim
  shell> cd mysql-test
  shell> ./mysql-test-run.pl --start alias &
  shell> ../mysql -S ./var/tmp/master.sock -h localhost -u root
      @endverbatim

      If no tests are named on the command line, the server(s) will be
      started with settings for the first test that would have been
      run without the <b>`--start`</b> option.

      @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will stop once
      the server has been started, but will terminate if the server
      dies. If killed, it will also shut down the server.
    </li>

    <li>
      <tt>`--start-and-exit`</tt>

      This is similar to <b>`--start`</b>, but @ref
      PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" terminates once the
      server has been started, leaving just the server process
      running.
    </li>

    <li>
      <tt>`--start-dirty`</tt>

      This is similar to <b>`--start`</b>, but will skip the
      database initialization phase and assume that database files are
      already available. Usually this means you must have run another
      test first.
    </li>

    <li>
      <tt>`--start-from`=<b>test_name</b></tt>

      @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" sorts the list
      of names of the test cases to be run, and then begins with
     <b><em>test_name</em></b>.
    </li>

    <li>
      <tt>`--strace-client`</tt>

      Create <b>strace</b> output for @ref PAGE_MYSQLTEST "mysqltest".
      Will produce default <b>strace</b> output as
      <b>mysqltest.strace</b>. Note that this will be overwritten
      for each new test case, so it's most useful for running only one
      test.

      The <b>strace-client</b> option is only supported on Linux.
    </li>

    <li>
      <tt>`--strace-server`</tt>

      Create <b>strace</b> output for the server. Will produce default
      <b>strace</b> output as <b>mysqld.1.strace</b>. Note that this
      will be overwritten each time the server is restarted, so it's
      most useful for running a single test, or if you want trace from
      the first test that fails.

      The <b>strace-server</b> option is available on Linux only.
    </li>

    <li>
      <tt>`--stress`=<b>stress options</b></tt>

      Start a server, but instead of running a test, run @ref
      PAGE_MYSQLTEST "mysql-stress-test.pl" with the supplied
      arguments. Arguments needed to communicate with the server will
      be automatically provided, the rest should be given as arguments
      to this option. Command line options for @ref PAGE_MYSQLTEST
      "mysql-stress-test.pl" should be separeted by a comma.
    </li>

    <li>
      <tt>`--suite(s)`=<b>{suite_name|suite_list|suite_set}</b></tt>

      Run a suite or a comma separated list of suites or those suites
      which fall under the specified suite set. If the option is not
      specified explicitly, MTR assumes that it needs to use the
      default suite set for the test run.

      The suite set's MTR can use are case insensitive and take the
      below values:
      @verbatim
  default     - To run the default list of suites.
  all         - To scan the mysql directory and run all available suites.
  non-default - To scan the mysql directory for available suites and
                collect only the non-default suites.
      @endverbatim

      The suite set 'all' is used implicitly if neither a suite nor a
      suite list is specified and\n
      (i)  There are tests given on the MTR command line <b>or</b>\n
      (ii) Any of the following options are used: <b>--do-suite</b>,
      <b>--do-test</b>, <b>--do-test-list</b>.

      Note that suite sets 'all' and 'non-default' do not include ndb test
      suites in case ndb cluster is not enabled.

      If a suite set is specified in the suite list, MTR does not treat it
      as a suite set and considers it as the name of a suite to be run.
    </li>

    <li>
      <tt>`--suite-timeout`=<b>minutes</b></tt>

      Specify the maximum test suite runtime in minutes.
    </li>

    <li>
      <tt>`--summary-report`=<b>file_name</b></tt>

      Generate a plain text version of the test summary only and write
      it to the file named as the option argument. The file is
      suitable for sending by email.
    </li>

    <li>
      <tt>`--test-progress`[={0|1}]</tt>

      Print the percentage of tests completed. This option is enabled
      by default. The argument to <b>`--test-progress`</b> is
      optional. To disable it, set the value to <b>0</b>.

      For example, consider a test suite <b>sample_test_suite</b>
      containing 3 tests.

      @verbatim
  ./mysql-test-run.pl --suite=sample_test_suite [--test-progress]
      @endverbatim

      Running the suite with <b>`--test-progress`</b> option enabled
      will print the percentage of tests completed.

      @verbatim
  [ 33%] sample_test_suite.1st                           [ pass ]      1
  [ 66%] sample_test_suite.2nd                           [ pass ]      1
  [100%] sample_test_suite.3rd                           [ pass ]      1
      @endverbatim
    </li>

    <li>
      <tt>`--testcase-timeout`=<b>minutes</b></tt>

      Specify the maximum test case runtime in minutes.

      @note
      When a test case is run with <b>valgrind</b> option enabled, the
      test case timeout value becomes 10 times the specified value.
      For example, running a test case with timeout value
      <b>1 minute</b> and <b>valgrind</b> enabled will increase the
      timeout value to <b>10 minutes</b>.
    </li>

    <li>
      <tt>`--timediff`</tt>

      Adds to each test report for a test case, the total time in
      seconds and milliseconds passed since the preceding test ended.
      This option can only be used together with
      <b>`--timestamp`</b>, and has no effect without it.
    </li>

    <li>
      <tt>`--timer`</tt>

      Cause @ref PAGE_MYSQLTEST "mysqltest" to generate a timing file.
      The default file is named <b>./var/log/timer</b>.
    </li>

    <li>
      <tt>`--timestamp`</tt>

      Prints a timestamp before the test case name in each test report
      line, showing when the test ended.
    </li>

    <li>
      <tt>`--tmpdir`=<b>path</b></tt>

      The directory where temporary file are stored. The default
      location is <b>./var/tmp</b>. The environment variable
      <b>MYSQL_TMP_DIR</b> will be set to the path for this directory,
      whether it has the default value or has been set explicitly.
      This may be referred to in tests.
    </li>

    <li>
      <tt>`--unit-tests`</tt>

      Force running of unit tests, overriding default behavior or
      setting of the <b>MTR_UNIT_TESTS</b> variable.
    </li>

    <li>
      <tt>`--unit-tests-report`</tt>

      Extend the unit test run by also outputting the log from the
      test run, independently of whether it succeeded or not. This
      option implies <b>`--unit-tests`</b> so it is not necessary to
      specify both.
    </li>

    <li>
      <tt>`--user`=<b>user_name</b></tt>

      The MySQL user name to use when connecting to the server.
    </li>

    <li>
      <tt>`--user-args`</tt>

      Drops all non-essential command line arguments to the
      <b>mysqld</b> server, except those supplied with
      <b>`--mysqld`</b> arguemnts, if any. Only works in combination
      with <b>`--start`</b>, <b>`--start-and-exit`</b> or
      <b>`--start-dirty`</b>, and only if no test name is given.
    </li>

    <li>
      <tt>`--valgrind`</tt>

      Run @ref PAGE_MYSQLTEST "mysqltest" and <b>mysqld</b> with
      <b>valgrind</b>. This and the following <b>`--valgrind`</b>
      options require that the executables have been build with
      <b>valgrind</b> support.

      When the server is run with valgrind, an extra pass over the
      server log file(s) will be performed after all tests are run,
      and any report with problems that have been reported at server
      shutdown will be extracted and printed. The most common warnings
      are memory leaks. With each report will also be listed all tests
      that were run since previous server restart; one of these is
      likely to have caused the problem.

      A final "pseudo" test named <b>valgrind_report</b> is added to
      the list of tests when the server is run in valgrind. This test
      is reported as failed if any such shutdown warnings were
      produced by valgrind. Pass or failure of this test is also added
      to the total test count reported.

      @note
      When a test case is run with <b>valgrind</b> option enabled, the
      test case timeout value becomes 10 times the specified value.
      For example, running a test case with timeout value
      <b>1 minute</b> and <b>valgrind</b> enabled will increase the
      timeout value to <b>10 minutes</b>.
    </li>

    <li>
      <tt>`--valgrind-clients`</tt>

      Run all clients started by <b>.test</b> files with
      <b>valgrind</b>. This option requires <b>valgrind 3.9</b> or
      later.
    </li>

    <li>
      <tt>`--valgrind-mysqld`</tt>

      Run the <b>mysqld</b> server with <b>valgrind</b>.
    </li>

    <li>
      <tt>`--valgrind-mysqltest`</tt>

      Run @ref PAGE_MYSQLTEST "mysqltest" with <b>valgrind</b>.
    </li>

    <li>
      <tt>`--valgrind-option`=<b>str</b></tt>

      Extra options to pass to <b>valgrind</b>.
    </li>

    <li>
      <tt>`--valgrind-path`=<b>path</b></tt>

      Specify the path name to the valgrind executable.
    </li>

    <li>
      <tt>`--vardir`=<b>path</b></tt>

      Specify the path where files generated during the test run are
      stored. The default location is <b>./var</b>. The environment
      variable <b>MYSQLTEST_VARDIR</b> will be set to the path for
      this directory, whether it has the default value or has been set
      explicitly. This may be referred to in tests.
    </li>

    <li>
      <tt>`--verbose`</tt>

      Give more verbose output regarding test execution. Note that the
      output generated within each test case is not affected.
    </li>

    <li>
      <tt>`--verbose-restart`</tt>

      Write when and why servers are restarted between test cases.
    </li>

    <li>
      <tt>`--view-protocol`</tt>

      Pass the <b>`--view-protocol`</b> option to @ref
      PAGE_MYSQLTEST "mysqltest".
    </li>

    <li>
      <tt>`--vs-config`=<b>config_val</b></tt>

      Specify the configuration used to build MySQL (for example,
      <b>--vs-config=debug --vs-config=release</b>). This option
      is for Windows only.
    </li>

    <li>
      <tt>`--wait-all`</tt>

      If <b>`--start`</b> or <b>`--start-dirty`</b> is used, wait
      for all servers to exit before termination. Otherise, it will
      terminate if one (of several) servers is restarted.
    </li>

    <li>
      <tt>`--warnings`</tt>

      Search the server log for errors or warning after each test and
      report any suspicious ones; if any are found, the test will be
      marked as failed. This is the default behavior, it may be turned
      off with <b>`--nowarnings`</b>.
    </li>

    <li>
      <tt>`--with-ndbcluster-only`, `--with-ndb-only`</tt>

      Run only <b>ndb</b> tests. If no suites are explicitly given, this
      option also skip all non <b>ndb</b> suites without checking
      individual test names.

    </li>

    <li>
      <tt>`--xml-report`=<b>file_name</b></tt>

      Generate an xml file containing result of the test run and write
      it to the file named as the option argument. It includes
      suite-wise and overall statistics like the number of tests run,
      number of failures, number of disabled and skipped tests along
      with information about each individual test according to its
      outcome.

      The xml report's structure can be found below :

      @verbatim
  <testsuites ... >
    <testsuite ... >
       <testcase ... />
       <testcase ... >
           <failure ... > ... </failure>
       </testcase>
    </testsuite>
  </testsuites>
      @endverbatim
    </li>
  </ul>

  @note
  The hostname resolves to 127.0.0.1 and not to the actual IP address.
*/

###############################################################################



## mysql-stress-test.pl — Server Stress Test Program

/**
  @page PAGE_MYSQL_STRESS_TEST_PL mysql-stress-test.pl — Server Stress Test Program

  The @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" Perl
  script performs stress-testing of the MySQL server.

  @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" requires
  version of Perl that has been built with threads support.

  Invoke @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" like
  this:

  @verbatim
  shell> mysql-stress-test.pl [options]
  @endverbatim

  @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" supports the
  following options:

  <ul>
    <li>
      <tt>`--help`</tt>

      Display a help message and exit.
    </li>

    <li>
      <tt>`--abort-on-erro`r=<b>N</b></tt>

      Causes the program to abort if an error with severity less than
      or equal to N was encountered. Set to 1 to abort on any error.
    </li>

    <li>
      <tt>`--check-tests-file`</tt>

      Periodically check the file that lists the tests to be run. If
      it has been modified, reread the file. This can be useful if you
      update the list of tests to be run during a stress test.
    </li>

    <li>
      <tt>`--cleanup`</tt>

      Force cleanup of the working directory.
    </li>

    <li>
      <tt>`--log-error-details`</tt>

      Log error details in the global error log file.
    </li>

    <li>
      <tt>`--loop-count`=<b>N</b></tt>

      In sequential test mode, the number of loops to execute before
      exiting.
    </li>

    <li>
      <tt>`--mysqltest`=<b>path</b></tt>

      The path name to the @ref PAGE_MYSQLTEST "mysqltest" program.
    </li>

    <li>
      <tt>`--server-database`=<b>db_name</b></tt>

      The database to use for the tests. The default is <b>test</b>.
    </li>

    <li>
      <tt>`--server-host`=<b>host_name</b></tt>

      The host name of the local host to use for making a TCP/IP
      connection to the local server. By default, the connection is
      made to <b>localhost</b> using a Unix socket file.
    </li>

    <li>
      <tt>`--server-logs-dir`=<b>path</b></tt>

      This option is required. path is the directory where all client
      session logs will be stored. Usually this is the shared
      directory that is associated with the server used for testing.
    </li>

    <li>
      <tt>`--server-password`=<b>password</b></tt>

      The password to use when connecting to the server.
    </li>

    <li>
      <tt>`--server-port`=<b>port_num</b></tt>

      The TCP/IP port number to use for connecting to the server.
      The default is <b>3306</b>.
    </li>

    <li>
      <tt>`--server-socket`=<b>file_name</b></tt>

      For connections to <b>localhost</b>, the Unix socket file to
      use, or, on Windows, the name of the named pipe to use. The
      default if <b>/tmp/mysql.sock</b>.
    </li>

    <li>
      <tt>`--server-user`=<b>user_name</b></tt>

      The MySQL user name to use when connecting to the server. The
      default is <b>root</b>.
    </li>

    <li>
      <tt>`--sleep-time`=<b>N</b></tt>

      The delay in seconds between test executions.
    </li>

    <li>
      <tt>`--stress-basedir`=<b>path</b></tt>

      This option is required. <b>path</b> is the working
      directory for the test run. It is used as the temporary location
      for result tracking during testing.
    </li>

    <li>
      <tt>`--stress-datadir`=<b>path</b></tt>

      The directory of data files to be used during testing. The
      default location is the <b>data</b> directory under the
      location given by the <b>`--stress-suite-basedir`</b> option.
    </li>

    <li>
      <tt>`--stress-init-file`[=<b>path</b>]</tt>

      <b>file_name</b> is the location of the file that
      contains the list of tests to be run once to initialize the
      database for the testing. If missing, the default file is
      <b>stress_init.txt</b> in the test suite directory.
    </li>

    <li>
      <tt>`--stress-mode`=<b>mode</b></tt>

      This option indicates the test order in stress-test mode. The
      <b>mode</b> value is either <b>random</b> to select tests in
      random order or <b>seq</b> to run tests in each thread in the
      order specified in the test list file. The default mode is
      <b>random</b>.
    </li>

    <li>
      <tt>`--stress-suite-basedir`=<b>path</b></tt>

      This option is required. <b>path</b> is the directory
      that has the <b>t</b> and <b>r</b> subdirectories containing
      the test case and result files. This directory is also the
      default location of the <b>stress-test.txt</b> file that
      contains the list of tests. (A different location can be
      specified with the <b>`--stress-tests-file`</b> option).
    </li>

    <li>
      <tt>`--stress-tests-file`[=<b>file_name</b>]</tt>

      Use this option to run the stress tests. <b>file_name</b>
      is the location of the file that contains the list of tests. If
      <b>file_name</b> is omitted, the default file is
      <b>stress-test.txt</b> in the stress suite directory. (See
      <b>`--stress-suite-basedir`</b>).
    </li>

    <li>
      <tt>`--suite`=<b>suite_name</b></tt>

      Run the named test suite. The default name is <b>main</b> (the
      regular test suite located in the mysql-test directory).
    </li>

    <li>
      <tt>`--test-count`=<b>N</b></tt>

      The number of tests to execute before exiting.
    </li>

    <li>
      <tt>`--test-duration`=<b>N</b></tt>

      The duration of stress testing in seconds.
    </li>

    <li>
      <tt>`--threads`=<b>N</b></tt>

      The number of threads. The default is 1.
    </li>

    <li>
      <tt>`--verbose`</tt>

      Verbose mode. Print more information about what the program does.
    </li>
  </ul>
*/

###############################################################################



## mysqltest Language Reference

/**
  @page PAGE_MYSQLTEST_LANGUAGE_REFERENCE mysqltest Language Reference

  <h3>Table of Contents</h3>
  - @subpage PAGE_MYSQL_TEST_INPUT_CONVENTIONS
  - @subpage PAGE_MYSQL_TEST_COMMANDS
  - @subpage PAGE_MYSQL_TEST_VARIABLES
  - @subpage PAGE_MYSQL_TEST_FLOW_CONTROL
  - @subpage PAGE_ERROR_HANDLING

  This chapter describes the test language implemented by @ref
  PAGE_MYSQLTEST "mysqltest". The language allows input to contain a
  mix of comments, commands executed by @ref PAGE_MYSQLTEST
  "mysqltest" itself, and SQL statements that @ref PAGE_MYSQLTEST
  "mysqltest" sends to a MySQL server for execution.

  Terminology notes:
  ------------------

  <ul>
    <li>
      A “command” is an input test that @ref PAGE_MYSQLTEST
      "mysqltest" recognizes and executes itself. A “statement” is
      an SQL statement or query that @ref PAGE_MYSQLTEST "mysqltest"
      sends to the MySQL server to be executed.
    </li>

    <li>
      When @ref PAGE_MYSQLTEST "mysqltest" starts, it opens a
      connection it calls <b>default</b> to the MySQL server, using
      any connection parameters specified by the command options. (For
      a local server, the default user name is <b>root</b>. For an
      external server, the default user name is <b>test</b> or the
      user specified with the <b>`--user`</b> option.) You can use
      the <b>connect</b> command to open other connections, the
      <b>connection</b> command to switch between connections, and
      the <b>disconnect</b> command to close connections. However,
      the capability for switching connections means that the
      connection named <b>default</b> need not be the connection in
      use at a given time. To avoid ambiguity, this document avoids
      the term “default connection”. It uses the term “current
      connection” to mean “the connection currently in use”, which
      might be different from “the connection named <b>default</b>”.
    </li>
  </ul>
*/

###############################################################################



## mysqltest Input Conventions

/**
  @page PAGE_MYSQL_TEST_INPUT_CONVENTIONS mysqltest Input Conventions

  @ref PAGE_MYSQLTEST "mysqltest" reads input lines and processes
  them as follows:

  <ul>
    <li>
      “End of line” means a newline (linefeed) character. A carriage
      return/linefeed (CRLF) pair also is allowable as as a line
      terminator (the carriage return is ignored). Carriage return by
      itself is not allowed as a line terminator.
    </li>

    <li>
      A line that begins with "#" as the first nonwhitespace content
      is treated as a comment that extends to the end of the line and
      is ignored. Example:

      @verbatim
  # this is a comment
      @endverbatim
    </li>

    <li>
      Earlier versions would also allow comments beginning with “`--`”
      unless the first word was a valid @ref PAGE_MYSQLTEST
      "mysqltest" command, but this has been deprecated and is longer
      allowed.
    </li>

    <li>
      Other input is taken as normal command input. The command
      extends to the next occurrence of the command delimiter, which
      is semicolon (“<b>;</b>”) by default. The delimiter can be
      changed with the <b>delimiter</b> command.

      If @ref PAGE_MYSQLTEST "mysqltest" recognizes the first word of
      the delimiter-terminated command, @ref PAGE_MYSQLTEST
      "mysqltest" executes the command itself. Otherwise, @ref
      PAGE_MYSQLTEST "mysqltest" assumes that the command is an SQL
      statement and sends it to the MySQL server to be executed.

      Because the command extends to the delimiter, a given input line
      can contain multiple commands, and a given command can span
      multiple lines. The ability to write multiple-line statements is
      useful for making long statements more readable, such as a
      <b>create table</b> statement for a table that has many
      columns.
    </li>
  </ul>

  After @ref PAGE_MYSQLTEST "mysqltest" reads a command up to a
  delimiter and executes it, input reading restarts following the
  delimiter and any remaining input on the line that contains the
  delimiter is treated as though it begins on a new line. Consider the
  following two input lines:

  @verbatim
  echo issue a select statement; select 1; echo done
  issuing the select statement;
  @endverbatim

  That input contains two commands and one SQL statement:

  @verbatim
  echo issue a SELECT statement
  SELECT 1;
  echo done issuing the SELECT statement
  @endverbatim

  Similarly, "#" comments can begin on a command line following
  a delimiter:

  @verbatim
  SELECT 'hello'; # select a string value
  @endverbatim

  On a multiple-line command, "#" or “`--`” at the beginning of the
  second or following lines is not special. Thus, the second and third
  lines of the following variable-assignment command are not taken as
  comments. Instead, the variable <b>$a</b> is set to a value that
  contains two linefeed characters:

  @verbatim
  let $a= This is a variable
  # assignment that sets a variable
  -- to a multiple-line value;
  @endverbatim

  `--` commands and normal commands have complementary properties with
  regard to how @ref PAGE_MYSQLTEST "mysqltest" reads them:

  - A “`--`” command is terminated by a newline, regardless of how
    many delimiters it contains.

  - A normal command (without “`--`”) is terminated by the delimiter
    (semicolon), no matter how many newlines it contains.

  @ref PAGE_MYSQLTEST "mysqltest" commands can be written either with
  a leading “`--`”) or as normal command input (no leading “`--`”).
  Use the command delimiter only in the latter case. Thus, these two
  lines are equivalent:

  @verbatim
  --sleep 2
  sleep 2;
  @endverbatim

  The equivalence is true even for the <b>delimiter</b> command. For
  example, to set the delimiter to “<b>//</b>”, either of these
  commands work:

  @verbatim
  --delimiter //
  delimiter //;
  @endverbatim

  To set the delimiter back to “;”, use either of these commands:

  @verbatim
  --delimiter ;
  delimiter ;//
  @endverbatim

  A potential ambiguity occurs because a command line can contain
  either a @ref PAGE_MYSQLTEST "mysqltest" command or an SQL
  statement. This has a couple of implications:

  - No @ref PAGE_MYSQLTEST "mysqltest" command should be the same
    as any keyword that can begin an SQL statement.

  - Should extensions to SQL be implemented in the future, it is
    possible that a new SQL keyword could be impossible for @ref
    PAGE_MYSQLTEST "mysqltest" to recognize as such if that keyword
    is already used as a @ref PAGE_MYSQLTEST "mysqltest" command.

  Any ambiguity can be resolved by using the “`--`” syntax to force
  interpetation as a @ref PAGE_MYSQLTEST "mysqltest" command, or the
  <b>query</b> command to force interpretation as SQL.

  All file paths used in test commands should use forward slash
  \"<b>/</b>\" as the directory separator as in Unix. They will be
  automatically converted when needed if the test is run on Windows.
  We also recommend putting all temporary or auxiliary files made
  during the test under the directory referred to by
  <b>$MYSQL_TMP_DIR</b>. Do not put them under fixed full paths like
  <b>/tmp</b>. This will help ensuring portability of the test, and
  avoiding conflicts with other programs.

  <b>$MYSQL_TMP_DIR</b> is equivalent to
  <b>$MYSQLTEST_VARDIR/tmp</b> if you are not running with parallel
  test threads, but if you run @ref PAGE_MYSQL_TEST_RUN_PL
  "mysql-test-run.pl" with <b>`--parallel`</b>, they will be different.
  It is therefore best to be consistent and use <b>$MYSQL_TMP_DIR</b>.

  Commands named <b>disable_X</b> or <b>enable_X</b>, except
  <b>parsing</b>, <b>reconnect</b> and <b>rpl_parse</b>, can take an
  optional modifier <b>ONCE</b>. If this is added, the relevant
  setting is enabled or disabled only for the next command or
  statement, after which it is reverted to whatever it was before.
  Note that the word <b>ONCE</b> must be in upper case; this was
  chosen in order to make it more visible when reading the test
  script.

  For example, <b>`--disable_query_log`</b> <b>ONCE</b> will
  ensure query log is disabled for the next statement, but will not
  affect whether or not query log is enabled for statements following
  the next. It is possible to enable/disable more that one property
  (e.g. both query log and result log) for a single statement using
  the <b>ONCE</b> modifier.
*/

###############################################################################



## mysqltest Commands

/**
  @page PAGE_MYSQL_TEST_COMMANDS mysqltest Commands

  @ref PAGE_MYSQLTEST "mysqltest" supports the commands described in
  this section. Command names are not case sensitive.

  Some examples of command use are given, but you can find many more
  by searching the test case files in the <b>mysql-test/t</b>
  directory.

  <ul>
    <li>
      <tt>append_file <b>file_name</b> [<b>terminator</b>]</tt>

      <b>append_file</b> is like <b>write_file</b> except that the
      lines up to the terminator are added to the end of the file.
      The file is created if it does not exist. The file name argument
      is subject to variable substitution.

      @verbatim
  write_file $MYSQL_TMP_DIR/data01;
  line one for the file
  line two for the file
  EOF
  append_file $MYSQL_TMP_DIR/data01;
  line three for the file
  EOF
      @endverbatim

      @verbatim
  write_file $MYSQL_TMP_DIR/data02 END_OF_FILE;
  line one for the file
  line two for the file
  END_OF_FILE
  append_file $MYSQL_TMP_DIR/data02 END_OF_FILE;
  line three for the file
  END_OF_FILE
      @endverbatim
    </li>

    <li>
      <tt>assert (<b>expr</b>)</tt>

      Assert that <b>expr</b> is true; otherwise terminate the test program
      with an error message indicating that <b>expr</b> failed.
      <b>expr</b> is parsed as it is for <tt>if()</tt> and <tt>while()</tt>,
      as described at @ref PAGE_MYSQL_TEST_FLOW_CONTROL.

      @verbatim
        assert(1);
        assert($x);
        assert(!$x);
        assert(`select count(*) from t1`);
      @endverbatim
    </li>

    <li>
      <tt>cat_file <b>file_name</b></tt>

      <b>cat_file</b> writes the contents of the file to the output.
      The file name argument is subject to variable substitution.

      @verbatim
  cat_file $MYSQL_TMP_DIR/data01;
      @endverbatim
    </li>

    <li>
      <tt>change_user [<b>user_name</b>], [<b>password</b>],
                      [<b>db_name</b>]</tt>

      Changes the current user and causes the database specified by
      <b><em>db_name</em></b> to become the default database for the
      current connection.

      @verbatim
  change_user root;
  --change_user root,,test
      @endverbatim
    </li>

    <li>
      <tt>character_set <b>charset_name</b></tt>

      Set the default character set to <b><em>charset_name</em></b>.
      Initially, the character set is latin1.

      @verbatim
  character_set utf8;
  --character_set sjis
      @endverbatim
    </li>

    <li>
      <tt>chmod <b>octal_mode file_name</b></tt>

      Change the mode of the given file. The file mode must be given as
      a four-digit octal number. The file name argument is subject to
      variable substitution, but must evaluate to a literal file name,
      not a file name pattern.

      @verbatim
  chmod 0644 $MYSQL_TMP_DIR/data_xxx01;
      @endverbatim
    </li>

    <li>
      <tt>connect (<b>name</b>, <b>host_name</b>,
                   <b>user_name</b>, <b>password</b>, <b>db_name</b>
                   [,<b>port_num</b> [,<b>socket</b> [,<b>options</b>
                   [,<b>default_auth</b> [,<b>compression algorithm</b>,
                   [,<b>compression level</b>]]]]]])</tt>

      Open a connection to the server and make the connection the
      current connection.

      The arguments to <b>connect</b> are:

      <ul>
        <li>
          <b><em>name</em></b> is the name for the connection (for use
          with the <b>connection</b>, <b>disconnect</b>, and
          <b>dirty_close</b> commands). This name must not already be
          in use by an open connection.
        </li>

        <li>
          <b><em>host_name</em></b> indicates the host where the
          server is running.
        </li>

        <li>
          <b><em>user_name</em></b> and <b><em>password</em></b> are
          the user name and password of the MySQL account to use.
        </li>

        <li>
          <b><em>db_name</em></b> is the default database to use. As a
          special case, *<b>NO-ONE</b>* means that no default database
          should be selected. You can also leave
          <b><em>db_name</em></b> blank to select no database.
        </li>

        <li>
          <b><em>port_num</em></b>, if given, is the TCP/IP port
          number to use for the connection. This parameter can be
          given by using a variable.
        </li>

        <li>
          <b><em>socket</em></b>, if given, is the socket file to use
          for connections to <b>localhost</b>. This parameter can be
          given by using a variable.
        </li>

        <li>
          <b><em>options</em></b> can be one or more of the following
          words, separated by spaces:

          <ul>
            <li>
              <tt>CLEARTEXT</tt>: Enable use of the cleartext authentication
              plugin.
            </li>

            <li>
              <tt>COMPRESS</tt>: Use the compressed client/server protocol,
              if available.
            </li>

            <li>
              <tt>PIPE</tt>: Use the named-pipe connection protocol, if
              available.
            </li>

            <li>
              <tt>SHM</tt>: Use the shared-memory connection protocol, if
              available.
            </li>

            <li>
              <tt>SOCKET</tt>: Use the socket-file connection protocol.
            </li>

            <li>
              <tt>SSL</tt>:Use SSL network protocol to have encrypted connection.
            </li>

            <li>
              <tt>TCP</tt>: Use the TCP/IP connection protocol.
            </li>

          </ul>

          Passing <b>PIPE</b> or <b>SHM</b> on non-Windows systems
          causes an error, and, similarly, passing <b>SOCKET</b> on
          Windows systems causes an error.
        </li>

        <li>
          <b><em>default_auth</em></b> is the name of an
          authentication plugin. It is passed to the
          <b>mysql_options()</b> C API function using the
          <b>MYSQL_DEFAULT_AUTH</b> option. If @ref PAGE_MYSQLTEST
          "mysqltest" does not find the plugin, use the
          <b>--plugin-dir</b> option to specify the directory where
          the plugin is located.
        </li>

        <li>
          <b><em>compression algorithm</em></b> is the name of compression
          algorithm to be used to compress data transferred between client
          server. It is passed to the <b>mysql_options()</b> C API function
          using the <b>MYSQL_OPT_COMPRESSION_ALGORITHMS</b> option.
        </li>

        <li>
          <b><em>zstd compression level</em></b> is the extent of compression
          to be applied when zstd compression algorithm is used. It is passed to
          the <b>mysql_options()</b> C API function using the
          <b>MYSQL_OPT_COMPRESSION_ALGORITHMS</b> option.
        </li>

        <li>
          <b><em>compression level</em></b> is the extent of compression to
          be applied based on the compression algorithm used. It is passed to
          the <b>mysql_options()</b> C API function using the
          <b>MYSQL_OPT_COMPRESSION_ALGORITHMS</b> option.
        </li>
      </ul>

      To omit an argument, just leave it blank. For an omitted
      argument, @ref PAGE_MYSQLTEST "mysqltest" uses an empty string
      for the first five arguments and the <b><em>options</em></b>
      argument. For omitted port or socket options, @ref
      PAGE_MYSQLTEST "mysqltest" uses the default port or socket.

      @verbatim
  connect (conn1,localhost,root,,);
  connect (conn2,localhost,root,mypass,test);
  connect (conn1,127.0.0.1,root,,test,$MASTER_MYPORT);
      @endverbatim

      The last example assumes that the <b>$MASTER_MYPORT</b> variable
      has already been set (perhaps as an environment variable).

      If a connection attempt fails initially, @ref PAGE_MYSQLTEST
      "mysqltest" retries five times if the abort-on-error setting is
      enabled.
    </li>

    <li>
      <tt>connection <b>connection_name</b></tt>

      Select <b><em>connection_name</em></b> as the current
      connection. To select the connection that @ref PAGE_MYSQLTEST
      "mysqltest" opens when it starts, use the name default.

      @verbatim
  connection master;
  connection conn2;
  connection default;
      @endverbatim

      A variable can be used to specify the
      <b><em>connection_name</em></b> value.
    </li>

    <li>
      <tt>let $var= convert_error(<b>error</b>)</tt>

      This is not a command as such but rather a function that can be
      used in <b>let</b> statements. If the argument is a number, it
      returns the name of the corresponding error, or <b>\<Unknown\></b>
      if no such error exists. If the argument is an error name, it
      returns the corresponding number, or fails if the error name is
      unknown. If the argument is 0 or an empty string, it returns 0.
      The function can also take a variable as argument.

      @verbatim
  let $errvar1=convert_error(ER_UNKNOWN_ERROR);
  let $errvar2=convert_error(1450);
  let $errvar3=convert_error($errvar1);
      @endverbatim
    </li>

    <li>
      <tt>copy_file <b>from_file to_file</b> [<b>retry</b>]</tt>

      Copy the file <b><em>from_file</em></b> to the file
      <b><em>to_file</em></b>. The command fails if
      <b><em>to_file</em></b> already exists. The file name arguments
      are subject to variable substitution.

      <b>copy_file</b> can also take an optional argument <b>retry</b>.
      If the command fails due to an environmental issue, the command
      can be retried for a <b>retry</b> number of times. Each retry
      happens after an interval of one second.

      @verbatim
  copy_file $MYSQL_TMP_DIR/copy1.txt $MYSQL_TMP_DIR/copy2.txt;
  copy_file $MYSQL_TMP_DIR/foo.txt $MYSQL_TMP_DIR/foo2.txt 5;
      @endverbatim
    </li>

    <li>
      <tt>copy_files_wildcard <b>src_dir_name dst_dir_name</b>
                              <b>pattern</b> [<b>retry</b>]</tt>

      Copy all files that match the pattern in the source directory to
      the destination directory. Patterns can use <b>?</b> to
      represent any single character, or <b>*</b> for any sequence of
      0 or more characters. The <b>.</b> character is treated like any
      other. The pattern may not include <b>/</b>. If all the files
      need to be copied, the <b>*</b> wildcard can be used.

      <b>copy_files_wildcard</b> can also take an optional argument
      <b>retry</b>. If the command fails due to an environmental
      issue, the command can be retried for a <b>retry</b> number
      of times. Each retry happens after an interval of one second.

      The command works like this:

      <ul>
        <li>
          Files that match the pattern are copied from the source
          directory to the destination directory. Overwriting of files
          is permitted.
        </li>

        <li>
          Copying does not apply to directories matching the pattern
          or matching files in subdirectories.
        </li>

        <li>
          If the source or destination directory is not present, an
          error occurs.
        </li>

        <li>
          If no files match the specified pattern, an error occurs.
        </li>

        <li>
          If the source directory has no files, an error occurs.
        </li>
      </ul>

      @verbatim
  copy_files_wildcard $MYSQLTEST_VARDIR/std_data/ $MYSQLTEST_VARDIR/copy1/ *.txt;
  copy_files_wildcard $MYSQLTEST_VARDIR/std_data/ $MYSQLTEST_VARDIR/copy1/ *.txt 5;
      @endverbatim
    </li>

    <li>
      <tt>dec <b>$var_name</b></tt>

      Decrement a numeric variable. If the variable does not have a
      numeric value, the result is undefined.

      @verbatim
  dec $count;
  dec $2;
      @endverbatim
    </li>

    <li>
      <tt>delimiter <b>str</b></tt>

      Set the command delimiter to <b><em>str</em></b>, which may consist
      of 1 to 15 characters. The default delimiter is the semicolon
      character (“<b>;</b>”).

      @verbatim
  delimiter /;
  --delimiter stop
      @endverbatim

      This is useful or needed when you want to include long SQL
      statements like <b>CREATE PROCEDURE</b> which include semicolon
      delimited statements but need to be interpreted as a single
      statement by @ref PAGE_MYSQLTEST "mysqltest". If you have set
      the delimiter to “<b>/</b>” as in the previous example, you can
      set it back to the default like this:

      @verbatim
  delimiter ;|
      @endverbatim
    </li>

    <li>
      <tt>die [<b>message</b>]</tt>

      Aborts the test with an error code after printing the given
      message as the reason. Suppose that a test file contains the
      following line:

      @verbatim
  die Cannot continue;
      @endverbatim

      When @ref PAGE_MYSQLTEST "mysqltest" encounters that line, it
      produces the following result and exits:

      @verbatim
  mysqltest: At line 1: Cannot continue
  not ok
      @endverbatim
    </li>

    <li>
      <tt>diff_files <b>file_name1 file_name2</b></tt>

      Compare the two files. The command succeeds if the files are the
      same, and fails if they are different or either file does not
      exist. The file name arguments are subject to variable
      substitution.
    </li>

    <li>
      <tt>dirty_close <b>connection_name</b></tt>

      Close the named connection. This is like <b>disconnect</b>
      except that it calls <b>vio_delete()</b> before it closes the
      connection. If the connection is the current connection, you
      should use the <b>connection</b> command to switch to a
      different connection before executing further SQL statements.

      A variable can be used to specify the
      <b><em>connection_name</em></b> value.
    </li>

    <li>
      <tt>disable_abort_on_error</tt>, <tt>enable_abort_on_error</tt>

      Disable or enable abort-on-error behavior. This setting is
      enabled by default. With this setting enabled, @ref
      PAGE_MYSQLTEST "mysqltest" aborts the test when a statement sent
      to the server results in an unexpected error, and does not
      generate the <b>.reject</b> file. For discussion of reasons why
      it can be useful to disable this behavior, see @ref
      PAGE_ERROR_HANDLING.

      @verbatim
  --disable_abort_on_error
  --enable_abort_on_error
      @endverbatim
    </li>

    <li>
      <tt>disable_async_client</tt>, <tt>enable_async_client</tt>

      Disable or enable asynchronous communication between the client and
      the server. With this setting enabled, @ref PAGE_MYSQLTEST "mysqltest"
      uses non-blocking C API's to communicate with the server.<br> Please
      note that <b>disable_async_client</b> disables asynchronous
      communication if it had been enabled either by the
      <b>--enable_async_client</b> command or the <b>--async-client</b>
      command line option.

      @verbatim
  --disable_async_client
  --enable_async_client
      @endverbatim
    </li>

    <li>
      <tt>disable_connect_log</tt>, <tt>enable_connect_log</tt>

      Disable or enable logging of creation or switch of connections.
      Connection logging is disabled by default. With this setting
      enabled, @ref PAGE_MYSQLTEST "mysqltest" enters lines in the
      test results to show when connections are created, switched or
      disconnected.

      If query logging is turned off using <b>disable_query_log</b>,
      connection logging is also turned off, until query log is re-
      enabled.

      @verbatim
  --disable_connect_log
  --enable_connect_log
      @endverbatim
    </li>

    <li>
      <tt>disable_info</tt>, <tt>enable_info</tt>

      Disable or enable additional information about SQL statement
      results. Information display is disabled by default. With this
      setting enabled, @ref PAGE_MYSQLTEST "mysqltest" displays the
      affected-rows count and the output from the <b>mysql_info()</b>
      C API function. The “affected-rows” value is “rows selected” for
      statements such as <b>SELECT</b> and “rows modified” for
      statements that change data.

      @verbatim
  --disable_info
  --enable_info
      @endverbatim
    </li>

    <li>
      <tt>disable_metadata</tt>, <tt>enable_metadata</tt>

      Disable or enable query metadata display. Metadata display is
      disabled by default. With this setting enabled, @ref
      PAGE_MYSQLTEST "mysqltest" adds query metadata to the result.
      This information consists of the values corresponding to the
      members of the <b>MYSQL_FIELD</b> C API data structure, for each
      column of the result.

      @verbatim
  --disable_metadata
  --enable_metadata
      @endverbatim
    </li>

    <li>
      <tt>disable_ps_protocol</tt>, <tt>enable_ps_protocol</tt>

      Disable or enable prepared-statement protocol. This setting is
      disabled by default unless the <b>`--ps-protocol`</b> option is
      given.

      @verbatim
  --disable_ps_protocol
  --enable_ps_protocol
      @endverbatim
    </li>

    <li>
      <tt>disable_query_log</tt>, <tt>enable_query_log</tt>

      Disable or enable query logging. This setting is enabled by
      default. With this setting enabled, @ref PAGE_MYSQLTEST
      "mysqltest" echoes input SQL statements to the test result.

      One reason to disable query logging is to reduce the amount of
      test output produced, which also makes comparison of actual and
      expected results more efficient.

      @verbatim
  --disable_query_log
  --enable_query_log
      @endverbatim
    </li>

    <li>
      <tt>disable_reconnect</tt>, <tt>enable_reconnect</tt>

      Disable or enable automatic reconnect for dropped connections.
      (The default depends on the client library version.) This
      command only applies to the current connection.

      @verbatim
  --disable_reconnect
  --enable_reconnect
      @endverbatim
    </li>

    <li>
      <tt>disable_result_log</tt>, <tt>enable_result_log</tt>

      Disable or enable the result log. This setting is enabled by
      default. With this setting enabled, @ref PAGE_MYSQLTEST
      "mysqltest" displays query results (and results from commands
      such as <b>echo</b> and <b>exec</b>).

      @verbatim
  --disable_result_log
  --enable_result_log
      @endverbatim
    </li>

    <li>
      <tt>disable_rpl_parse</tt>, <tt>enable_rpl_parse</tt>

      Disable or enable parsing of statements to determine whether
      they go to the master or slave. The default is whatever the
      default is for the C API library.

      @verbatim
  --disable_rpl_parse
  --enable_rpl_parse
      @endverbatim
    </li>

    <li>
      <tt>disable_session_track_info</tt>, <tt>enable_session_track_info</tt>

      Disable or enable display of session tracking information.
      Session-tracking display disabled by default.

      @verbatim
  --disable_session_track_info
  --enable_session_track_info
      @endverbatim
    </li>

    <li>
      <tt>disable_testcase bug_number</tt>, <tt>enable_testcase</tt>

      Disable or enable a section from a test case. This setting is
      enabled by default. When disabled, @ref PAGE_MYSQLTEST
      "mysqltest" ignores everything until <b>enable_testcase</b>.
      These commands are useful for disabling a section inside a
      <b>.test</b> file that fails due to a known bug, without having
      to disable the entire test case.

      Bug number is a mandatory argument to <b>disable_testcase</b>
      command and it should be in <b>BUG\#XXXX</b> format where keyword
      <b>BUG</b> is case-insensitive and <b>XXXX</b> should contain
      only digits.

      @verbatim
  --disable_testcase BUG#XXXX
  --enable_testcase
      @endverbatim
    </li>

    <li>
      <tt>disable_warnings</tt>, <tt>enable_warnings</tt>

      Disable or enable warnings. This setting is enabled by default.
      With this setting enabled, @ref PAGE_MYSQLTEST "mysqltest" uses
      <b>SHOW WARNINGS</b> to display any warnings produced by SQL
      statements.

      @verbatim
  --disable_warnings
  --enable_warnings
      @endverbatim

      <b>disable_warnings</b> and <b>enable_warnings</b> commands can
      take an optional argument specifying one or more comma-separated
      list of warnings to be disabled or enabled. Each warning
      specified should be in symbolic error name format.

      @verbatim
  --disable_warnings ER_BAD_TABLE_ERROR,ER_YES
  DROP TABLE IF EXISTS t1;
  --enable_warnings ER_BAD_TABLE_ERROR,ER_YES
      @endverbatim

      Or

      @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1;
  --enable_warnings ER_BAD_TABLE_ERROR,ER_YES
  DROP TABLE IF EXISTS t1;
  --disable_warnings ER_BAD_TABLE_ERROR,ER_YES
  --enable_warnings
      @endverbatim

      <b>disable_warnings</b> and <b>enable_warnings</b> commands can
      also take a second optional argument, <b>"ONCE"</b>, which when
      specified will disable or enable the warnings for next
      statement only.

      @verbatim
  --disable_warnings ER_BAD_TABLE_ERROR ONCE
  DROP TABLE IF EXISTS t1;
      @endverbatim

      Or

      @verbatim
  --disable_warnings
  DROP TABLE IF EXISTS t1;
  --enable_warnings ER_BAD_TABLE_ERROR ONCE
  DROP TABLE IF EXISTS t1;
  --enable_warnings
      @endverbatim
    </li>

      @note
      At the end of a test, all the disabled warnings must be enabled,
      else @ref PAGE_MYSQLTEST "mysqltest" will throw an error and
      abort the test run.
    <li>
      <tt>disconnect <b>connection_name</b></tt>

      Close the named connection. If the connection is the current
      connection, you should use the <b>connection</b> command to
      switch to a different connection before executing further SQL
      statements.

      @verbatim
  disconnect conn2;
  disconnect slave;
      @endverbatim
    </li>

    <li>
      <tt>echo <b>text</b></tt>

      Echo the text to the test result. References to variables within
      the text are replaced with the corresponding values. The text
      does not need to be enclosed in quotation marks; if it is, the
      quotation marks will be included in the output.

      @verbatim
  --echo Another sql_mode test
  echo should return only 1 row;
      @endverbatim
    </li>

    <li>
      <tt>end</tt>

      End an <b>if</b> or <b>while</b> block. If there is no such
      block open, @ref PAGE_MYSQLTEST "mysqltest" exits with an error.
      See @ref PAGE_MYSQL_TEST_FLOW_CONTROL, for information on
      flow-control constructs.

      @ref PAGE_MYSQLTEST "mysqltest" considers <b>}</b> and end the
      same: Both end the current block.
    </li>

    <li>
      <tt>end_timer</tt>

      Stop the timer. By default, the timer does not stop until just
      before @ref PAGE_MYSQLTEST "mysqltest" exits.
    </li>

    <li>
      <tt>error <b>error_code</b> [, <b>error_code</b>] ...</tt>

      Specify one or more comma-separated error values that the next
      command is expected to return. Each <b><em>error_code</em></b>
      value is a MySQL-specific error number or an <b>SQLSTATE</b>
      value. (These are the kinds of values returned by the
      <b>mysql_errno()</b> and <b>mysql_sqlstate()</b> C API
      functions, respectively.)

      If you specify an <b>SQLSTATE</b> value, it should begin with an
      <b>S</b> to enable @ref PAGE_MYSQLTEST "mysqltest" to
      distinguish it from a MySQL error number. For example, the error
      number 1050 and the SQLSTATE value <b>42S01</b> are equivalent,
      so the following commands specify the same expected error:

      @verbatim
  --error 1050
  --error S42S01
      @endverbatim

      <b>SQLSTATE</b> values should be five characters long and may
      contain only digits and uppercase letters.

      It is also possible to use the symbolic error name from
      <b>mysqld_error.h</b>:

      @verbatim
  --error ER_TABLE_EXISTS_ERROR
      @endverbatim

      It is also possible to use symbolic error names from
      <b>errmsg.h</b> to refer to client errors:

      @verbatim
  --error CR_SERVER_GONE_ERROR
      @endverbatim

      Finally, you can assign either a numerical code or a symbolic
      error name to a variable and refer to that in the <b>error</b>
      command. Numbers, symbolic names and variables may be freely
      mixed.

      If a statement fails with an error that has not been specified
      as expected by means of a <b>error</b> command, @ref
      PAGE_MYSQLTEST "mysqltest" aborts and reports the error message
      returned by the MySQL server.

      If a statement fails with an error that has been specified as
      expected by means of a <b>error</b> command, @ref PAGE_MYSQLTEST
      "mysqltest" does not abort. Instead, it continues and writes a
      message to the result output.

      <ul>
        <li>
          If an <b>error</b> command is given with a single error
          value and the statement fails with that error, @ref
          PAGE_MYSQLTEST "mysqltest" reports the error message
          returned by the MySQL server.

          Input:

          @verbatim
  --error S42S02
  DROP TABLE t;
          @endverbatim

          @ref PAGE_MYSQLTEST "mysqltest" reports:

          @verbatim
  ERROR 42S02: Unknown table 't'
          @endverbatim
        </li>

        <li>
          If an <b>error</b> command is given with multiple error
          values and the statement fails with any of those errors,
          @ref PAGE_MYSQLTEST "mysqltest" reports a generic message.
          (This is true even if the error values are all the same, a
          fact that can be used if you want a message that does not
          contain varying information such as table names.)

          Input:

          @verbatim
  --error S41S01,S42S02
  DROP TABLE t;
          @endverbatim

          @ref PAGE_MYSQLTEST "mysqltest" reports:

          @verbatim
  Got one of the listed errors
          @endverbatim
        </li>
      </ul>

      An error value of <b>0</b> or <b>S00000</b> means “no error,”
      so using either for an <b>error</b> command is the same as saying
      explicitly, “no error is expected, the statement must succeed.”.

      To indicate that you expect success or a given error or errors,
      specify <b>0</b> or <b>S00000</b> first in the error list. If
      you put the no-error value later in the list, the test will abort
      if the statement is successful. That is, the following two commands
      have different effects: The second form literally means the next
      command may fail with error code 0, (rather than succeed) which
      in practice never happens:

      @verbatim
  --error 0,1051
  --error 1051,0
      @endverbatim

      You can use <b>error</b> to specify shell status values for
      testing the value of shell commands executed using the
      <b>exec</b> command. This does not apply to <b>system</b>, for
      which the command status is ignored.

      If you use <b>error</b> in combination with <b>send</b> and
      <b>reap</b>, the <b>error</b> should be used just before the
      <b>reap</b>, as this is the command that actually gives the
      result and the potential error.

      Variables may also be used as arguments to the <b>error</b>
      command; these may contain a number (including 0), an SQLSTATE
      or a symbolic error name. Variables and constant values may be
      freely combined.
    </li>

    <li>
      <tt>eval <b>statement</b></tt>

      Evaluate the statement by replacing references to variables
      within the text with the corresponding values. Then send the
      resulting statement to the server to be executed. Use
      “<b>\$</b>” to specify a literal “<b>$</b>” character.

      The advantage of using <b>eval <em>statement</em></b> versus just
      <b><em>statement</em></b> is that <b>eval</b> provides variable
      expansion.

      @verbatim
  eval USE $DB;
  eval CHANGE MASTER TO MASTER_PORT=$SLAVE_MYPORT;
  eval PREPARE STMT1 FROM "$my_stmt";
      @endverbatim
    </li>

    <li>
      <tt>exec <b>command</b> [<b>arg</b>] ...</tt>

      Execute the shell command using the <b>popen()</b> library call.
      References to variables within the command are replaced with the
      corresponding values. Use “<b>\\$</b>” to specify a literal
      “<b>$</b>” character.

      On Cygwin, the command is executed from <b>cmd.exe</b>, so
      commands such as <b>rm</b> cannot be executed with <b>exec</b>.
      Use <b>system</b> instead.

      @verbatim
  --exec $MYSQL_DUMP --xml --skip-create test
  --exec rm $MYSQLTEST_VARDIR/tmp/t1
  exec $MYSQL_SHOW test -v -v;
      @endverbatim

      @note
      <b>exec</b> or <b>system</b> are sometimes used to perform
      file system operations, but the command for doing so tend to be
      operating system specific, which reduces test portability. @ref
      PAGE_MYSQLTEST "mysqltest" now has several commands to perform
      these operations portably, so they should be used instead:
      <b>remove_file</b>, <b>chmod</b>, <b>mkdir</b>, and so forth.
    </li>

    <li>
      <tt>exec_in_background <b>command</b> [<b>arg</b>] ...</tt>

      Execute a shell command using the <b>popen()</b> library call in
      the background. When a command is executed using
      <b>exec_in_background</b>, @ref PAGE_MYSQLTEST "mysqltest" does
      not wait for it to finish, nor attempt to read the output of it.

      @verbatim
  --exec_in_background $MYSQL_DUMP --xml --skip-create test
  --exec_in_background rm $MYSQLTEST_VARDIR/tmp/t1
      @endverbatim
    </li>

    <li>
      <tt>execw <b>command</b> [<b>arg</b>] ...</tt>

      This is a variant of the <b>exec</b> command which is needed on
      Windows if the command line contains non-ASCII characters.
      Otherwise it works exactly the same. On platforms other than
      Windows there is no difference, but on Windows it uses a
      different version of the <b>popen()</b> library call. So if your
      command line contains non-ASCII characters, it is recommended to
      use <b>execw</b> instead of <b>exec</b>.
    </li>

    <li>
      <tt>exit</tt>

      Terminate the test case. This is considered a “normal
      termination.” That is, using <b>exit</b> does not result in
      evaluation of the test case as having failed. It is not
      necessary to use <b>exit</b> at the end of a test case, as the
      test case will terminate normally when reaching the end without
      failure.
    </li>

    <li>
      <tt>expr <b>$var_name</b>= <b>operand1 operator operand2</b></tt>

      Evaluate an expression and assign the result to a variable. The
      result is also the return value of the expr command itself.

      @verbatim
  --let $val1= 10
  --let $var2= 20
  --expr $res= $var1 + $var2
  --echo $res
      @endverbatim

      <b>operand1</b> and <b>operand2</b> must be valid variables.

      expr supports these mathematical operators:

      @verbatim
  +  Addition
  -  Subtraction
  *  Multiplication
  /  Division
  %  Modulo
      @endverbatim

      expr supports these logical operators:

      @verbatim
  &&  Logical AND
  ||  Logical OR
      @endverbatim

      expr supports these bitwise operators:

      @verbatim
  &  Binary AND
  |  Binary OR
  ^  Binary XOR
  << Binary left shift
  >> Binary right shift
      @endverbatim

      Operations are subject to these conditions:

      - Operations that do not support noninteger operands truncate
        such operands to integer values.

      - If the result is an infinite value, expr returns the inf
        keyword.

      - Division by 0 results in an infinite value.
    </li>

    <li>
      <tt>file_exists <b>file_name</b> [<b>retry</b>]</tt>

      <b>file_exists</b> succeeds if the named file exists and fails
      otherwise. The file name argument is subject to variable
      substitution.

      <b>file_exists</b> can also take an optional argument
      <b>retry</b>. If the command fails due to an environmental
      issue, the command can be retried for a <b>retry</b> number
      of times. Each retry happens after an interval of one second.

      @verbatim
  file_exists /etc/passwd;
  file_exists /etc/passwd 5;
      @endverbatim
    </li>

    <li>
      <tt>force-cpdir <b>src_dir_name dst_dir_name</b></tt>

      Copies the source directory, <b><em>src_dir_name</em></b>, to
      the destination directory, <b><em>dst_dir_name</em></b>. The
      copy operation is recursive, so it copies subdirectories.
      Returns 0 for success and 1 for failure.

      @verbatim
  --force-cpdir testdir testdir2
      @endverbatim

      If the source directory does not exist, an error occurs.

      If the destination directory does not exist, @ref PAGE_MYSQLTEST
      "mysqltest" creates it before copying the source directory.
    </li>

    <li>
      <tt>force-rmdir <b>dir_name</b></tt>

      Remove a directory named <b><em>dir_name</em></b>. Returns 0 for
      success and 1 for failure.

      @verbatim
  --force-rmdir testdir
      @endverbatim

      <b>force-rmdir</b> removes the directory as well as its
      contents, if any, unlike <b>rmdir</b>, which fails if the
      directory to be removed contains any files or directories.
    </li>

    <li>
      <tt>horizontal_results</tt>

      Set the default query result display format to horizontal.
      Initially, the default is to display results horizontally.

      @verbatim
  --horizontal_results
      @endverbatim
    </li>

    <li>
      <tt>if (<b>expr</b>)</tt>

      Begin an if block, which continues until an <b>end</b> or
      <b>}</b> line. @ref PAGE_MYSQLTEST "mysqltest" executes the
      block if the expression is non-zero. There is no provision for
      <b>else</b> with <b>if</b>. See @ref
      PAGE_MYSQL_TEST_FLOW_CONTROL, for further information about
      <b>if</b> statements.

      @verbatim
  let $counter= 0;
  if ($counter)
  {
    echo Counter is not 0;
  }

  if (!$counter)
  {
    echo Counter is 0;
  }
      @endverbatim
    </li>

    <li>
      <tt>inc <b>$var_name</b></tt>

      Increment a numeric variable. If the variable does not have a
      numeric value, the result is undefined.

      @verbatim
  --inc $i;
  inc $3;
      @endverbatim
    </li>

    <li>
      <tt>let <b>$var_name</b> = <b>value</b></tt>

      <tt>let <b>$var_name</b> = query_get_value(<b>query</b>,
                                                 <b>col_name</b>,
                                                 <b>row_num</b>)</tt>

      Assign a value to a variable. The variable name cannot contain
      whitespace or the “<b>=</b>” character. Except for the one-digit
      <b>$0</b> to <b>$9</b>, it cannot begin with a number. @ref
      PAGE_MYSQLTEST "mysqltest" aborts with an error if the value is
      erroneous.

      References to variables within <b><em>value</em></b> are replaced
      with their corresponding values.

      If the <b>let</b> command is specified as a normal command (that
      is, not beginning with “<b>--</b>”), <b><em>value</em></b>
      includes everything up to the command delimiter, and thus can
      span multiple lines.

      @verbatim
  --let $1= 0
  let $count= 10;
      @endverbatim

      When assigning a literal string to a variable, no quoting is
      required even if the string contains spaces. If the string does
      include quotation marks, they will be trated like any other
      characters and be included in the string value. This is
      important to be aware of when using the variable in an SQL
      statement.

      The result from executing a query can be assigned to a variable
      by enclosing the query within backtick (“`”) characters:

      @verbatim
  let $q= `SELECT VERSION()`;
      @endverbatim

      The <b>let</b> command can set environment variables, not just
      @ref PAGE_MYSQLTEST "mysqltest" test language variables. To
      assign a value to an environment variable rather than a test
      language variable, just omit the dollar sign:

      @verbatim
  let $mysqltest_variable= foo;
  let ENV_VARIABLE= bar;
      @endverbatim

      This is useful in interaction with external tools. In
      particular, when using the <b>perl</b> command, the Perl code
      cannot access test language variables, but it can access
      environment variables. For example, the following statement can
      access the <b>ENV_VARIABLE</b> value:

      @verbatim
  print $ENV{'ENV_VARIABLE'};
      @endverbatim

      The <b>let</b> syntax also allows the retrieval of a value from
      a query result set produced by a statement such as <b>SELECT</b>
      or <b>SHOW</b>. See the description of <b>query_get_value()</b>
      for more information.
    </li>

    <li>
      <tt>mkdir <b>dir_name</b></tt>

      Create a directory named <b><em>dir_name</em></b>. Returns 0 for
      success and 1 for failure.

      @verbatim
  --mkdir testdir
      @endverbatim
    </li>

    <li>
      <tt>list_files <b>dir_name</b> [<b>pattern</b>]</tt>

      <b>list_files</b> lists the files in the named directory. If a
      pattern is given, lists only file(s) matching the pattern, which
      may contain wild cards.

      @verbatim
  --list_files $MYSQLD_DATADIR/test t1*
      @endverbatim
    </li>

    <li>
      <tt>list_files_append_file <b>file_name dir_name</b>
                                 [<b>pattern</b>]</tt>

      <b>list_files_append_file</b> works like <b>list_files</b>,
      but rather than outputting the file list, it is appended to the
      file named in the first argument. If the file does not exist, it
      is created.

      @verbatim
  --list_files_append_file $MYSQL_TMP_DIR/filelist $MYSQL_TMP_DIR/testdir *.txt;
      @endverbatim
    </li>

    <li>
      <tt>list_files_write_file <b>file_name dir_name</b>
                                [<b>pattern</b>]</tt>

      <b>list_files_write_file</b> works like
      <b>list_files_append_file</b>, but creates a new file to write
      the file list to. If the file already exists, it will be
      replaced.

      @verbatim
  --list_files_write_file $MYSQL_TMP_DIR/filelist $MYSQL_TMP_DIR/testdir *.txt;
      @endverbatim
    </li>

    <li>
      <tt>lowercase_result</tt>

      Output from the following SQL statement will be converted to
      lowercase. This is sometimes needed to ensure consistent result
      across different platforms. If this is combined with one of the
      <b>replace</b> commands or with <b>sorted_result</b>, both will
      take effect on the output, with conversion to lowercase being
      applied first.

      @verbatim
  --lowercase_result
      @endverbatim
    </li>

    <li>
      <tt>move_file <b>from_name to_name</b> [<b>retry</b>]</tt>

      <b>move_file</b> renames <b><em>from_name</em></b> to
      <b><em>to_name</em></b>. The file name arguments are subject to
      variable substitution, but must evaluate to a literal file name,
      not a file name pattern.

      <b>move_file</b> can also take an optional argument <b>retry</b>.
      If the command fails due to an environmental issue, the command
      can be retried for a <b>retry</b> number of times. Each retry
      happens after an interval of one second.

      @verbatim
  move_file $MYSQL_TMP_DIR/data01 $MYSQL_TMP_DIR/test.out;
  move_file $MYSQL_TMP_DIR/data01 $MYSQL_TMP_DIR/test.out 5;
      @endverbatim
    </li>

    <li>
      <tt>output <b>file_name</b></tt>

      Direct output from the next SQL statement to the named file
      rather than to the test output. If the file already exists, it
      will be overwritten. Only the next SQL statement will have its
      output redirected.

      @verbatim
  output $MYSQL_TMP_DIR/out-file
      @endverbatim
    </li>

    <li>
      <tt>perl [<b>terminator</b>]</tt>

      Use Perl to execute the following lines of the test file. The
      lines end when a line containing the terminator is encountered.
      The default terminator is <b>EOF</b>, but a different terminator
      can be provided.

      @verbatim
  perl;
  print "This is a test\n";
  EOF
      @endverbatim

      @verbatim
  perl END_OF_FILE;
  print "This is another test\n";
  END_OF_FILE
      @endverbatim
    </li>

    <li>
      <tt>ping</tt>

      Ping the server. This executes the <b>mysql_ping()</b> C API
      function. The function result is discarded. The effect is that
      if the connection has dropped and reconnect is enabled, pinging
      the server causes a reconnect.
    </li>

    <li>
      <tt>query [<b>statement</b>]</tt>

      Send the statement to the server to be executed. The
      <b>query</b> command can be used to force @ref PAGE_MYSQLTEST
      "mysqltest" to send a statement to the server even if it begins
      with a keyword that is a @ref PAGE_MYSQLTEST "mysqltest"
      command.
    </li>

    <li>
      <tt>query_get_value(<b>query</b>, <b>col_name</b>, <b>row_num</b>)</tt>

      The <b>query_get_value()</b> function can be used only on the
      right hand side of a variable assigment in a <b>let</b>
      statement.

      <b>query_get_value()</b> enables retrieval of a value from a
      query result set produced by a statement such as <b>SELECT</b>
      or <b>SHOW</b>. The first argument indicates the query to
      execute. The second and third arguments indicate the column name
      and row number that specify which value to extract from the
      result set. The column name is case sensitive. Row numbers begin
      with 1. The arguments can be given literally or supplied using
      variables.

      Suppose that the test file contains this input:

      @verbatim
  CREATE TABLE t1(a INT, b VARCHAR(255), c DATETIME);
  SHOW COLUMNS FROM t1;
  let $value= query_get_value(SHOW COLUMNS FROM t1, Type, 1);
  echo $value;
      @endverbatim

      The result will be:

      @verbatim
  CREATE TABLE t1(a INT, b VARCHAR(255), c DATETIME);
  SHOW COLUMNS FROM t1;
  Field   Type    Null    Key     Default Extra
  a       int(11) YES             NULL
  b       varchar(255)    YES             NULL
  c       datetime        YES             NULL
  int(11)
      @endverbatim

      If the query fails, an error message occurs and the test fails.
    </li>

    <li>
      <tt>query_horizontal <b>statement</b></tt>

      Execute the statement and display its result horizontally.

      @verbatim
  query_horizontal SELECT PI();
      @endverbatim
    </li>

    <li>
      <tt>query_vertical <b>statement</b></tt>

      Execute the statement and display its result vertically.

      @verbatim
  query_vertical SELECT PI();
      @endverbatim
    </li>

    <li>
      <tt>--let <b>$var</b> = escape(<b>characters</b>,<b>string</b>)</tt>

      Insert a backslash character before every character in
      <b>string</b> that occurs in <b>characters</b>, and assign the
      result to <b>$var</b>.

      <b>characters</b> must not contain newline. If it contains a
      comma, it must be the first character.  All whitespace is
      significant and preserved in both arguments, so you typically
      want to omit the space after the comma for this command.

      This can be useful when you need to include <b>string</b> in an
      SQL string in a query, and <b>string</b> may contain quote
      characters.

      @verbatim
      --let <b>$var</b> = escape(\',The 'path separator' in Windows is \.)
      --echo $var
      # Prints: The \'path separator\' in Windows is \\.
      @endverbatim
    </li>

    <li>
      <tt>reap</tt>

      Receive the result of the statement sent with the <b>send</b>
      command within the current session. You should not use
      <b>reap</b> unless a statement has been sent with <b>send</b>,
      and you should not use <b>send</b> again if there is an
      outstanding <b>send</b> that has not been processed with
      <b>reap</b>.
    </li>

    <li>
      <tt>remove_file <b>file_name</b> [<b>retry</b>]</tt>

      <b>remove_file</b> removes the file. It fails with an error if
      the file does not exist. The file name argument is subject to
      variable substitution, but must evaluate to a literal file name,
      not a file name pattern.

      <b>remove_file</b> also takes an optional argument <b>retry</b>.
      If the command fails due to an environmental issue, the command
      can be retried for a <b>retry</b> number of times. Each retry
      happens after an interval of one second.

      @verbatim
  remove_file $MYSQL_TMP_DIR/data01;
  remove_file $MYSQL_TMP_DIR/data01 5
      @endverbatim
    </li>

    <li>
      <tt>remove_files_wildcard <b>dir_name</b> <b>pattern</b> [<b>retry</b>]</tt>

      Remove all files in the named directory that match the pattern.
      Removal does not apply to directories matching the pattern or
      matching files in subdirectories. Patterns can use <b>?</b> to
      represent any single character, or <b>*</b> for any sequence of
      0 or more characters. The <b>.</b> character is treated like any
      other. The pattern may not include <b>/</b>. If all the files
      need to be removed, the <b>*</b> wildcard can be used.

      <b>remove_files_wildcard</b> can also take an optional argument
      <b>retry</b>. If the command fails due to an environmental
      issue, the command can be retried for a <b>retry</b> number of
      times. Each retry happens after an interval of one second.

      @verbatim
  remove_files_wildcard $MYSQL_TMP_DIR file*.txt;
  remove_files_wildcard $MYSQL_TMP_DIR file*.txt 5;
      @endverbatim
    </li>

    <li>
      <tt>replace_column <b>col_num</b> <b>value</b>
                         [<b>col_num value</b>] ...</tt>

      Replace strings in the output from the next statement. The value
      in <b><em>col_num</em></b> is replaced by the corresponding
      <b><em>value</em></b>. There can be more than one
      <b><em>col_num/value</em></b> pair. Column numbers start with 1.

      A replacement value can be double-quoted. (Use “<b>\"</b>” to
      specify a double quote within a replacement string.) Variables
      can be used in a replacement value if it is not double-quoted.

      If mixed <b>replace_<em>xxx</em></b> commands are given, only the
      final one applies.

      @note
      Although <b>replace_regex</b> and <b>replace_result</b> affect
      the output from <b>exec</b>, <b>replace_column</b> does not
      because <b>exec</b> output is not necessarily columnar.

      @verbatim
  --replace_column 9 #
  replace_column 1 b 2 d;
      @endverbatim
    </li>

    <li>
      <tt>replace_numeric_round <b>precision</b></tt>

      Rounds off floating point numbers. <b>precision</b> is the
      number of digits after the decimal point to which the result will
      be rounded off to. It can only be a number between
      0 and 16.

      If the precision is 0, then the value is rounded off to the
      nearest whole number. Trailing zeroes after the decimal point are
      removed from the numbers.

      Numbers which are <b>greater than <em>1e10</em></b> or <b> lesser
      than <em> -1e10 </em></b> are represented using the
      exponential notation after they are rounded off.

      @verbatim
  --replace_numeric_round 6
  SELECT 12379.79459775333;
  12379.794598

  replace_numeric_round 0;
  SELECT 876.87302;
  877
      @endverbatim
    </li>


    <li>
      <tt>replace_regex <b>/pattern/replacement/</b>[i] ...</tt>

      In the output from the next statement, find strings within
      columns of the result set that match <b><em>pattern</em></b> (a
      regular expression) and replace them with
      <b><em>replacement</em></b>. Each instance of a string in a
      column that matches the pattern is replaced. Matching is case
      sensitive by default. Specify the optional <b>i</b> modifier to
      cause matching to be case insensitive.

      The syntax for allowable patterns is same as the <b>std::ECMAScript</b>
      syntax. The pattern can contain parentheses to mark substrings
      matched by parts of the pattern. These substrings can be referenced in
      the replacement string: An instance of <b><em>N</em></b> in the
      replacement string causes insertion of the <b><em>N</em>-th</b> substring
      matched by the pattern. For example, the following command matches
      <b>strawberry</b> and replaces it with <b>raspberry</b> and
      <b>strawberry</b>:

      @verbatim
  --replace_regex /(strawberry)/raspberry and \1/
      @endverbatim

      Multiple <b><em>pattern/replacement</em></b> pairs may be given.
      The following command replaces instances of <b>A</b> with
      <b>C</b> (the first pattern replaces <b>A</b> with <b>B</b>, the
      second replaces <b>B</b> with <b>C</b>):

      @verbatim
  --replace_regex /A/B/ /B/C/
      @endverbatim

      If a given pattern is not found, no error occurs and the input
      is unchanged.
    </li>

    <li>
      <tt>replace_result <b>from_val to_val</b>
                         [<b>from_val to_val</b>] ...</tt>

      Replace strings in the result. Each occurrence of
      <b><em>from_val</em></b> is replaced by the corresponding
      <b><em>to_val</em></b>. There can be more than
      <b><em>from_val/to_val</em></b> pair. Arguments can be quoted
      with single quotation marks or double quotation marks. Variable
      references within the arguments are expanded before replacement
      occurs. Values are matched literally. To use patterns, use the
      <b>replace_regex</b> command.

      @verbatim
  --replace_result 1024 MAX_KEY_LENGTH 3072 MAX_KEY_LENGTH
  replace_result $MASTER_MYPORT MASTER_PORT;
      @endverbatim
    </li>

    <li>
      <tt>reset_connection</tt>

      Reset the connection state by calling [mysql_reset_connection()]
      (https://dev.mysql.com/doc/refman/8.0/en/mysql-reset-connection.html).
    </li>

    <li>
      <tt>result_format <b>version</b></tt>

      Set the format to the specified version, which is either 1 for
      the current, default behavior, or to 2 which is an extended
      alternative format. The setting is in effect until it is changed
      or until the end of the test.

      In format version 2, empty lines and indentation in the test
      file are preserveded in the result. Also, comments indicated by
      a double \## are copied verbatim to the result. Comments using a
      single \# are not copied. Format version 2 makes it easier for
      humans to read the result output, but at the cost of somewhat
      larger files due to the white space and comments.

      @verbatim
  --result_format 2
      @endverbatim
    </li>

    <li>
      <tt>rmdir <b>dir_name</b></tt>

      Remove a directory named <b><em>dir_name</em></b>. Returns 0 for
      success and 1 for failure.

      @verbatim
  --rmdir testdir
      @endverbatim

      <b>rmdir</b> fails if the directory to be removed contains any
      files or directories. To remove the directory as well as its
      contents, if any, use <b>force-rmdir</b>.
    </li>

    <li>
      <tt>save_master_pos</tt>

      For a master replication server, save the current binary log
      file name and position. These values can be used for subsequent
      <b>sync_with_master</b> or <b>sync_slave_with_master</b>
      commands.
    </li>

    <li>
      <tt>send [<b>statement</b>]</tt>

      Send a statement to the server but do not wait for the result.
      The result must be received with the <b>reap</b> command. You
      cannot execute another SQL statement on the same connection
      between <b>send</b> and <b>reap</b>.

      If <b><em>statement</em></b> is omitted, the <b>send</b> command
      applies to the next statement executed. This means that
      <b>send</b> can be used on a line by itself before a statement.
      Thus, this command:

      @verbatim
  send SELECT 1;
      @endverbatim

      Is equivalent to these commands:

      @verbatim
  send;
  SELECT 1;
      @endverbatim
    </li>

    <li>
      <tt>send_eval [<b>statement</b>]</tt>

      Evaluate the command, then send it to the server. This is a
      combination of the <b>send</b> and <b>eval</b> commands, giving
      the functionality of both. After variable replacement has been
      done, it behaves like the <b>send</b> command.

      If <b><em>statement</em></b> is omitted, the <b>send_eval</b>
      command applies to the next statement executed. This means that
      <b>send_eval</b> can be used on a line by itself before a
      statement. Thus, this command:

      @verbatim
  --send_eval $my_stmt
      @endverbatim
    </li>

      Is equivalent to these commands:

      @verbatim
  --send_eval
  $my_stmt;
      @endverbatim
    </li>

    <li>
      <tt>send_quit <b>connection</b></tt>

      Sends a <b>COM_QUIT</b> command to the server on the named
      connection.

      @verbatim
  send_quit con;
      @endverbatim
    </li>

    <li>
      <tt>send_shutdown</tt>

      Sends a shutdown command to the server but does not wait for it
      to complete the shutdown. Test execution proceeds as soon as the
      shutdown command is sent.
    </li>

    <li>
      <tt>shutdown_server [<b>timeout</b>]</tt>

      Stops the server. This command waits for the server to shut down
      by monitoring its process ID (PID) file. If the server's process
      ID file is not gone after <b><em>timeout</em></b> seconds, the
      process will be killed. If <b><em>timeout</em></b> is omitted,
      the default is 60 seconds.

      @verbatim
  shutdown_server;
  shutdown_server 30;
      @endverbatim
    </li>

    <li>
      <tt>skip [<b>message</b>]</tt>

      Skips the rest of the test file after printing the given message
      as the reason. This can be used after checking a condition that
      must be satisfied, as a way of performing an exit that displays
      a reason. References to variables within the message are
      replaced with the corresponding values. Suppose that the test
      file <b>mytest.test</b> has these contents:

      @verbatim
  let $var= 0;
  if (!$var)
  {
    skip value of \$var is '$var', skipping test;
  }
  echo This command is never reached;
      @endverbatim

      Executing @ref PAGE_MYSQLTEST "mysqltest" -x mytest.test yields
      these results:

      @verbatim
  The test './mytest' is not supported by this installation
  Detected in file ./mytest at line 4
  reason: value of $var is '0', skipping test
      @endverbatim

      If the test is run from @ref PAGE_MYSQL_TEST_RUN_PL
      "mysql-test-run.pl", you will instead see the test result as
      <b>[ skipped ]</b> followed by the message.

      @verbatim
  main.mytest          [ skipped ]  value of $var is '0', skipping test
      @endverbatim
    </li>

    <li>
      <tt>sleep <b>num</b></tt>

      Sleep <b><em>num</em></b> seconds. <b><em>num</em></b> need not be
      a whole number and can have a fractional part as well i.e., the
      command supports sleeping for intervals shorter than 1 second.

      @verbatim
  --sleep 10
  sleep 0.5;
      @endverbatim

      Try not to use <b>sleep</b> command more than necessary. The
      more of them there are, the slower the test suite becomes.
    </li>

    <li>
      <tt>sorted_result</tt>

      Sort the output from the next statement if it produces a result
      set. <b>sorted_result</b> is applied just before displaying the
      result, after any other result modifiers that might have been
      specified, such as <b>replace_result</b> or
      <b>replace_column</b>. If the next statement produces no result
      set, <b>sorted_result</b> has no effect because there is nothing
      to sort.

      @verbatim
  sorted_result;
  SELECT 2 AS "my_col" UNION SELECT 1;
  let $my_stmt=SELECT 2 AS "my_col" UNION SELECT 1;
  --sorted_result
  eval $my_stmt;
  --sorted_result
  --replace_column 1 #
  SELECT '1' AS "my_col1",2 AS "my_col2"
  UNION
  SELECT '2',1;
      @endverbatim

      <b>sorted_result</b> sorts the entire result of the next query.
      If this involves constructs such as <b>UNION</b>, stored
      procedures, or multi-statements, the output will be in a fixed
      order, but all the results will be sorted together and might
      appear somewhat strange.

      The purpose of the <b>sorted_result</b> command is to produce
      output with a deterministic order for a given set of result
      rows. It is possible to use <b>ORDER BY</b> to sort query
      results, but that can sometimes present its own problems. For
      example, if the optimizer is being investigated for some bug,
      <b>ORDER BY</b> might order the result but return an incorrect
      set of rows. <b>sorted_result</b> can be used to produce sorted
      output even in the absence of <b>ORDER BY</b>.

      <b>sorted_result</b> is useful for eliminating differences
      between test runs that may otherwise be difficult to compensate
      for. Results without <b>ORDER BY</b> are not guaranteed to be
      returned in any given order, so the result for a given query
      might differ between test runs. For example, the order might
      vary between different server versions, so a result file created
      by one server might fail when compared to the result created by
      another server. The same is true for different storage engines.
      <b>sorted_result</b> eliminates these order differences by
      producing a deterministic row order.

      Other ways to eliminate differences from results without use of
      <b>sorted_result</b> include:

      - Remove columns from the select list to reduce variability in
        the output

      - Use aggregate functions such as <b>AVG()</b> on all columns
        of the select list

      - Use <b>ORDER BY</b>

      The use of aggregate functions or <b>ORDER BY</b> may also have
      the advantage of exposing other bugs by introducing additional
      stress on the server. The choice of whether to use
      <b>sorted_result</b> or <b>ORDER BY</b> (or perhaps both) may be
      dictated by whether you are trying to expose bugs, or avoid
      having them affect results. This means that care should be taken
      with <b>sorted_result</b> because it has the potential of hiding
      server bugs that result in true problems with result order.
    </li>

    <li>
      <tt>partially_sorted_result <b>start_column</b></tt>

      Similarly to <b>sorted_result</b>, but assumes that the first
      <b>start_column</b> columns are already sorted, sorting only
      on the remaining ones. This is useful if you wish to verify
      a partial ordering property, e.g. that <b>SELECT a,b,c ORDER BY
      a</b>. In this case, you could use <b>partially_sorted_result 1</b>
      to sort only on b and c, giving deterministic results without
      masking bugs where sorting on a column would not work correctly.

      <tt>sorted_result</tt> is functionally equal to
      <tt>partially_sorted_result 0</tt>.
    </li>

    <li>
      <tt>source <b>file_name</b></tt>

      Read test input from the named file.

      If you find that several test case files contain a common
      section of commands (for example, statements that create a
      standard set of tables), you can put those commands in another
      file and those test cases that need the file can include it by
      means of a <b>source <em>file_name</em></b> command. This
      enables you to write the code just once rather than in multiple
      test cases.

      Normally, the file name in the <b>source</b> command is relative
      to the <b>mysql-test</b> directory because @ref PAGE_MYSQLTEST
      "mysqltest" usually is invoked in that directory.

      A sourced file can use <b>source</b> to read other files, but
      take care to avoid a loop. The maximum nesting level is 16.

      @verbatim
  --source include/have_csv.inc
  source include/varchar.inc;
      @endverbatim

      The file name can include variable references. Variables are
      expanded including any quotation marks in the values, so
      normally the values should not include quotation marks. Suppose
      that <b>/tmp/junk</b> contains this line:

      @verbatim
  SELECT 'I am a query';
      @endverbatim

      The following example shows one way in which variable references
      could be used to specify the file name:

      @verbatim
  let $dir= /tmp;
  let $file= junk;
  source $dir/$file;
      @endverbatim
    </li>

    <li>
      <tt>start_timer</tt>

      Restart the timer, overriding any timer start that occurred
      earlier. By default, the timer starts when @ref PAGE_MYSQLTEST
      "mysqltest" begins execution.
    </li>

    <li>
      <tt>sync_slave_with_master [<b>connection_name</b>]</tt>

      Executing this command is equivalent to executing the following
      commands:

      @verbatim
  save_master_pos;
  connection connection_name;
  sync_with_master 0;
      @endverbatim

      If <b><em>connection_name</em></b> is not specified, the
      connection named <b>slave</b> is used.

      The effect is to save the replication coordinates (binary log
      file name and position) for the server on the current connection
      (which is assumed to be a master replication server), and then
      switch to a slave server and wait until it catches up with the
      saved coordinates. Note that this command implicitly changes the
      current connection.

      A variable can be used to specify the
      <b><em>connection_name</em></b> value.
    </li>

    <li>
      <tt>sync_with_master <b>offset</b></tt>

      For a slave replication server, wait until it has caught up with
      the master. The position to synchronize to is the position saved
      by the most recent <b>save_master_pos</b> command plus offset.

      To use this command, <b>save_master_pos</b> must have been
      executed at some point earlier in the test case to cause @ref
      PAGE_MYSQLTEST "mysqltest" to save the master's replication
      coordinates.
    </li>

    <li>
      <tt>vertical_results</tt>

      Set the default query result display format to vertical.
      Initially, the default is to display results horizontally.

      @verbatim
  --vertical_results
      @endverbatim
    </li>

    <li>
      <tt>wait_for_slave_to_stop</tt>

      Poll the current connection, which is assumed to be a connection
      to a slave replication server, by executing <b>SHOW STATUS LIKE
      'Slave_running'</b> statements until the result is <b>OFF</b>.

      For information about alternative means of slave server control,
      see @ref PAGE_WRITING_REPLICATION_TESTS.
    </li>

    <li>
      <tt>while (<b>expr</b>)</tt>

      Begin a <b>while</b> loop block, which continues until an
      <b>end</b> line. @ref PAGE_MYSQLTEST "mysqltest" executes the
      block repeatedly as long as the expression is true (non-zero).
      See @ref PAGE_MYSQL_TEST_FLOW_CONTROL, for further information
      about while statements.

      Make sure that the loop includes some exit condition that
      eventually occurs. This can be done by writing
      <b><em>expr</em></b> so that it becomes false at some point.

      @verbatim
  let $i=5;
  while ($i)
  {
    echo $i;
    dec $i;
  }
      @endverbatim
    </li>

    <li>
      <tt>write_file <b>file_name</b> [<b>terminator</b>]</tt>

      Write the following lines of the test file to the given file,
      until a line containing the terminator is encountered. The
      default terminator is <tt>EOF</tt>, but a different terminator
      can be provided. The file name argument is subject to variable
      substitution. An error occurs if the file already exists.

      @verbatim
  write_file $MYSQL_TMP_DIR/data01;
  line one for the file
  line two for the file
  EOF
      @endverbatim

      @verbatim
  write_file $MYSQL_TMP_DIR/data02 END_OF_FILE;
  line one for the file
  line two for the file
  END_OF_FILE
      @endverbatim
    </li>
  </ul>
*/

###############################################################################



## mysqltest Variables

/**
  @page PAGE_MYSQL_TEST_VARIABLES mysqltest Variables

  You can define variables and refer to their values. You can also
  refer to environment variables, and there is a built-in variable
  that contains the result of the most recent SQL statement.

  To define a variable, use the <b>let</b> command. Examples:

  @verbatim
  let $a= 14;
  let $b= this is a string;
  --let $a= 14
  --let $b= this is a string
  @endverbatim

  The variable name cannot contain whitespace or the “=” character.

  If a variable has a numeric value, you can increment or decrement
  the value:

  @verbatim
  inc $a;
  dec $a;
  --inc $a
  --dec $a
  @endverbatim

  <b>inc</b> and <b>dec</b> are commonly used in <b>while</b>
  loops to modify the value of a counter variable that controls loop
  execution.

  The result from executing a query can be assigned to a variable by
  enclosing the query within backtick (“`”) characters:

  @verbatim
  let $q= `select version()`;
  @endverbatim

  References to variables can occur in the <b>echo</b>,
  <b>eval</b>, <b>exec</b>, and <b>system</b> commands. Variable
  references are replaced by their values. A nonquery value assigned
  to a variable in a <b>let</b> command can also refer to variables.

  Variable references that occur within
  \`<b><em>query</em></b>\` are expanded before the query is
  sent to the server for execution.

  You can refer to environment variables. For example, this command
  displays the value of the <b>$PATH</b> variable from the
  environment:

  @verbatim
  --echo $PATH
  @endverbatim

  <b>$mysql_errno</b> is a built-in variable that contains the
  numeric error returned by the most recent SQL statement sent to the
  server, or 0 if the statement executed successfully.
  <b>$mysql_errno</b> has a value of −1 if no statement has yet been
  sent.

  <b>$mysql_errname</b> similarly contains the symbolic name of the
  last error, or an empty string if there was no error.

  @ref PAGE_MYSQLTEST "mysqltest" first checks @ref PAGE_MYSQLTEST
  "mysqltest" variables and then environment variables. @ref
  PAGE_MYSQLTEST "mysqltest" variable names are not case sensitive.
  Environment variable names are case sensitive.
*/

###############################################################################



## mysqltest Flow Control Constructs

/**
  @page PAGE_MYSQL_TEST_FLOW_CONTROL mysqltest Flow Control Constructs

  The syntax for <b>if</b> block looks like this:

  @verbatim
  if (expr)
  {
    command list
  }
  @endverbatim

  The syntax for <b>while</b> block looks like this:

  @verbatim
  while (expr)
  {
    command list
  }
  @endverbatim

  An expression result is true if nonzero, false if zero. If the
  expression begins with <b>!</b>, the sense of the test is reversed.

  If the expression is a string that does not begin with a numeric
  digit (possibly preceeded by a plus or minus sign), it evaluates as
  true if non-empty. Any white space is ignored in this case, so a
  string consisting of only white space is false.

  There is no provision for <b>else</b> with <b>if</b>.

  For a <b>while</b> loop, make sure that the loop includes some
  exit condition that eventually occurs. This can be done by writing
  <b><em>expr</em></b> so that it becomes false at some point.

  The allowable syntax for <b><em>expr</em></b> is
  <b>$<em>var_name</em></b>, <b>!$<em>var_name</em></b>,
  <b><em>integer</em></b>, or \`<b><em>query</em></b>\`.

  The expression can also be a simple comparison, where the left hand
  side must be a variable, and the right hand side can be string or any type
  valid for the single expression except the negated variable. The
  supported operators are ==, !=, <, <=, > and >=. Only the first two
  may be used if the right hand side does not evaluate to an integer.
  With ==, strings must match exactly.

  If you use a string on the right hand side of the comparison, it
  does not have to be quoted even if it contains spaces. It may
  optionally be enclosed in single or double quotation marks which
  will then be stripped off before comparison. This is in contrast to
  <b>let</b> statements, where quoting is not stripped.

  Examples of the expression syntax with and without comparisons (only the header
  shown):

  @verbatim
  while ($counter<5) ...
  if ($value == 'No such row') ...
  if ($slave_count != $master_count) ...
  if ($counter < `SELECT 10`) ...
  if (-1) ...
  if (20) ...
  if (`SELECT 1`) ...
  if(! $cnt) ...
  @endverbatim

  The opening <b>{</b> (curly brace) must be separated from the
  preceding <b>)</b> (right parenthesis) by whitespace, such as a
  space or a line break.

  Variable references that occur within
  \`<b><em>query</em></b>\` are expanded before the query is
  sent to the server for execution.
*/

###############################################################################



## Error Handling

/**
  @page PAGE_ERROR_HANDLING Error Handling

  If an expected error is specified and that error occurs, @ref
  PAGE_MYSQLTEST "mysqltest" continues reading input. If the command
  is successful or a different error occurs, @ref PAGE_MYSQLTEST
  "mysqltest" aborts.

  If no expected error is specified, @ref PAGE_MYSQLTEST "mysqltest"
  aborts unless the command is successful. (It is implicit that you
  expect <b>$mysql_errno</b> to be 0.)

  By default, @ref PAGE_MYSQLTEST "mysqltest" aborts for certain
  conditions:

  <ul>
    <li>
      A statement that fails when it should have succeeded. The
      following statement should succeed if table <b>t</b> exists;

      @verbatim
  SELECT * FROM t;
      @endverbatim
    </li>

    <li>
      A statement that fails with an error different from that
      specified:

      @verbatim
  --error 1
  SELECT * FROM no_such_table;
      @endverbatim
    </li>

    <li>
      A statement that succeeds when an error was expected:

      @verbatim
  --error 1
  SELECT 'a string';
      @endverbatim
    </li>
  </ul>

  You can disable the abort for errors of the first type by using the
  <b>disable_abort_on_error</b> command. In this case, when errors
  occur for statements that should succeed, @ref PAGE_MYSQLTEST
  "mysqltest" continues processing intput.

  <b>disable_abort_on_error</b> does <b>not</b> cause @ref
  PAGE_MYSQLTEST "mysqltest" to ignore errors for the other two types,
  where you explicitly state which error you expect. This behavior is
  intentional. The rationale is that if you use the <b>error</b>
  command to specify an expected error, it is assumed that the test is
  sufficiently well characterized that only the specified error is
  accceptable.

  If you do not use the <b>error</b> command, it is assumed that you
  might not know which error to expect or that it might be difficult
  to characterize all possible errors that could occur. In this case,
  <b>disable_abort_on_error</b> is useful for causing @ref
  PAGE_MYSQLTEST mysqltest to continue processing input. This can be
  helpful in the following circumstances:

  <ul>
    <li>
      During test case development, it is useful to process all input
      even if errors occur so that you can see all errors at once, such
      as those that occur due to typographical or syntax errors.
      Otherwise, you can see and fix only one scripting problem at a
      time.
    </li>

    <li>
      Within a file that is included with a <b>source</b> command by
      several different test cases, errors might vary depending on the
      processing environment that is set up prior to the <b>source</b>
      command.
    </li>

    <li>
      Tests that follow a given statement that can fail are independent
      of that statement and do not depend on its result.
    </li>
  </ul>
*/

###############################################################################



## Creating and Executing Unit Tests

/**
  @page PAGE_UNIT_TESTS Creating and Executing Unit Tests

  <h3>Table of Contents</h3>
  - @subpage PAGE_UNIT_TESTS_TAP
  - @subpage PAGE_UNIT_TESTS_GOOGLE
  - @subpage PAGE_UNIT_TESTS_MAIN_TEST_RUNS

  Storage engines and plugins can have unit tests to test their
  components. The top-level <b>Makefile</b> target
  <b>test-unit</b> runs all unit tests: It scans the storage engine
  and plugin directories, recursively, and executes all executable
  files having a name that ends with <b>-t</b>.

  The following sections describe MySQL unit testing using TAP and
  the Google Test framework.
*/

###############################################################################



## Unit Testing Using TAP

/**
  @page PAGE_UNIT_TESTS_TAP Unit Testing Using TAP

  The unit-testing facility based on the Test Anything Protocol (TAP)
  is mainly used when developing Perl and PHP modules. To write unit
  tests for C/C++ code, MySQL has developed a library for generating
  TAP output from C/C++ files. Each unit test is written as a separate
  source file that is compiled to produce an executable. For the unit
  test to be recognized as a unit test, the executable file has to be
  of the format <b>mytext-t</b>. For example, you can create a
  source file named <b>mytest-t.c</b> that compiles to produce an
  executable <b>mytest-t</b>. The executable will be found and run
  when you execute <b>make test</b> or <b>make test-unit</b> in the
  distribution top-level directory.

  Example unit tests can be found in the <b>unittest/examples</b>
  directory of a MySQL source distribution. The code for the MyTAP
  protocol is located in the <b>unittest/mytap</b> directory.

  Each unit test file should be stored in a storage engine or plugin
  directory (<b>storage/<em>engine_name</em></b> or
  <b>plugin/<em>plugin_name</em></b>), or one of its
  subdirectories. A reasonable convention is to create a
  <b>unittest</b> subdirectory under the storage engine or plugin
  directory and create unit test files in <b>unittest</b>.
*/

###############################################################################



## Unit Testing Using the Google Test Framework

/**
  @page PAGE_UNIT_TESTS_GOOGLE Unit Testing Using the Google Test Framework

  The Google Test unit-testing framework is available in MySQL source
  trees and distributions. Google Test, like MyTAP, provides a unit-
  testing framework, but Google Test provides richer functionality,
  such as:

  - A rich set of predicates

  - User-defined predicates and assertions

  - Automatic test registration

  - Nice error reporting when a predicate fails (with line number, expected
    and actual values, and additional comments)

  - Test fixtures, and setup/teardown logic

  - Death tests

  - Disabled tests

  - Test filtering and shuffling

  Google Test runs on many platforms. Some functionality is missing on
  some platforms (such as death tests and parameterized tests), so
  those features should not be used.

  This section provides notes about using Google Test within the
  context of MySQL development. For general-purpose information about
  Google Test, see these resources:

  - Main Google Test page: http://code.google.com/p/googletest

  - Primer: http://code.google.com/p/googletest/wiki/GoogleTestPrimer

  - Advanced guide:
    http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide

  - Google presentation:
    http://docs.google.com/present/view?id=dfsbxvm5_0f5s4pvf9

  @section SECTION_WRITING_UNIT_TESTS Writing Unit Tests for Google Test

  Most of The Google Test unit test files are located in the
  <b>unittest/gunit</b> directory. You can look at these files to
  see how tests are written. Here are some examples:

  - <tt>sql_list-t.cc</tt>: A simple test of some list classes

  - <tt>mdl-t.cc</tt>: Some tests of metadata locking (MDL),
    including testing of lock acquisition from multiple threads

  - <tt>mdl_mytap-t.cc</tt>: The same tests as <b>mdl-t.cc</b>, but
    written for MyTAP, to illustrate some features of Google Test

  Most MyTAP-based tests are likely easily converted to Google Test.
  However, there might be low-level tests that absolutely must be run
  on every platform, and thus require MyTAP.

  To see what options are available, run a test executable with the
  <b>`--help`</b> option.
*/

###############################################################################



## Unit Tests Added to Main Test Runs

/**
  @page PAGE_UNIT_TESTS_MAIN_TEST_RUNS Unit Tests Added to Main Test Runs

  @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will also run unit
  tests at the end of full test runs, when being run from within a
  build directory. It depends on the unit tests having been built and
  defined in a file <b>CTestTestfile.cmake</b> in the top level build
  directory. Those will normally be there after a build using
  <b>CMake</b>, but will not be in a binary package.

  The unit tests are run simply by executing <b>ctest</b> with no
  arguments from the top level build directory. The result will be
  shown as a single test at the end, named <b>unit_tests</b> which
  passes if and only if all unit tests pass. A summary of the result
  will be printed, including the name of any failed unit tests. The
  set of unit tests will be counted as one test (either passed or
  failed) in the overall test summary.

  Unit tests will by default be run only if you have not specified any
  specific tests or suites on the command line for @ref
  PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". This can be overridden
  by setting the environment variable <b>MTR_UNIT_TESTS</b> to 0 or 1.
  This in turn can be overriden by a command line argument
  <b>`--unit-tests`</b> or <b>`--nounit-tests`</b>.

  If the file <b>CTestTestfile.cmake</b> and the <b>ctest</b>
  command are not both available, unit tests will be silently skipped,
  unless you have used the command line option <b>`--unit-tests`</b>.
*/

###############################################################################



## Plugins for Testing Plugin Services

/**
  @page PAGE_PLUGINS Plugins for Testing Plugin Services

  MySQL server plugins have access to server “services”, as described in
  [MySQL Services for Plugins]
  (https://dev.mysql.com/doc/refman/8.0/en/plugin-services.html).
  MySQL distributions include plugins that demonstrate how to test
  plugin service APIs:

  <ul>
    <li>
      The <b>test_framework</b> plugin is a bare bones plugin that
      shows the minimum required framework for service testing.
    </li>

    <li>
      The <b>test_services</b> plugin demonstrates how to test the
      <b>my_plugin_log_service</b> service in unthreaded context.
    </li>

    <li>
      The <b>test_services_threaded</b> plugin is like
      <b>test_services</b>, but for threaded context.
    </li>
  </ul>

  The source code for the plugins is located in the
  <b>plugin/test_services</b> directory of MySQL source
  distributions. The <b>README</b> file in that directory contains
  instructions for running the test cases available for the
  <b>test_services</b> and <b>test_services_threaded</b> plugins.

  @note
  The test plugins in <b>plugin/test_services</b> are daemon plugins
  (see [Daemon Plugins]
  (https://dev.mysql.com/doc/refman/8.0/en/plugin-types.html#daemon-plugin-type)).
  For an example of a nondaemon service-testing plugin plugin, see the
  <b>test_security_context.cc</b> file in the <b>plugin/audit_null</b>
  directory. This file creates an audit plugin for testing the
  <b>security_context</b> service.

  Use the following procedure to create a new service-testing plugin
  based on one of those provided in the <b>plugin/test_services</b>
  directory. Assume that you want to create a new plugin named
  <b>test_myservice</b> (or <b>test_myservice_threaded</b> to test
  in threaded context).

  <ol>
    <li>
      Select a source file to use as a basis for the new plugin:

      <ul>
        <li>
          To begin with a bare bones plugin, copy <b>test_framework.cc</b>
          to <b>test_myservice.cc</b>.
        </li>

        <li>
          To begin with a plugin that already includes code for running tests
          in unthreaded context, copy <b>test_services.cc</b> to
          <b>test_myservice.cc</b>.
        </li>

        <li>
          To begin with a plugin that already includes code for running tests
          in threaded context, copy <b>test_services_threaded.cc</b> to
          <b>test_myservice_threaded.cc</b>.
        </li>
      </ul>
    </li>

    <li>
      There is a plugin descriptor near the end of the new source file. Modify
      this descriptor appropriately for your plugin. Change the <b>name</b>,
      <b>author</b>, and <b>descr</b> members that indicate the plugin
      name and author and provide a description. For example, if you copied
      <b>test_framework.cc</b>, those members look like this:

      @verbatim
  "test_framework",
  "Horst Hunger",
  "Test framework",
      @endverbatim

      Change them to something like this:

      @verbatim
  "test_myservice",
  "Your Name Here",
  "Test My Service",
      @endverbatim
    </li>

    <li>
      Modify your source file appropriately for the service to be tested:

      <ul>
        <li>
          If you copied <b>test_framework.cc</b>, your file has no
          tests initially and is set up for unthreaded context. In
          this case, add code to the
          <b>test_services_plugin_init()</b> function. This code
          should invoke the service to be tested.
        </li>

        <li>
          If you copied <b>test_services.cc</b> or
          <b>test_services_threaded.cc</b>, the file contains tests
          for the <b>my_plugin_log_service</b> service in unthreaded or
          threaded contexts. Replace or modify those tests with code
          for your own tests.
          those tests with code for your own tests.
        </li>
      </ul>
    </li>
  </ol>

  Compiling your plugin creates a plugin library file, which you
  should install in the directory named by the
  [plugin_dir]
  https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_plugin_dir)
  system variable. The file base name is the same as that of the
  source file. The file name suffix differs per platform (for example,
  <b>.so</b> for Unix and Unix-like systems, <b>.dll</b> for
  Windows).

  To install or unintall your plugin at server startup, use the
  [`--plugin-load`]
  (https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_plugin-load) or
  [`--plugin-load-add`]
  (https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_plugin-load-add)
  option. For example, you can use these lines in an option file
  (adjust the file name as necessary):

  @verbatim
  [mysqld]
  plugin-load-add=test_myservice.so
  @endverbatim

  To install or uninstall the plugin at runtime, use these statements
  (adjust the plugin name and file name as necessary):

  @verbatim
  INSTALL PLUGIN test_myservice SONAME 'test_myservice.so';
  UNINSTALL PLUGIN test_myservice;
  @endverbatim

  For addition information about plugin loading, see
  [Installing and Uninstalling Plugins]
  (https://dev.mysql.com/doc/refman/8.0/en/server-plugin-loading.html).

  For information about creating and running test cases for your new
  plugin, adapt the instructions in the <b>README</b> file in the
  <b>plugin/test_services</b> directory. Test cases for the
  <b>test_services</b> and <b>test_services_threaded</b> plugins
  are located in <b>mysql-test/suite/test_services</b>.
*/

###############################################################################