File: extract.html

package info (click to toggle)
fcm 2021.05.01-2
links: PTS, VCS
area: main
in suites: bookworm, forky, sid, trixie
size: 7,788 kB
sloc: perl: 26,014; sh: 10,510; javascript: 4,043; f90: 774; python: 294; ansic: 29; makefile: 14; cpp: 5
file content (1175 lines) | stat: -rw-r--r-- 51,554 bytes
<!DOCTYPE html>
<html>
<head>
  <title>FCM: User Guide: Annex: The FCM 1 Extract System</title>
  <meta name="author" content="FCM team" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <link rel="icon" href="../etc/fcm-icon.png" type="image/png" />
  <link rel="shortcut icon" href="../etc/fcm-icon.png" type="image/png" />
  <link href="../etc/bootstrap/css/bootstrap.min.css" rel="stylesheet" media="screen" />
  <link href="../etc/fcm.css" rel="stylesheet" media="screen" />
</head>
<body>
  <div class="navbar navbar-inverse">
    <div class="container-fluid">
      <div class="navbar-header">
        <a class="navbar-brand" href=".."><span class="fcm-version">FCM</span></a>
      </div>
      <div class="collapse navbar-collapse">
        <ul class="nav navbar-nav">
          <li><a href="../installation/">Installation</a></li>

          <li><a class="active" href="#">User Guide</a></li>
        </ul>
      </div>
    </div>
  </div>

  <div class="page-header">
    <div class="fcm-page-content pull-right well well-sm"></div>
    <h1>FCM: User Guide: Annex: The FCM 1 Extract System</h1>
  </div>

  <div class="container">
  <div class="row">
  <div class="col-md-12">

  <h2 id="introduction">Introduction</h2>

  <p><em>The FCM 1 extract system is deprecated. The documentation for the
  current extract system can be found at <a href="make.html">FCM
  Make</a>.</em></p>

  <p>The extract system provides an interface between the revision control
  system (currently Subversion) and the build system. Where appropriate, it
  extracts code from the repository and other user-defined locations to a
  directory tree suitable for feeding into the build system. In this chapter,
  we shall use many examples to explain how to use the extract system. At the
  end of this chapter, you will be able to extract code from the local file
  system as well as from different branches of different repository URLs. You
  will also learn how to mirror code to an alternate destination. Finally, you
  will be given an introduction on how to specify configurations for the build
  system via the extract configuration file. (For further information on the
  build system, please see the next chapter <a href="build.html">The Build
  System</a>.) The last section of the chapter tells you what you can do in the
  case when Subversion is not available.</p>

  <h2 id="command">The Extract Command</h2>

  <p>To invoke the extract system, simply issue the command:</p>
  <pre>
fcm extract
</pre>

  <p>By default, the extract system searches for an extract configuration file
  <samp>ext.cfg</samp> in <samp>$PWD</samp> and then <samp>$PWD/cfg</samp>. If
  an extract configuration file is not found in these directories, the command
  fails with an error. If an extract configuration file is found, the system
  will use the configuration specified in the file to perform the current
  extract.</p>

  <p>If the destination of the extract does not exist, the system performs a
  new full extract to the destination. If a previous extract already exists at
  the destination, the system performs an incremental extract, updating any
  modifications if necessary. If a full (fresh) extract is required for
  whatever reason, you can invoke the extract system using the <code>-f</code>
  option, (i.e. the command becomes <code>fcm extract -f</code>). If you simply
  want to remove all the items generated by a previous extract in the
  destination, you can invoke the extract system using the <code>--clean</code>
  option.</p>

  <p>For further information on the extract command, please see <a href=
  "command_ref.html#fcm-extract">FCM Command Reference &gt; fcm extract</a>.</p>

  <h2 id="simple">Simple Usage</h2>

  <p>The extract configuration file is the main user interface of the extract
  system. It is a line based text file. For a complete set of extract
  configuration file declarations, please refer to the <a href=
  "annex_ext_cfg.html">Annex: Declarations in FCM extract configuration
  file</a>.</p>

  <h3 id="simple_local">Extract from a local path</h3>

  <p>A simple example of a basic extract configuration file is given below:</p>
  <pre id="example_1">
# Example 1
# ----------------------------------------------------------------------
cfg::type         ext       # line 1
cfg::version      1.0       # line 2
                            # line 3
dest              $PWD      # line 4
                            # line 5
repos::var::user  $HOME/var # line 6
                            # line 7
expsrc::var::user code      # line 8
</pre>

  <p>The above demonstrates how to use the extract system to extract code from
  a local user directory. Here is an explanation of what each line does:</p>

  <ul>
    <li><dfn>line 1</dfn>: the label <code>CFG::TYPE</code> declares the type
    of the configuration file. The value <samp>ext</samp> tells the system that
    it is an extract configuration file.</li>

    <li><dfn>line 2</dfn>: the label <code>CFG::VERSION</code> declares the
    version of the extract configuration file. The current default is
    <samp>1.0</samp>. Although it is not currently used, if we have to change
    the format of the configuration file at a later stage, we shall be able to
    use this number to determine whether we are reading a file with an older
    format or one with a newer format.</li>

    <li><dfn>line 3</dfn>: a blank line or a line beginning with a
    <code>#</code> is a comment, and is ignored by the interpreter.</li>

    <li><dfn>line 4</dfn>: the label <code>DEST</code> declares the destination
    root directory of this extract. The value <samp>$PWD</samp> expands to the
    current working directory.</li>

    <li><dfn>line 5</dfn>: comment line, ignored.</li>

    <li><dfn>line 6</dfn>: the label
    <code>REPOS::&lt;pck&gt;::&lt;branch&gt;</code> declares the top level URL
    or path of a repository. The package name of the repository is given by
    &lt;pck&gt;. In our example, we choose <samp>var</samp> as the name of the
    package. (You can choose any name you like, however, it is usually sensible
    to use a package name that matches the name of the project or system you
    are working with.) The branch name in the repository is given by
    &lt;branch&gt;. (Again, you can choose any name you like, however, it is
    usually sensible to use a name such as <samp>base</samp>, <samp>user</samp>
    or something that matches your branch name.) In our example, the word
    <samp>user</samp> is normally used to denote a local user directory. Hence
    the statement declares that the repository path for the <samp>var</samp>
    package in the <samp>user</samp> branch can be found at
    <samp>$HOME/var</samp>.</li>

    <li><dfn>line 7</dfn>: comment line, ignored.</li>

    <li><dfn>line 8</dfn>: the label
    <code>EXPSRC::&lt;pck&gt;::&lt;branch&gt;</code> declares an
    <em>expandable</em> source directory for the package &lt;pck&gt; in the
    branch &lt;branch&gt;. In our example, the package name is
    <samp>var</samp>, and the branch name is <samp>user</samp>. These match the
    package and the branch names of the repository declaration in line 6. It
    means that the source directory declaration is associated with the path
    <samp>$HOME/var</samp>. The value of the declaration <samp>code</samp> is
    therefore a sub-directory under <samp>$HOME/var</samp>. By declaring a
    source directory using an <code>EXPSRC</code> label, the system
    automatically searches for all sub-directories (recursively) under the
    declared source directory.</li>
  </ul>

  <p>Invoking the extract system using the above configuration file will
  extract all sub-directories under <samp>$HOME/var/code</samp> to
  <samp>$PWD/src/var/code</samp>. Note: the extract system ignores symbolic
  links and hidden files, (i.e. file names beginning with a <samp>.</samp>). It
  will write a build configuration file to <samp>$PWD/cfg/bld.cfg</samp>. The
  configuration used for this extract will be written to the configuration file
  at <samp>$PWD/cfg/ext.cfg</samp>.</p>

  <dl>
    <dt>Note - incremental extract</dt>

    <dd>Suppose you have already performed an extract using the above
    configuration file. At a later time, you have made some changes to some of
    the files in the source directory. Re-running the extract system on the
    same configuration will trigger an incremental extract. In an incremental
    extract, the system will update only those files that are modified. If the
    last modified time (or last commit revision) of a source file in the
    current extract differs from that in the previous extract, the system will
    attempt a content comparison. The system updates the destination only if
    the content and/or file access permission of the source differs from that
    of the destination.</dd>
  </dl>

  <h3 id="simple_url">Extract from a Subversion URL</h3>

  <p>The next example demonstrates how to extract from a Subversion repository
  URL:</p>
  <pre id="example_2">
# Example 2
# ----------------------------------------------------------------------
cfg::type           ext                    # line 1
cfg::version        1.0                    # line 2
                                           # line 3
dest                $PWD                   # line 4
                                           # line 5
repos::var::base    svn://server/var/trunk # line 6
revision::var::base 1234                   # line 7
                                           # line 8
expsrc::var::base   code                   # line 9
</pre>

  <ul>
    <li><dfn>line 1-5</dfn>: same as <a href="#example_1">example 1</a>.</li>

    <li><dfn>line 6</dfn>: the line declares the repository location of the
    <samp>base</samp> branch of the <samp>var</samp> package to be the
    Subversion URL <samp>svn://server/var/trunk</samp>.</li>

    <li><dfn>line 7</dfn>: the label
    <code>REVISION::&lt;pck&gt;::&lt;branch&gt;</code> declares the revision of
    the repository associated with the package &lt;pck&gt; in the branch
    &lt;branch&gt;. The current line tells the extract system to use revision
    1234 of <samp>svn://server/var/trunk</samp>. It is worth noting that the
    declared revision must be a revision when the declared branch exists. The
    actual revision used is the last changed revision of the declared one. If
    the revision is not declared, the default is to use the last changed
    revision at the HEAD of the branch.</li>

    <li><dfn>line 8</dfn>: comment line, ignored.</li>

    <li><dfn>line 9</dfn>: the line declares an expandable source directory in
    the repository <samp>svn://server/var/trunk</samp>.</li>
  </ul>

  <p>Invoking the extract system using the above configuration file will
  extract all sub-directories under <samp>svn://server/var/trunk/code</samp> to
  <samp>$PWD/src/var/code</samp>. It will write a build configuration file to
  <samp>$PWD/cfg/bld.cfg</samp>. The configuration used for this extract will
  be written to the configuration file at <samp>$PWD/cfg/ext.cfg</samp>.</p>

  <dl>
    <dt>EXPSRC or SRC?</dt>

    <dd>
      <p>So far, we have only declared source directories using the
      <code>EXPSRC</code> statement, which stands for <em>expandable source
      directory</em>. A source directory declared using this statement will
      trigger the system to search recursively for any sub-directories under
      the declared one. Any sub-directories containing regular source files
      will be included in the extract. Symbolic links, hidden files and empty
      directories (or those containing only symbolic links and/or hidden files)
      are ignored.</p>

      <p>If you do not want the system to search for sub-directories underneath
      your declared source directory, you can declare your source directory
      using the <code>SRC</code> statement. The <code>SRC</code> statement is
      essentially the same as <code>EXPSRC</code> except that it does not
      trigger the automatic recursive search for source directories. In fact,
      the system implements the <code>EXPSRC</code> statement by expanding it
      into a list of <code>SRC</code> statements.</p>
    </dd>

    <dt>Package and sub-package</dt>

    <dd>
      <p>The second field of a repository, revision or source directory
      declaration label is the name of the container package. It is a name
      selected by the user to identify the system or project he/she is working
      on. (Therefore, it is often sensible to choose an identifier that matches
      the name of the project or system.) The package name provides a unique
      namespace for a file container. Source directories are automatically
      arranged into sub-packages, using the names of the sub-directories as the
      names of the sub-packages. For example, the declaration at line 9 in
      <a href="#example_2">example 2</a> will put the source directory in the
      <samp>var/code</samp> sub-package automatically.</p>

      <p>Note that, in additional to slash <code>/</code>, double colon
      <code>::</code> and double underscore <code>__</code> (internal only)
      also act as delimiters for package names. Please avoid using them for
      naming your files and directories.</p>

      <p>You can declare a sub-package name explicitly in your source directory
      statement. For example, the following two lines are equivalent:</p>
      <pre>
src::var::base                      code/VarMod_Surface
src::var/code/VarMod_Surface::base  code/VarMod_Surface
</pre>

      <p>Explicit sub-package declaration should not be used normally, as it
      requires a lot more typing (although there are some situations where it
      can be useful, e.g. if you need to re-define the package name).</p>

      <p>Currently, the extract system only supports non-space characters in
      the package name, as the space character is used as a delimiter between
      the declaration label and its value. If there are spaces in the path name
      to a file or directory, you should explicity re-define the package name
      of that path to a package name with no space using the above method.
      However, we recommend that only non-space characters are used for naming
      directories and files to make life simpler.</p>
    </dd>
  </dl>

  <dl>
    <dt>The expanded extract configuration file</dt>

    <dd>
      <p>At the end of a successful extract, the configuration used by the
      current extract is written in <samp>cfg/ext.cfg</samp> under the extract
      destination root. This file is an <em>expanded</em> version of the
      original, with changes in the following declarations:</p>

      <ul>
        <li>All revision keywords are converted into revision numbers.</li>

        <li>If a revision is not defined for a repository, it is set to the
        corresponding revision number of the HEAD revision.</li>

        <li>All URL keywords are converted into the full URLs.</li>

        <li>All <code>EXPSRC</code> declarations are expanded into
        <code>SRC</code> declarations.</li>

        <li>All other variables are expanded.</li>
      </ul>

      <p>With this file, it should be possible for a later extract to re-create
      the current configuration even if the contents of the repository have
      changed. (This applies only to code stored in the repository.)</p>
    </dd>
  </dl>

  <h3 id="simple_mirror">Mirror code to an alternate location</h3>

  <p>The next example demonstrates how to extract from a repository and mirror
  the code to an alternate location. It is essentially the same as <a href=
  "#example_2">example 2</a>, except that it has three new lines to describe
  how the system can mirror the extracted code to an alternate location.</p>
  <pre id="example_3">
# Example 3
# ----------------------------------------------------------------------
cfg::type           ext
cfg::version        1.0

dest                $PWD

rdest::machine      tx01                           # line 6
rdest::logname      frva                           # line 7
rdest               /scratch/frva/extract/example3 # line 8

repos::var::base    svn://server/var/trunk
revision::var::base 1234

expsrc::var::base   code
</pre>

  <p>Here is an explanation of what each line does:</p>

  <ul>
    <li><dfn>line 6</dfn>: <code>RDEST::MACHINE</code> declares the target
    machine to which the code will be mirrored. The example mirrors the code to
    the machine named <samp>tx01</samp>.</li>

    <li><dfn>line 7</dfn>: <code>RDEST::LOGNAME</code> declares the user name
    of the target machine, to which the user has login access. If this is not
    declared, the system uses the login name of the current user on the local
    machine.</li>

    <li><dfn>line 8</dfn>: <code>RDEST</code> declares the root directory of
    the alternate destination, where the mirror version of the extract will be
    sent.</li>
  </ul>

  <p>Invoking the extract system on the above configuration will trigger an
  extract similar to that given in <a href="#example_2">example 2</a>, but it
  will also attempt to mirror the contents at <samp>$PWD/src/var/code</samp> to
  <samp>/scratch/frva/extract/example3/src</samp> on the alternate destination.
  It will also mirror the expanded extract configuration file
  <samp>$PWD/cfg/ext.cfg</samp> to
  <samp>/scratch/frva/extract/example3/cfg/ext.cfg</samp> and
  <samp>$PWD/cfg/bld.cfg</samp> to
  <samp>/scratch/frva/extract/example3/cfg/bld.cfg</samp>. It is also worth
  noting that the content of the build configuration file will be slightly
  different, since it will include directory names appropriate for the
  alternate destination.</p>

  <dl>
    <dt>Note - mirroring command</dt>

    <dd>
      <p>The extract system currently supports <code>rdist</code> and
      <code>rsync</code> as its mirroring tool. The default is
      <code>rsync</code>. To use <code>rdist</code> instead of
      <code>rsync</code>, add the following line to your extract configuration
      file:</p>
      <pre>
rdest::mirror_cmd  rdist
</pre>

      <p>If <code>rsync</code> is used to mirror an extract, the system needs to
      issue a separate remote shell command to create the container directory of
      the mirror destination. The default is to issue a shell command in the
      form <samp>ssh -n -oBatchMode=yes LOGNAME@MACHINE mkdir -p DEST</samp>.
      The following declarations can be used to modify the command:</p>
      <pre>
# Examples using the default settings:
rdest::rsh_mkdir_rsh         ssh
rdest::rsh_mkdir_rshflags    -n -oBatchMode=yes
rdest::rsh_mkdir_mkdir       mkdir
rdest::rsh_mkdir_mkdirflags  -p
</pre>

      <p>In addition, the default <code>rsync</code> shell command is
      <samp>rsync -a --exclude='.*' --delete-excluded --timeout=900 --rsh='ssh
      -oBatchMode=yes' SOURCE DEST</samp>. The following declarations can be
      used to modify the command:</p>
      <pre>
# Examples using the default settings:
rdest::rsync       rsync
rdest::rsyncflags  -a --exclude='.*' --delete-excluded --timeout=900 \
                   --rsh='ssh -oBatchMode=yes'
</pre>
    </dd>
  </dl>

  <h2 id="advanced">Advanced Usage</h2>

  <h3 id="advanced_multi">Extract from multiple repositories</h3>

  <p>So far, we have only extracted from a single location. The extract system
  is not much use if that is the only thing it can do. In fact, the extract
  system supports extract of multiple source directories from multiple branches
  in multiple repositories. The following configuration file is an example of
  how to extract from multiple repositories:</p>
  <pre id="example_4">
# Example 4
# ----------------------------------------------------------------------
cfg::type           ext
cfg::version        1.0

dest                $PWD

repos::var::base    fcm:var_tr              # line 6
repos::ops::base    fcm:ops_tr              # line 7
repos::gen::base    fcm:gen_tr              # line 8

revision::gen::base 2468                    # line 10

expsrc::var::base   src/code                    # line 12
expsrc::var::base   src/scripts                 # line 13
expsrc::ops::base   src/code                    # line 14
src::gen::base      src/code/GenMod_Constants   # line 15
src::gen::base      src/code/GenMod_Control     # line 16
src::gen::base      src/code/GenMod_FortranIO   # line 17
src::gen::base      src/code/GenMod_GetEnv      # line 18
src::gen::base      src/code/GenMod_ModelIO     # line 19
src::gen::base      src/code/GenMod_ObsInfo     # line 20
src::gen::base      src/code/GenMod_Platform    # line 21
src::gen::base      src/code/GenMod_Reporting   # line 22
src::gen::base      src/code/GenMod_Trace       # line 23
src::gen::base      src/code/GenMod_UMConstants # line 24
src::gen::base      src/code/GenMod_Utilities   # line 25
</pre>

  <p>Here is an explanation of what each line does:</p>

  <ul>
    <li><dfn>line 6-8</dfn>: these lines declare the repositories for the
    <samp>base</samp> branches of the <samp>var</samp>, <samp>ops</samp> and
    <samp>gen</samp> packages respectively. It is worth noting that the values
    of the declarations are no longer Subversion URLs but are FCM URL keywords.
    These keywords are normally declared in the central configuration file of
    the FCM system, and will be expanded into the corresponding Subversion URLs
    by the FCM system. For further information on URL keywords, please see
    <a href="code_management.html#svn_basic_keywords">Code Management System
    &gt; Using Subversion &gt; Basic Command Line Usage &gt; Repository &amp;
    Revision Keywords</a>.</li>

    <li><dfn>line 10</dfn>: this line declares the revision number for the
    <samp>base</samp> branch of the <samp>gen</samp> package, i.e. for the
    <samp>fcm:gen_tr</samp> repository. It is worth noting that the revision
    numbers for the <samp>var</samp> and <samp>ops</samp> packages have not
    been declared. By default, their revision numbers will be set to the last
    changed revision at the HEAD.</li>

    <li><dfn>line 12-14</dfn>: these line declares the source directories for
    the <samp>base</samp> branches of the <samp>var</samp> and <samp>ops</samp>
    packages. For the <samp>var</samp> package, we are extracting everything
    from the <samp>code</samp> and the <samp>scripts</samp> sub-directory. For
    the <samp>ops</samp> package, we are extracting everything from the
    <samp>code</samp> directory.</li>

    <li><dfn>line 15-25</dfn>: these line declares the source directories for
    the <samp>base</samp> branch of the <samp>gen</samp> package. The source
    directories declared will not be searched for sub-directories underneath
    the declared directories.</li>
  </ul>

  <p>We shall end up with a directory tree such as:</p>
  <pre>
$PWD
   |
   |--- cfg
   |      |
   |      |--- bld.cfg
   |      |--- ext.cfg
   |
   |--- src
          |
          |--- gen
          |      |
          |      |--- code
          |              |
          |              |--- GenMod_Constants
          |              |--- GenMod_Control
          |              |--- GenMod_FortranIO
          |              |--- GenMod_GetEnv
          |              |--- GenMod_ModelIO
          |              |--- GenMod_ObsInfo
          |              |--- GenMod_Platform
          |              |--- GenMod_Reporting
          |              |--- GenMod_Trace
          |              |--- GenMod_UMConstants
          |              |--- GenMod_Utilities
          |
          |--- ops
          |      |
          |      |--- code
          |              |
          |              |--- ...
          |
          |--- var
                 |
                 |--- code
                 |       |
                 |       |--- ...
                 |
                 |--- scripts
                         |
                         |--- ...
</pre>

  <dl>
    <dt>Note - revision number</dt>

    <dd>
      <p>As seen in the above example, if a revision number is not specified
      for a repository URL, it defaults to the last changed revision at the
      HEAD of the branch. The revision number can also be declared in other
      ways:</p>

      <ul>
        <li>Any revision arguments acceptable by Subversion are allowed. You
        can use a valid revision number, a date between a pair of curly
        brackets (e.g. <samp>{2005-05-01T12:00}</samp>) or the keyword HEAD.
        However, please do not use the keywords BASE, COMMITTED or PREV as
        these are reserved for working copy only.</li>

        <li>FCM revision keywords are allowed. These must be defined for the
        corresponding repository URLs in either the central or the user FCM
        configuration file. For further information on revision keywords,
        please see <a href="code_management.html#svn_basic_keywords">Code
        Management &gt; Using Subversion &gt; Basic Command Line Usage &gt;
        Repository &amp; Revision Keywords</a>.</li>

        <li>Do not use the keyword USER, as it is used internally by the
        extract system.</li>
      </ul>

      <p>If a revision number is specified for a branch, the actual revision
      used by the extract system is the last changed revision of the branch,
      which may differ from the declared revision. While this behaviour is
      useful in most situations, some users may find it confusing to work with.
      It is possible to alter this behaviour so that extract will fail if the
      declared revision does not correspond to a changeset of the declared
      branch. Make the following declaration to switch on this checking:</p>
      <pre>
revmatch  true
</pre>
    </dd>
  </dl>

  <h3 id="advanced_branches">Extract from multiple branches</h3>

  <p>We have so far dealt with a single branch in any package. The extract
  system can be used to <em>combine</em> changes from different branches of a
  package. An example is given below:</p>
  <pre id="example_5">
# Example 5
# ----------------------------------------------------------------------
cfg::type               ext
cfg::version            1.0
 
dest                    $PWD
 
repos::var::base        fcm:var_tr
repos::ops::base        fcm:ops_tr
repos::gen::base        fcm:gen_tr
 
revision::gen::base     2468
 
expsrc::var::base       src/code
expsrc::var::base       src/scripts
expsrc::ops::base       src/code
src::gen::base          src/code/GenMod_Constants
src::gen::base          src/code/GenMod_Control
src::gen::base          src/code/GenMod_FortranIO
src::gen::base          src/code/GenMod_GetEnv
src::gen::base          src/code/GenMod_ModelIO
src::gen::base          src/code/GenMod_ObsInfo
src::gen::base          src/code/GenMod_Platform
src::gen::base          src/code/GenMod_Reporting
src::gen::base          src/code/GenMod_Trace
src::gen::base          src/code/GenMod_UMConstants
src::gen::base          src/code/GenMod_Utilities

repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 27
repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 28
repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 29
</pre>

  <p>The configuration file in <a href="#example_5">example 5</a> is similar to
  that of <a href="#example_4">example 4</a> except for the last three lines.
  Here is an explanation of what they do:</p>

  <ul>
    <li>
      <dfn>line 27</dfn>: this line declares a repository URL for the
      <samp>branch1</samp> branch of the <samp>var</samp> package. From the URL
      of the branch, we know that the branch was created by the user
      <samp>frva</samp> based on the trunk at revision at 1234. The description
      of the branch is <samp>branch1</samp>. The following points are worth
      noting:

      <ul>
        <li>By declaring a new branch with the same package name to a
        previously declared branch, it is assumed that both branches reside in
        the same Subversion repository.</li>

        <li>No revision is declared for this URL, so the default is used which
        is the last changed revision at the HEAD of the branch.</li>

        <li>No source directory is declared for this URL. By default, if no
        source directory is declared for a branch repository, it will attempt
        to use the same set of source directories as the first declared branch
        of the package. In this case, the source directories declared for the
        <samp>base</samp> branch of the <samp>var</samp> package will be
        used.</li>
      </ul>
    </li>

    <li><dfn>line 28</dfn>: this line declares another branch called
    <samp>branch2</samp> for the <samp>var</samp> package. No source directory
    is declared for this URL either, so it will use the same set of source
    directories declared for the <samp>base</samp> branch.</li>

    <li><dfn>line 29</dfn>: this line declares a branch called
    <samp>branch1</samp> for the <samp>ops</samp> package. It will use the same
    set of source directories declared for the <samp>ops</samp> package
    <samp>base</samp> branch.</li>
  </ul>

  <p>When we invoke the extract system, it will attempt to extract from the
  first declared branch of a package, if the last changed revision of the
  source directory is the same in all the branches. However, if the last
  changed revision of the source directory differs for different branches, the
  system will attempt to obtain an extract priority list for each source
  directory, using the following logic:</p>

  <ol>
    <li>The system looks for source directory packages from the first declared
    branch to the last declared branch.</li>

    <li>The branch in which a source directory package is first declared is the
    <samp>base</samp> branch of the source directory package.</li>

    <li>The last changed revision of a source directory package in a
    subsequently declared repository branch is compared with that of the base
    branch. If the last changed revision is the same as that of the base
    branch, the source directory of this branch is discarded. Otherwise, it is
    placed at the end of the extract priority list.</li>
  </ol>

  <p>For the <samp>var</samp> package in the above example, let us assume that
  we have three source directory packages X, Y and Z under <samp>code</samp>,
  and their last changed revisions under <samp>base</samp> are 100. Let's say
  we have committed some changes to X and Z in the <samp>branch1</samp> branch
  at revision 102, and other changes to Y and Z in the <samp>branch2</samp>
  branch at revision 104, the extract priority lists for X, Y and Z will look
  like:</p>

  <ul>
    <li>X: base (100, base), branch1 (102), branch2 (100, discarded)</li>

    <li>Y: base (100, base), branch1 (100, discarded), branch2 (104)</li>

    <li>Z: base (100, base), branch1 (102), branch2 (104)</li>
  </ul>

  <p>Once we have an extract priority list for a source directory, we can begin
  extracting source files in the source directory. The source directory of the
  base branch is extracted first, followed by that in the subsequent branches.
  If a source file in a subsequent branch has the same content as the that in
  the base branch, it is discarded. Otherwise, the following logic determines
  the branch to use:</p>

  <ol>
    <li>If a source file is modified in only one subsequent branch, the source
    file in that branch is extracted.</li>

    <li>If a source file is modified in two or more subsequent branches, but
    their modifications are the same, then the source file in the first
    modification is used.</li>

    <li>If a source file is modified in two or more subsequent branches and
    their modifications differ, then the behaviour depends on the "conflict
    mode" setting, which can be <code>fail</code>, <code>merge</code> (default)
    and <code>override</code>. If the conflict mode is <code>fail</code>, the
    extract fails. If the conflict mode is <code>merge</code>, the system will
    attempt to merge the changes using a tool such as <code>diff3</code>. The
    result of the merge will be used to update the destination. The extract
    fails only if there are unresolved conflicts in the merge. (In which case,
    the conflict should be resolved using the version control system before
    re-running the extract system.) If the conflict mode is
    <code>override</code>, the change in the latest declared branch takes
    precedence, and the changes in all other branches will be ignored. The
    conflict mode can be changed using the <code>CONFLICT</code> declaration in
    the extract configuration file. E.g:
      <pre>
conflict  fail
</pre>
    </li>
  </ol>

  <p>Once the system has established which source files to use, it determines
  whether the destination file is out of date or not. The destination file is
  out of date if it does not exist or if its content differs from the version
  of the source file we are using. The system only updates the destination if
  it is considered to be out of date.</p>

  <p>The extract system can also combine changes from branches in the Subversion
  repository and the local file system. This is demonstrated in the next
  example.</p>
  <pre id="example_6">
# Example 6
# ----------------------------------------------------------------------
cfg::type               ext
cfg::version            1.0
 
dest                    $PWD
  
repos::var::base        fcm:var_tr
repos::ops::base        fcm:ops_tr
repos::gen::base        fcm:gen_tr
 
revision::gen::base     2468
 
expsrc::var::base       src/code
expsrc::var::base       src/scripts
expsrc::ops::base       src/code
src::gen::base          src/code/GenMod_Constants
src::gen::base          src/code/GenMod_Control
src::gen::base          src/code/GenMod_FortranIO
src::gen::base          src/code/GenMod_GetEnv
src::gen::base          src/code/GenMod_ModelIO
src::gen::base          src/code/GenMod_ObsInfo
src::gen::base          src/code/GenMod_Platform
src::gen::base          src/code/GenMod_Reporting
src::gen::base          src/code/GenMod_Trace
src::gen::base          src/code/GenMod_UMConstants
src::gen::base          src/code/GenMod_Utilities

repos::var::branch1     fcm:var_br/frva/r1234_new_stuff
repos::var::branch2     fcm:var_br/frva/r1516_bug_fix
repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff

repos::var::user        $HOME/var                         # line 31
repos::gen::user        $HOME/gen                         # line 32
</pre>

  <p><a href="#example_6">Example 6</a> is similar to <a href=
  "#example_5">example 5</a> except that it is also extracting from local
  directories. Here is an explanation of the lines:</p>

  <ul>
    <li><dfn>line 31-32</dfn>: these line declare the repositories for the
    <samp>user</samp> branches of the <samp>var</samp> and <samp>gen</samp>
    packages respectively. Both are local paths at the local file system. There
    are no declarations for source directories for the <samp>user</samp>
    branches, so they use the same set of source directories of the first
    declared branches, the <samp>base</samp> branches in both cases.</li>
  </ul>

  <dl>
    <dt>Note - the INC declaration</dt>

    <dd>
      You have probably realised that the above examples have many repeated
      lines. To avoid having repeated lines in multiple extract configuration
      files, you can use <code>INC</code> declarations to include other extract
      configuration files. For example, if the configuration file of example 5
      is stored in the file <samp>$HOME/example5/ext.cfg</samp>, line 1 to 29
      of <a href="#example_6">example 6</a> can be replaced with an
      <code>INC</code> declaration. <a href="#example_6">Example 6</a> can then
      be written as:
      <pre>
inc                     $HOME/example5/ext.cfg

repos::var::user        $HOME/var
repos::gen::user        $HOME/gen
</pre>

      <p>Note: the <code>INC</code> declaration supports the special
      environment variable <var>$HERE</var>. If this variable is already set in
      the environment, it acts as a normal environment variable. However, if it
      is not set, it will be expanded into the container directory of the
      current extract configuration file. This feature is particularly useful
      if you are including a hierarchy of extract configurations from files in
      the same container directory in a repository.</p>
    </dd>
  </dl>

  <h3 id="advanced_inherit">Inherit from a previous extract</h3>

  <p>All the examples above dealt with standalone extract, that is, the current
  extract is independent of any other extract. If a previous extract exists in
  another location, the extract system can inherit from this previous extract
  in your current extract. This works like a normal incremental extract, except
  that your extract will only contain the changes you have specified (compared
  with the inherited extract) instead of the full source directory tree. This
  type of incremental extract is useful in several ways. For instance:</p>

  <ul>
    <li>It is fast, because you only have to extract and mirror files that you
    have changed.</li>

    <li>The subsequent build will also be fast, since it will use incremental
    build.</li>

    <li>You do not need write access to the original extract. A system
    administrator can set up a stable version in a central account, which
    developers can then inherit from.</li>

    <li>You want an incremental extract, but you need to leave the original
    extract unmodified.</li>
  </ul>

  <p>The following example is based on <a href="#example_4">example 4</a> and
  <a href="#example_6">example 6</a>. The assumption is that an extract has
  already been performed at the directory <samp>~frva/var/vn22.0</samp> based
  on the configuration file in <a href="#example_4">example 4</a>.</p>
  <pre id="example_7">
# Example 7
# ----------------------------------------------------------------------
cfg::type               ext
cfg::version            1.0
 
dest                    $PWD

use                     ~frva/var/vn22.0                  # line 6

repos::var::branch1     fcm:var_br/frva/r1234_new_stuff   # line 8
repos::var::branch2     fcm:var_br/frva/r1516_bug_fix     # line 9
repos::ops::branch1     fcm:ops_br/opsrc/r3188_good_stuff # line 10

repos::var::user        $HOME/var                         # line 12
repos::gen::user        $HOME/gen                         # line 13
</pre>

  <ul>
    <li><dfn>line 6</dfn>: this line replaces line 1 to 25 of <a href=
    "#example_6">example 6</a>. It declares that the current extract should
    inherit from the previous extract located at
    <samp>~frva/var/vn22.0</samp>.</li>
  </ul>

  <p>Running the extract system using the above configuration will trigger an
  incremental extract, as if you are running an incremental extract having
  modified the configuration file in <a href="#example_4">example 4</a> to that
  of <a href="#example_6">example 6</a>. The only difference is that the
  original extract using the <a href="#example_4">example 4</a> configuration
  will be left untouched at <samp>~frva/var/vn22.0</samp>, and the new extract
  will contain only the changes in the branches declared from line 8 to 13.</p>

  <p>Note: extract inheritance allows you to add more branches to a package,
  but you should not redefine the <code>REPOS</code>, <code>REVISION</code>,
  <code>EXPSRC</code> or <code>SRC</code> declarations of a branch that is
  already declared (and already extracted) in the inherited extract. Although
  the system will not stop you from doing so, you may end up with an extract
  that does not quite do what it is supposed to do. For example, if the
  <samp>base</samp> branch in the <samp>foo</samp> package
  (<tt>repos::foo::base</tt>) is already defined and extracted in an extract
  you are inheriting from, you should not redefine any of the
  <tt>*::foo::base</tt> declarations in your current extract. However, you are
  free to add more branches for the same package with new labels (e.g.
  <tt>repos::foo::b1</tt>), and indeed new packages that are not already
  defined in the inherited extract (e.g. <tt>repos::bar::base</tt>).</p>

  <p>If you are setting up an extract to be inherited, you do not have to
  perform a build. If you don't you will still gain the benefit of incremental
  file extract, but you will be performing a full build of the code.</p>

  <dl>
    <dt>Note - inherit and mirror</dt>

    <dd>
      <p>It is worth bearing in mind that <tt>rdest::*</tt> settings are not
      inherited. If mirroring is required in the inheriting extract, it will
      require its own set of <tt>rdest::*</tt> declarations.</p>

      <p>The system will, however, assume that a mirrored version of the
      inherited extract is available for inheritance from the mirrored
      destination of the current extract.</p>
      
      <p>E.g.: Consider an extract at <samp>/path/to/inherited/</samp> and an
      inheriting extract at <samp>/path/to/current/</samp>. If the former does
      not have a mirror, the latter should not have one either. If the former
      mirrors to <samp>machine@/path/to/inherited/mirror/</samp> and the latter
      mirrors to <samp>machine@/path/to/current/mirror/</samp>, the system will
      assume that the subsequent build at
      <samp>machine@/path/to/current/mirror/</samp> can inherit from the build
      at <samp>machine@/path/to/inherited/mirror/</samp>. This is illustrated
      below:</p>

      <pre>
/path/to/current/       =&gt; at machine: /path/to/current/mirror/
use /path/to/inherited/ =&gt; at machine: use /path/to/inherited/mirror/
</pre>
    </dd>
  </dl>

  <h3 id="advanced_build">Extract - Build Configuration</h3>

  <p>Configuration settings for feeding into the build system can be declared
  through the extract configuration file using the <code>BLD::</code> prefix.
  Any line in an extract configuration containing a label with such a prefix
  will be considered a build system variable. At the end of a successful
  extract, the system strips out the <code>BLD::</code> prefix before writing
  these variables to the build configuration file. Some example entries are
  given between line 17 and 22 in the following configuration file:</p>
  <pre id="example_8">
# Example 8
# ----------------------------------------------------------------------
cfg::type           ext
cfg::version        1.0

dest                $PWD

repos::var::base    fcm:var_tr
repos::ops::base    fcm:ops_tr
repos::gen::base    fcm:gen_tr

revision::gen::base 2468

expsrc::var::base   src/code
expsrc::var::base   src/scripts
expsrc::ops::base   src/code
src::gen::base      src/code/GenMod_Constants
src::gen::base      src/code/GenMod_Control
src::gen::base      src/code/GenMod_FortranIO
src::gen::base      src/code/GenMod_GetEnv
src::gen::base      src/code/GenMod_ModelIO
src::gen::base      src/code/GenMod_ObsInfo
src::gen::base      src/code/GenMod_Platform
src::gen::base      src/code/GenMod_Reporting
src::gen::base      src/code/GenMod_Trace
src::gen::base      src/code/GenMod_UMConstants
src::gen::base      src/code/GenMod_Utilities

bld::target         VarProg_AnalysePF.exe   # line 27

bld::tool::fc       sxmpif90                # line 29
bld::tool::cc       sxmpic++                # line 30
bld::tool::ld       sxmpif90                # line 31
</pre>

  <p>The above example is essentially the same as <a href="#example_4">example
  4</a>, apart from the additional build configuration. The following is a
  simple explanation of what the lines represent: (For detail of the build
  system, please see the next chapter on <a href="build.html">The Build
  System</a>.)</p>

  <ul>
    <li><dfn>line 27</dfn>: the line declares a default target of the
    build.</li>

    <li><dfn>line 29-31</dfn>: the lines declare the Fortran compiler, the C
    compiler and the linker respectively.</li>
  </ul>

  <dl>
    <dt>Note - use of variables</dt>

    <dd>
      <p>When you start using the extract system to define compiler flags for
      the build system, you may end up having to make a lot of long and
      repetitive declarations. In this case, you may want to define variables
      to replace the repetitive parts of the declarations.</p>

      <p>Environment variables whose names contain only upper case latin
      alphabets, numbers and underscores can be referenced in a declaration
      value via the syntax <code>$NAME</code> or <code>${NAME}</code>. For
      example:</p>
      <pre>
repos::um::base    ${HOME}/svn-wc/um
bld::tool::fflags  $MY_FFLAGS
</pre>
      
      <p>You can define a user variable by making a declaration with a label
      that begins with a percent sign <code>%</code>. The value of a user
      variable remains in memory until the end of the current file is reached.
      You can reference a user variable in a declaration value via the syntax
      <code>%NAME</code> or <code>%{NAME}</code>. For example:</p>
      <pre>
# Declare a variable %fred
%fred                     -Cdebug -eC -Wf,-init heap=nan stack=nan

bld::tool::fflags         %fred
# bld::tool::fflags       -Cdebug -eC -Wf,-init heap=nan stack=nan

bld::tool::fflags::foo    %fred -f0
# bld::tool::fflags::foo  -Cdebug -eC -Wf,-init heap=nan stack=nan -f0

bld::tool::fflags::bar    -w %fred
# bld::tool::fflags::bar  -w -Cdebug -eC -Wf,-init heap=nan stack=nan
</pre>

      <p>Further to this, each declaration results in an internal variable of
      the same name and you can also refer to any of these internal variables in
      the same way. So, the example given above could also be written as
      follows:</p>
      <pre>
bld::tool::fflags         -Cdebug -eC -Wf,-init heap=nan stack=nan
bld::tool::fflags::foo    %bld::tool::fflags -f0
bld::tool::fflags::bar    -w %bld::tool::fflags
</pre>
    </dd>

    <dt>Note - as-parsed configuration</dt>

    <dd>
      <p>If you use a hierarchy of <code>INC</code> declarations or variables,
      you may end up with a configuration file that is difficult to understand.
      To help you with this, the extract system generates an as-parsed
      configuration file at <samp>cfg/parsed_ext.cfg</samp> of the destination.
      The content of the as-parsed configuration file is what the extract
      system actually reads. It should contain everything in your original
      extract configuration file, except that all <code>INC</code>
      declarations, environment variables and user/internal variables are
      expanded.</p>
    </dd>
  </dl>

  <h2 id="verbose">Diagnostic verbose level</h2>

  <p>The amount of diagnostic messages generated by the extract system is
  normally set to a level suitable for normal everyday operation. This is the
  default diagnostic verbose level 1. If you want a minimum amount of
  diagnostic messages, you should set the verbose level to 0. If you want more
  diagnostic messages, you can set the verbose level to 2 or 3. You can modify
  the verbose level in two ways. The first way is to set the environment
  variable <var>FCM_VERBOSE</var> to the desired verbose level. The second way
  is to invoke the extract system with the <code>-v &lt;level&gt;</code>
  option. (If set, the command line option overrides the environment
  variable.)</p>

  <p>The following is a list of diagnostic output at each verbose level:</p>

  <dl>
    <dt>Level 0</dt>

    <dd>
      <ul>
        <li>Report the time taken to extract the code.</li>

        <li>Report the time taken to mirror the code.</li>

        <li>If <code>rdist</code> is used to mirror the code, run the command
        with the <code>-q</code> option.</li>
      </ul>
    </dd>

    <dt>Level 1</dt>

    <dd>
      <ul>
        <li>Everything at verbose level 0.</li>

        <li>Report the name of the extract configuration file.</li>

        <li>Report the location of the extract destination.</li>

        <li>Report date/time at the beginning of the extract step.</li>

        <li>If the revision specified for a repository branch is not its last
        changed revision, print an information statement to inform the user of
        the last changed revision of the branch.</li>

        <li>Summarises the destination status and the source status.</li>

        <li>Report date/time at the beginning of the mirror step.</li>

        <li>Report the location of the alternate destination.</li>

        <li>Report total time.</li>
      </ul>
    </dd>

    <dt>Level 2</dt>

    <dd>
      <ul>
        <li>Everything at verbose level 1.</li>

        <li>If the revision specified for a repository branch is not current
        (i.e. the specified revision number is less than the revision number of
        the last commit revision), print an information statement to inform the
        user of the last commit revision of the branch.</li>

        <li>Report the detail of each change in the destination.</li>

        <li>If <code>rdist</code> is used to mirror the code, run the command
        without the <code>-q</code> option.</li>
      </ul>
    </dd>

    <dt>Level 3</dt>

    <dd>
      <ul>
        <li>Everything at verbose level 2.</li>

        <li>Report all shell commands invoked by the extract system with
        timestamp.</li>

        <li>If <code>rdist</code> is used to mirror the code, print the
        <samp>distfile</samp> supplied to the command.</li>

        <li>If <code>rsync</code> is used to mirror the code, invoke the
        command with the <code>-v</code> option.</li>
      </ul>
    </dd>
  </dl>

  <h2 id="nosvn">When Subversion Is Not Available</h2>

  <p>The extract system can still be used if Subversion is not available.
  Clearly, you can only use local repositories. However, you can still do
  incremental extract, mirror an extract to an alternate location, or combine
  code from multiple local repositories.</p>

  <p>If you are using Subversion but your server is down then clearly there is
  little you can do. However, if you already have an extract then you can
  re-run <code>fcm extract</code> as long as the extract configuration file
  only refers to fixed revisions. If this is not the case then you can always
  use the expanded extract configuration file which can be found in
  <samp>cfg/ext.cfg</samp> under the extract destination root. This means that
  you can continue to makes changes to local code and do incremental extracts
  even whilst your Subversion server is down.</p>

  </div>
  </div>
  </div>

  <hr/>
  <div class="container-fluid text-center">
    <div class="row"><div class="col-md-12">
    <address><small>
      Copyright &copy; 2006-2021 British Crown (Met Office) &amp; Contributors.
      <a href="http://www.metoffice.gov.uk">Met Office</a>.
      See <a href="../etc/fcm-terms-of-use.html">Terms of Use</a>.<br />
      This document is released under the British <a href=
      "http://www.nationalarchives.gov.uk/doc/open-government-licence/" rel=
      "license">Open Government Licence</a>.<br />
    </small></address>
    </div></div>
  </div>

  <script type="text/javascript" src="../etc/jquery.min.js"></script>
  <script type="text/javascript" src="../etc/bootstrap/js/bootstrap.min.js"></script>
  <script type="text/javascript" src="../etc/fcm.js"></script>
  <script type="text/javascript" src="../etc/fcm-version.js"></script>
</body>
</html>