<#

    This is the template file. For the generated output; provide the "this is auto-generated" marker

    Want to improve some docs from the generated file that is not in this template? Have a look
    in:
      * src/debputy/plugins/debputy
      * src/debputy/plugin/api/std_docs.py.

    Most of the online documentation comes from those places.

#>
<!--
  THIS FILE IS AUTO-GENERATED; Please edit MANIFEST-FORMAT.md.j2 instead or the `debputy` command
  generating this out.

  REGENERATE WITH:

    debputy internal-command process-template docs/MANIFEST-FORMAT.md.j2 MANIFEST-FORMAT.md

-->

# The debputy manifest format

_This is [reference documentation] and is primarily useful if you have an idea of what you are looking for._
_If you are new to `debputy`, maybe you want to read [GETTING-STARTED-WITH-dh-debputy.md](GETTING-STARTED-WITH-dh-debputy.md) first._

<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ -->

<#

## Template form

This file is a python3-jinja2 template file (https://jinja.palletsprojects.com/en/stable/templates/).
This section serves as instructions for how the template works, but it will **not** be included in
the generated form courtesy of the surrounding markers.

The jinja template is triggered by using `<` rather than `{` and `>` for `}`. So `<<foo>>` is a template
variable rather than the default of `{{foo}}`. This is to avoid problems with the `debputy` example
syntax, where `{{FOO}}` is a `debputy` substitution variable.

Relevant features:

  * <<render_pmr("foo::bar", heading_level)>> will render the docs a la `debputy plugin show p-m-r foo::bar`.
  * <<render_tm("Foo", heading_level)>> will render the docs a la `debputy plugin show type-mappings Foo`.
  * <<render_all_tms(heading_level)>> will render the docs for all non-simple type mappings. Basically, a loop
    over all type mappings (a la: `debputy plugin list type-mappings`) followed by rendering each type mapping
    (<<render_tm(NAME, heading_level)>>). The heading_level will be passed to <<render_tm(...)>>.

For `heading_level`, use the header level desired (`1` for `#` or `<h1>`, `2` for `##` or `<h2>`, etc.).

#>
## Prerequisites

This guide assumes familiarity with Debian packaging in general.  Notably, you should understand
the different between a (Debian) source package and a (Debian) binary package (e.g., `.deb`) plus
how these concepts relates to `debian/control` (the source control file).

Additionally, the reader is expected to have an understanding of globs and substitution variables.

It is probably very useful to have an understanding on how a binary package is
assembled.  While `debputy` handles the assembly for you, this document will not go in details
with this. Prior experience with `debhelper` (notably the `dh`-style `debian/rules`) may be
useful but should not be a strict requirement.


# The basic manifest

The manifest is a YAML document with a dictionary at the top level layer.  As usual with YAML
versions, you can choose to leave it implicit.  All manifests must include a manifest version, which
will enable the format to change over time.  For now, there is only one version (`0.1`) and you have
to include the line:

    manifest-version: "0.1"


On its own, the manifest containing only `manifest-version: "..."` will not do anything.  So if you
end up only having the `manifest-version` key in the manifest, you can just remove the manifest and
rely entirely on the built-in rules.

## Feature limitations due to Integration Mode

The `debputy` program is build around different integration modes as documented
[INTEGRATION-MODES.md](INTEGRATION-MODES.md). Certain features require a certain integration
mode to work, which the documentation mention where relevant.

# Path matching rules

Most of the manifest is about declaring rules for a given path such as "foo must be a symlink"
or "bar must be owned by root:tty and have mode 02755".

The manifest contains the following types of matches:

 1) Exact path matches.  These specify the path inside the Debian package exactly without any
    form of wildcards (e.g., `*` or `?`).  However, they can use substitution variables.
    Examples include:
    * `usr/bin/sudo`
    * `usr/lib/{{DEB_HOST_MULTIARCH}}/libfoo.so`

    Having a leading `/` is optional.  I.e. `/usr/bin/sudo` and `usr/bin/sudo` are considered the
    same path.

 2) Glob based path matches.  These specify a rule that match any path that matches a given
    glob.  These rules must contain a glob character (e.g., `*`) _and_ a `/`. Examples include:

    * `foo/*`
    * `foo/*.txt`
    * `usr/lib/{{DEB_HOST_MULTIARCH}}/lib*.so*`

    Note that if the glob does not contain a `/`, then it falls into the Basename glob rule
    below.

 3) Basename glob matches.  These specify a rule that match any path where the basename matches
    a given glob.  They must contain a glob character (e.g., `*`) but either must not have a
    `/` in them at all or start with `**/` and not have any `/` from there.
    Examples include:

    * `"*.la"`
    * `"**/*.md"`
    * `"**/LICENSE"`

    The examples use explicit quoting because YAML often interprets `*` as an anchor rule in the
    places where you are likely to use this kind of glob.  The use of the `**/`-prefix is
    optional when the basename is a glob.  If you wanted to match all paths with the basename
    of exactly `LICENSE`, then you have to use the prefix (that is, use `**/LICENSE`) as `LICENSE`
    would be interpreted as an exact match for a top-level path.

However, there are also cases where these matching rules can cause conflicts.  This is covered
in the [Conflict resolution](#conflict-resolution) section below.


## Limitations on debputy path rules

Path rules:

 1) Must match relatively to the package root.
 2) Never resolves a symlink.
    * If `bin/ls` is a symlink, then `bin/ls` matches the symlink (and never the target)
    * If `bin` is a symlink, then `bin/ls` (or `bin/*`) would fail to match, because
      matching would require resolving the symlink.

These limitations are in place because of implementation details in `debputy`.


## Conflict resolution

The manifest can contain seemly mutually exclusive rules.  As an example, if you ask for
`foo/symlink` to be a symlink but also state that you want to remove `foo` entirely
from the Debian package then the manifest now has two mutually exclusive requests.

To resolve these problems, `debputy` generally relies on ordered instructions to
resolve conflicts. Consider `installations` and `transformations`:

   1. `installations` (ordered list using "first match wins")
   2. `package.<package-name>.transformations` (ordered list using "all matching rules applies")

The "first match wins" rule means the first rule that applies will be used for that match
and no further rules will be used for that match.

The "all matching rules applies" rule means that each rule is applied in order.  Often
this behaves like a simple case of either "first match wins" or "last match wins" (depending
on the context).

Note for transformation rules, an early rule can change the file system layout, which will
affect whether a later rule matches.  This is similar to how shell globs commands work:

      $ rm usr/lib/libfoo.la
      $ chmod 0644 usr/lib/*

Here the glob used with `chmod` will not match `usr/lib/libfoo.la` because it was removed.
As noted, a similar logic applies to transformation rules.

## All definitions must be used

Whenever you define a request or rule in the manifest, `debputy` will insist on it being used
at least once.  The exception to this rule being conditional rules where the condition
evaluates to `false` in which case the rule does not trigger an error when unused.

This is useful for several reasons:

 1. The definition may have been shadowed by another rule and would have been ignored otherwise
 2. The definition may no longer be useful, but its present might confuse a future reader of
    the manifest.

In all cases, `debputy` will tell you if a definition was unused and where you can find that
definition.

## debputy globs

In general, the following rules applies to globs in `debputy`.

 * The `*` match 0 or more times any characters except `/`.
 * The `?` match exactly one character except `/`.
 * The glob `foo/*` matches _everything_ inside `foo/` including hidden files (i.e., paths starting
   with `.`) unlike `bash`/`sh` globs. However, `foo/*` does _not_ match `foo/` itself (this latter
   part matches `bash`/`sh` behavior and should be unsurprising).
 * For the special-cases where `**` is supported, then `**` matches zero or more levels of directories.
   This means that `foo/**/*` match any path beneath `foo` (but still not `foo`).  This is mostly relevant
   for built-in path matches as it is currently not possible to define `foo/**/...` patterns in the manifest.

Note that individual rules (such as `clean-after-removal`) may impose special cases to how globs
work. The rules will explicitly list if they divert from the above listed glob rules.

# Rules for substituting manifest variables

The `debputy` tool supports substitution in various places (usually paths) via the following
rules.  That means:

 1) All substitutions must start with `{{` and end with `}}`.  The part between is
    the `MANIFEST_VARIABLE` and must match the regular expression `[A-Za-z0-9][-_:0-9A-Za-z]*`.
    Note that you can use space around the variable name if you feel that increases readability.
    (That is, `{{ FOO }}` can be used as an alternative to `{{FOO}}`).
 2) The `MANIFEST_VARIABLE` will be result from a set of built-in variables and the variables from
    `dpkg-architecture`.
 3) You can use `{{token:DOUBLE_OPEN_CURLY_BRACE}}` and `{{token:DOUBLE_CLOSE_CURLY_BRACE}}` (built-in
    variables)  if you want a literal `{{` or `}}`  would otherwise have triggered an undesired expansion.
 4) All `{{MANIFEST_VARIABLE}}` must refer to a defined variable.
    - You can see the full list of `debputy` and plugin provided manifest variables via:
      `debputy plugin list manifest-variables`. The manifest itself can declare its own variables
      beyond that list. Please refer to the [Manifest Variables](#manifest-variables-variables)
      section manifest variables declared inside the manifest.
 5) There are no expression syntax inside the `{{ ... }}` (unlike jinja2 and other template languages).
    This rule may be changed in the future (with a new manifest version).

Keep in mind that substitution _cannot_ be used everywhere.  There are specific places where
it can be used.  Also, substitution _cannot be used_ to introduce globs into paths.  When a
substitution occurs inside a path all characters inserted are treated as literal characters.

Note: While manifest variables can be substituted into various places in the `debputy` manifest, they
are distinct from `dpkg`'s "substvars" (`man 5 deb-substvars`) that are used in the `debian/control`
file.

## Built-in or common substitution variables

 * `{{token:NEWLINE}}` or `{{token:NL}}` expands to a literal newline (LF) `\n`.
 * `{{token:TAB}}` expands to a literal tab `\t`.
 * `{{token:OPEN_CURLY_BRACE}}`  / `{{token:CLOSE_CURLY_BRACE}}` expand to `{` / `}`
 * `{{token:DOUBLE_OPEN_CURLY_BRACE}}` / `{{token:DOUBLE_CLOSE_CURLY_BRACE}}` expands to `{{` / `}}`.
 * `{{PACKAGE}}` expands to the binary package name of the current binary package. This substitution
   only works/applies when the substitution occurs in the context of a concrete binary package.
 * Plus any of the variables produced by `dpkg-architecture`, such as `{{DEB_HOST_MULTIARCH}}`.

The following variables from `/usr/share/dpkg/pkg-info.mk` (`dpkg`) are also available:

 * DEB_SOURCE (as `{{DEB_SOURCE}}`)
 * DEB_VERSION (as `{{DEB_VERSION}}`)
 * DEB_VERSION_EPOCH_UPSTREAM (as `{{DEB_VERSION_EPOCH_UPSTREAM}}`)
 * DEB_VERSION_UPSTREAM_REVISION (as `{{DEB_VERSION_UPSTREAM_REVISION}}`)
 * DEB_VERSION_UPSTREAM (as `{{DEB_VERSION_UPSTREAM}}`)
 * SOURCE_DATE_EPOCH (as `{{SOURCE_DATE_EPOCH}}`)

These have the same definition as those from the `dpkg` provided makefile.


# Restrictions on defining ownership of paths

In some parts of the manifest, you can specify which user or group should have ownership of
a given path.  As an example, you can define a directory to be owned by `root` with group `tty`
(think `chown root:tty <some/path>`).

Ownership is generally defined via the keys `owner` and `group`.  For each of them, you can use
one of the following formats:

 1) A name (e.g., `owner: root`).
 2) An id (e.g., `owner: 0`). Please avoid using quotes around the ID in YAML as that can
    cause `debputy` to read the number as a name.
 3) A name and an id with a colon in between (e.g., `owner: "root:0"`).  The name must always
    come first here.  You may have to quote the value to prevent the YAML parser from being
    confused.

All three forms are valid and provide the same result.  Unless you have a compelling reason to
pick a particular form, the name-only is recommended for simplicity.  Notably, it does not
require your co-maintainer or future you to remember what the IDs mean.

Regardless of which form you pick:

 1) The provided owner must be defined by Debian `base-passwd` file, which are the only users guaranteed
    to be present on every Debian system.
    * Concretely, `debputy` verifies the input against `/usr/share/base-passwd/passwd.master` and
      `/usr/share/base-passwd/group.master` (except for `root` / `0` as an optimization).

 2) If the `name:id` form is used, then the name and the id values must match.  I.e., `root:2` would
    be invalid as the id for `root` is defined to be `0` in the `base-passwd` data files.

 3) The `debputy` tool maintains a `deny`-list of owners that it refuses even though `base-passwd`
    defines them. As a notable non-exhaustive example, `debputy` considers `nobody` or id `65534`
    (the ID of `nobody` / `nogroup`) to be invalid owners.


# Conditional rules

There are cases, where a given rule should only apply in certain cases - such as only when a given
build profile is active (`DEB_BUILD_PROFILES` / `dpkg-buildpackage -P`).  For rules that
*support being conditional*, the condition is generally defined via the `when:` key and the condition
is then described beneath the `when:`.

As an example:

    packages:
        util-linux:
            transformations:
            - create-symlink
                  path: sbin/agetty
                  target: /sbin/getty
                  when:
                      # On Hurd, the package "hurd" ships "sbin/getty".
                      arch-matches: '!hurd-any'


When the condition under `when:` resolves to `true`, the rule will and must be used.  When the
condition resolves to `false`, the rule will not be applied even if it could have been.  However,
the rule may still be "partially" evaluated.  As an example, for installation rules, the source
patterns will still be evaluated to reserve what it would have matched, so that following rules
behave deterministically regardless of how the condition evaluates.

Note that conditions are *not* used as a conflict resolution and as such two conditional rules
can still cause conflicts even though their conditions are mutually exclusive.  This may be
changed in a later version of `debputy` provided `debputy` can assert the two conditions
are mutually exclusive.

The `when:` key has either a mapping, a list or a string as value depending on the condition.
Each supported condition is described in the following subsections.


<<render_pmr('ManifestCondition::arch-matches', 2)>>
<#- source-context-arch-matches and package-context-arch-matches share docs, so we only need one of them -#>
<<render_pmr('ManifestCondition::source-context-arch-matches', 2)>>
<<render_pmr('ManifestCondition::build-profiles-matches', 2)>>
<<render_pmr('ManifestCondition::can-execute-compiled-binaries', 2)>>
<<render_pmr('ManifestCondition::cross-compiling', 2)>>
<<render_pmr('ManifestCondition::run-build-time-tests', 2)>>
<<render_pmr('ManifestCondition::not', 2)>>
<# all-of and any-of share docs, so we only need one of them -#>
<<render_pmr('ManifestCondition::all-of', 2)>>

# Packager provided definitions

For more complex manifests or packages, it is possible define some common attributes for reuse.

<<render_pmr('definitions::variables', 2)>>

# Interacting with upstream build system

In the `full` integration mode, `debputy` also attempts to interact with the upstream build
systems (if any). In other integration modes, then it is generally `dh` that manages the
upstream build system (via the `dh_auto_*` commands). By default, `debputy` in the `full`
integration mode will auto-detect a supported build system and apply some defaults for a
build. However, in some cases, you may need to tweak some settings, which will covered
with these rules.

<<render_pmr('::default-build-environment', 2)>>
<<render_pmr('::build-environments', 2)>>
<<render_pmr('::builds', 2)>>

<<render_pmr('BuildRule::autoconf', 3)>>
<<render_pmr('BuildRule::make', 3)>>
<<render_pmr('BuildRule::perl-build', 3)>>
<<render_pmr('BuildRule::perl-makemaker', 3)>>
<<render_pmr('BuildRule::debhelper', 3)>>
<<render_pmr('BuildRule::cmake', 3)>>
<<render_pmr('BuildRule::meson', 3)>>
<<render_pmr('BuildRule::qmake', 3)>>
<<render_pmr('BuildRule::qmake6', 3)>>

## Test rules (`builds.*.test-rule`)

Build systems can be associated with a `test-rule`, which adds a maintainer restriction running
the build time test suite.

<<render_pmr('TestRule::skip-tests', 3)>>
<<render_pmr('TestRule::skip-tests-when', 3)>>

<<render_pmr('::installations', 1)>>

## Install rule search directories

Most install rules apply their patterns against search directories such as `debian/tmp` by default.

The default search directory order (highest priority first) is:

 1) The upstream install directory (usually, `debian/tmp`)
 2) The source package root directory (`.`)

Each search directory is tried in order.  When a pattern matches an entry in a search directory (even
if that entry is reserved by another package), further search directories will *not* be tried. As an example,
consider the pattern `usr/bin/foo*` and the files:

  `SOURCE_ROOT/debian/tmp/usr/bin/foo.sh`
  `SOURCE_ROOT/usr/bin/foo.pl`

Here the pattern will only match `SOURCE_ROOT/debian/tmp/usr/bin/foo.sh` and not `SOURCE_ROOT/usr/bin/foo.pl`.

## Automatic discard rules

The `debputy` framework provides some built-in discard rules that are applied by default during installation
time.  These are always active and implicit, but can be overwritten by exact path matches for install rules.

The `debputy` tool itself provides the following discard rules:

 * Discard of `.la` files. Their use is rare but not unheard of. You may need to overwrite this.
 * Discard of python byte code (such as `__pycache__` directories).
 * Discard of editor backup files (such as `*~`, `*.bak`, etc.).
 * Discard of Version control files (such as `.gitignore`, etc.).
 * Discard of GNU info's `dir` (`usr/share/info/dir`) as it causes file conflicts with other packages.
 * Discard of `DEBIAN` directory.

Note: Third-party plugins may provide additional automatic discard rules. Please use
`debputy plugin list automatic-discard-rules` to see all known automatic discard rules.

If you find yourself needing a particular path installed that has been discarded by default, you can overrule
the default discard by spelling out the path. As an example, if you needed to install a `libfoo.la` file,
you could do:

    installations:
      - install:
            sources:
            # By-pass automatic discard of `libfoo.la` - globs *cannot* be used!
             - "usr/lib/libfoo.la"
             - "usr/lib/libfoo*.so*"
            into: libfoo1

<<render_pmr('InstallRule::install', 2)>>
<<render_pmr('InstallRule::install-docs', 2)>>
<<render_pmr('InstallRule::install-examples', 2)>>
<<render_pmr('InstallRule::install-man', 2)>>
<<render_pmr('InstallRule::discard', 2)>>
<<render_pmr('InstallRule::multi-dest-install', 2)>>

# Binary package rules

Inside the manifest, the `packages` mapping can be used to define requests for the binary packages
you want `debputy` to produce.  Each key inside `packages` must be the name of a binary package
defined in `debian/control`.  The value is a dictionary defining which features that `debputy`
should apply to that binary package.  An example could be:


    packages:
        foo:
            transformations:
                - create-symlink:
                      path: usr/share/foo/my-first-symlink
                      target: /usr/share/bar/symlink-target
                - create-symlink:
                      path: usr/lib/{{DEB_HOST_MULTIARCH}}/my-second-symlink
                      target: /usr/lib/{{DEB_HOST_MULTIARCH}}/baz/symlink-target
        bar:
            transformations:
            - create-directories:
               - some/empty/directory.d
               - another/empty/integration-point.d
            - create-directories:
                 path: a/third-empty/directory.d
                 owner: www-data
                 group: www-data

In this case, `debputy` will create some symlinks inside the `foo` package and some directories for
the `bar` package.  The following subsections define the keys you can use under each binary package.


<<render_pmr('::transformations', 2)>>
<<render_pmr('TransformationRule::remove', 3)>>
<<render_pmr('TransformationRule::move', 3)>>
<<render_pmr('TransformationRule::create-symlink', 3)>>
<<render_pmr('TransformationRule::create-directories', 3)>>
<<render_pmr('TransformationRule::path-metadata', 3)>>

<<render_pmr('packages.{{PACKAGE}}::services', 2)>>

### Service managers and aliases

When defining a service rule, you can use any name that any of the relevant service managers would call the
service. As an example, consider a package that has the following services:

 * A `sysvinit` service called `foo`
 * A `systemd` service called `bar.service` with `Alias=foo.service` in its definition.

Here, depending on which service managers are relevant to the rule, you can use different names to match.
When the rule applies to the `systemd` service manager, then either of the following names can be used:

 * `bar.service` (the "canonical" name in the systemd world)
 * `foo.service` (the defined alias)
 * `bar` + `foo` (automatic aliases based on the above)

Now, if rule *also* applies to the `sysvinit` service manager, then any of those 4 names would cause the
rule to apply to both the `systemd` and the `sysvinit` services.

To show concrete examples:

    ...:
            services:
              # Only applies to systemd. Either of the 4 names would have work.
              - service: "foo.service"
                on-upgrade: stop-then-start
                service-manager: systemd

    ...:
            services:
              # Only applies to sysvinit. Must use `foo` since the 3 other names only applies when systemd
              # is involved.
              - service: "foo"
                on-upgrade: stop-then-start
                service-manager: sysvinit

    ...:
            services:
              # Applies to both systemd and sysvinit; this works because the `systemd` service provides an
              # alias for `foo`. If the systemd service did not have that alias, only the `systemd` service
              # would have been matched.
              - service: bar
                enable-on-install: false

<<render_pmr('packages.{{PACKAGE}}::binary-version', 2)>>
<<render_pmr('packages.{{PACKAGE}}::clean-after-removal', 2)>>
<<render_pmr('packages.{{PACKAGE}}::installation-search-dirs', 2)>>
<<render_pmr('packages.{{PACKAGE}}::built-using', 2)>>
<<render_pmr('packages.{{PACKAGE}}::static-built-using', 2)>>

<<render_pmr('::remove-during-clean', 1)>>

# Type listing

During the documentation, you may see references to a types such as `ManifestCondition` or `TestRule`.

The documentation for these are:

 * [`ManifestCondition`](#conditional-rules)
 * [`BuildRule`](#build-rules-builds) or [`BuildSystemRule`](#build-rules-builds)
 * [`TestRule`](#test-rules-buildstest-rule)
 * [`InstallRule`](#installations-installations)
 * [`TransformationRule`](#transformations-transformations)

Additionally, there are some type mappings such as `OctalMode`. These are documented below.

## Base types

These are the base types, which you may see.

 * `string`: A simple string. Often accepts substitutions in form of `{{VARIABLE}}`.
 * `int`: An integer.
 * `list of X`: A list of `X`, where `X` is a separate type.
 * `mapping of X`: A mapping. The keys are always `string` and the values are of type `X`,
   where `X` is a separate type.

## Type mappings

Here are some named types used in the documentation.

<<render_all_tms(3)>>

## Other type mappings

If you see a type that is not listed anywhere in this document, then it might be from a third-party
plugin. Try one of the following commands to resolve it:

 * `debputy plugin show type-mappings <TypeMapping>`
 * `debputy plugin show p-m-r <attribute-name>` (or `debputy plugin show p-m-r <BaseType>::<attribute-name>`)

You can also do this with `debputy` provided types. As an example `debputy plugin show type-mappings OctalMode`
or `debputy plugin show p-m-r install-man` (a.k.a. `debputy plugin show p-m-r InstallRule::install-man`).

[reference documentation]: https://documentation.divio.com/reference/