.. workflow:: qa

Workflow ``qa``
===============

* ``task_data``:

  * ``prefix`` (string, optional): prefix this string to the item names
    provided in the internal collection
  * ``reference_prefix`` (string, optional unless
    ``enable_regression_tracking`` is set): prefix for the item names
    provided in the internal collection in the corresponding workflow run
    for reference tests

  * ``source_artifact`` (:ref:`lookup-single`, required): the
    :artifact:`debian:source-package` or :artifact:`debian:upload` artifact
    representing the source package to test
  * ``binary_artifacts`` (:ref:`lookup-multiple`, required): the
    :artifact:`debian:binary-package` or :artifact:`debian:upload`
    artifacts representing the binary packages to test

  * ``package_build_logs`` (:ref:`lookup-multiple`, optional): the
    :artifact:`debian:package-build-log` artifacts representing the build logs.
    Required if ``enable_blhc`` is True

  * ``qa_suite`` (:ref:`lookup-single`, optional unless
    ``enable_regression_tracking``,
    ``enable_reverse_dependencies_autopkgtest``, ``update_qa_results``,
    or ``enable_debdiff`` is True): the :collection:`debian:suite`
    collection that reference tests are being run against to detect
    regressions, to search for reverse-dependencies, and to debdiff
    against

  * ``reference_qa_results`` (:ref:`lookup-single`, optional unless
    ``enable_regression_tracking`` or ``update_qa_results`` is True): the
    :collection:`debian:qa-results` collection that contains the reference
    results of QA tasks to use to detect regressions
  * ``enable_regression_tracking`` (boolean, defaults to False): configure
    the workflow to detect and display regressions in QA results
  * ``update_qa_results`` (boolean, defaults to False): when set to True,
    the workflow runs QA tasks and updates the collection passed in
    ``reference_qa_results`` with the results.

  * ``vendor`` (string, required): the distribution vendor on which to run
    tests
  * ``codename`` (string, required): the distribution codename on which to
    run tests
  * ``extra_repositories`` (optional): see :task:`PackageBuild`
  * ``architectures`` (list of strings, optional): if set, only run on any
    of these architecture names

  * ``architectures_allowlist`` (list of strings, optional, either concrete
    architecture names or ``all``): if set, only run on any of these
    architecture names; while ``architectures`` is intended to be supplied
    by users or passed down from a higher-level workflow, this field is
    intended to be provided via :ref:`task-configuration`
  * ``architectures_denylist`` (list of strings, optional, either concrete
    architecture names or ``all``): if set, do not run on any of these
    architecture names; this field is intended to be provided via
    :ref:`task-configuration`
  * ``arch_all_build_architecture`` (string, defaults to ``amd64``): concrete
    architecture on which to run tasks for ``Architecture: all`` packages

  * ``enable_check_installability`` (boolean, defaults to True): whether to
    include installability-checking tasks
  * ``check_installability_suite`` (:ref:`lookup-single`, required if
    ``enable_check_installability`` is True): the
    :collection:`debian:suite` collection to check installability against;
    once we have a good way to look up the primary suite for a vendor and
    codename, this could default to doing so

  * ``enable_autopkgtest`` (boolean, defaults to True): whether to include
    autopkgtest tasks
  * ``autopkgtest_backend`` (string, optional): see :task:`Autopkgtest`

  * ``enable_reverse_dependencies_autopkgtest`` (boolean, defaults to
    False): whether to include autopkgtest tasks for reverse-dependencies

  * ``enable_lintian`` (boolean, defaults to True): whether to include
    lintian tasks
  * ``lintian_backend`` (string, optional): see :task:`Lintian`
  * ``lintian_fail_on_severity`` (string, optional): see :task:`Lintian`

  * ``enable_piuparts`` (boolean, defaults to True): whether to include
    piuparts tasks
  * ``piuparts_backend`` (string, optional): see :task:`Piuparts`
  * ``piuparts_environment`` (string, optional): the environment to run
    piuparts in

  * ``enable_debdiff`` (boolean, defaults to False): whether to include
    debdiff tasks for source and binary packages. Compares the supplied source
    package and the binary packages against the packages available in the
    distribution identified by ``qa_suite``.

  * ``enable_blhc`` (boolean, defaults to False): whether to include ``blhc``
    tasks for the build logs

  * ``fail_on`` (string, optional): indicate the conditions to trigger a
    failure of the whole workflow.  Allowed values are ``failure``,
    ``regression``, ``never``.  With ``failure``, the workflow is marked as
    failed if one of the QA task fails.  With ``regression``, the workflow
    fails only if one of the QA result is a regression compared to the
    former result.  With ``never``, the workflow always succeeds.  The
    default value is ``regression`` if ``enable_regression_tracking`` is
    True, otherwise it is ``failure``.

The workflow computes dynamic metadata as:

.. dynamic_data::
  :method: debusine.server.workflows.qa::QAWorkflow.build_dynamic_data

Any of the lookups in ``source_artifact`` or ``binary_artifacts`` may result
in :bare-data:`promises <debusine:promise>`, and in that case the workflow
adds corresponding dependencies.  Binary promises must include an
``architecture`` field in their data.

The effective set of architectures is ``{architectures}`` (defaulting to all
architectures supported by this Debusine instance and the
``{vendor}:{codename}`` suite, plus ``all``), intersecting
``{architectures_allowlist}`` if set, and subtracting
``{architectures_denylist}`` if set.

The workflow creates sub-workflows and tasks as follows, with substitutions
based on its own task data:

* if ``enable_check_installability`` is set, a single
  :task:`CheckInstallability`, with task data:

  * ``suite``: ``{check_installability_suite}``
  * ``binary_artifacts``: the subset of the lookup in this workflow's
    ``binary_artifacts`` for each available architecture

* if ``enable_autopkgtest`` is set, an :workflow:`autopkgtest` sub-workflow,
  with task data:

  * ``source_artifact``: ``{source_artifact}``
  * ``binary_artifacts``: the subset of the lookup in this workflow's
    ``binary_artifacts`` for each of ``all`` and the concrete architecture
    in question that exist
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``backend``: ``{autopkgtest_backend}``
  * ``architectures``: the effective set of architectures
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``

* if ``enable_reverse_dependencies_autopkgtest`` is set, a
  :workflow:`reverse_dependencies_autopkgtest` sub-workflow, with task data:

  * ``source_artifact``: ``{source_artifact}``
  * ``binary_artifacts``: the subset of the lookup in this workflow's
    ``binary_artifacts`` for each of ``all`` and the concrete architecture
    in question that exist
  * ``qa_suite``: ``{qa_suite}``
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``backend``: ``{autopkgtest_backend}``
  * ``architectures``: the effective set of architectures
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``

* if ``enable_lintian`` is set, a :workflow:`lintian` sub-workflow, with
  task data:

  * ``source_artifact``: ``{source_artifact}``
  * ``binary_artifacts``: the subset of the lookup in this workflow's
    ``binary_artifacts`` for each of ``all`` and the concrete architecture
    in question that exist
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``backend``: ``{lintian_backend}``
  * ``architectures``: the effective set of architectures
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``
  * ``fail_on_severity``: ``{lintian_fail_on_severity}``

* if ``enable_piuparts`` is set, a :workflow:`piuparts` sub-workflow, with
  task data:

  * ``binary_artifacts``: the subset of the lookup in this workflow's
    ``binary_artifacts`` for each of ``all`` and the concrete architecture
    in question that exist
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``backend``: ``{piuparts_backend}``
  * ``architectures``: the effective set of architectures
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``

.. todo::

    Not implemented: ``enable_check_installability`` and
    ``check_installability_suite``.

Behavior with ``update_qa_results`` set to True
-----------------------------------------------

When ``update_qa_results`` is set to True, the goal of the workflow
is modified: its only purpose is to provide reference results to
be stored in a :collection:`debian:qa-results` collection. Task failures are
never fatal for the parent workflow or for dependent tasks.

During orchestration, the workflow compares the data available in the
:collection:`debian:qa-results` collection together with information about
the submitted ``source_artifact`` and ``binary_artifacts``.

When a missing or outdated QA result is detected, it schedules the
appropriate QA task, and it creates a corresponding promise in the internal
collection (the name of the promise is the prefix followed by the expected
name of the collection entry).  The QA task has the following event
reactions:

* ``on_assignment``: an action to :ref:`skip the work request
  <action-skip-if-lookup-result-changed>` if the latest relevant item in the
  :collection:`debian:qa-results` collection has changed since the work
  request has created; this avoids wasting resources if multiple parallel
  workflows trigger an update of the same QA results
* ``on_success``: an action to add the result to the
  :collection:`debian:qa-results` collection
* ``on_failure``: same as ``on_success``

Note that when ``enable_reverse_dependencies_autopkgtest`` is set to True,
it must also update the autopkgtest results of the reverse dependencies
and thus compute the same list of packages as the
``reverse_dependencies_autopkgtest`` workflow (using the same
``qa_suite`` collection).

Behavior with ``enable_regression_tracking`` set to True
--------------------------------------------------------

When ``enable_regression_tracking`` is set to True, the orchestrator
of the ``qa`` workflow schedules :ref:`workflow callbacks
<workflow-callback>` that will perform the regression analysis. In order
to wait for the availability of the QA result(s), those callbacks have
dependencies against:

* the promises associated to the QA result(s) that are required from the
  additional ``qa`` workflow building reference results
* the promises associated to the QA result(s) that are required from the
  sub-workflows

The ``workflow_data`` field for those workflow callbacks have:

* ``visible`` set to False so that they do not show up in the workflow
  hierarchy (new feature to implement)
* ``step`` set to ``regression-analysis``

As part of the callback, the analysis is performed and the result
of the analysis is stored in the ``output_data`` field of the workflow.

.. note::

   We use simple workflow callbacks instead of full-fledged worker
   tasks or server tasks because we assume that regression analysis
   can be completed just by comparing the artifact metadata and/or
   the collection item. Workflow callbacks are already dealt through
   celery tasks so they are relatively cheap. Note however that the
   large number of callbacks requires use of careful locking to
   serialize the operations between concurrent runs trying to update
   the same workflow.

Handling of ``fail_on``
-----------------------

With ``fail_on: never`` or ``fail_on: regression``, all the sub-workflows
are run with ``workflow_data.allow_failure: true``.

With ``fail_on: regression``, a final orchestrator callback is scheduled:

* it depends on all the ``regression-analysis`` callbacks
* ``workflow_data.visible`` is set to True
* ``workflow_data.step`` is ``final-regression-analysis``
* ``workflow_data.display_name`` is ``Regression analysis``

The callback reviews the data in ``output_data.regression_analysis``
and sets its own result to FAILURE in case of regression, or SUCCESS
otherwise.